Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup partially fails: error taking snapshot of volume: error reading from server: EOF #182

Open
NicoG60 opened this issue Oct 13, 2022 · 1 comment
Labels
Bug issue/pr is a bug/fix to existing feature

Comments

@NicoG60
Copy link

NicoG60 commented Oct 13, 2022

What steps did you take and what happened:
I followed the various steps from velero and this repo documentations to create a remote snapshot location to a minio instance running on a machine outside the cluster.

I deployed a couple of services and tried to do a backup, unfortunately I am facing an error. The backup reports as Partially Failed and the logs shows that certain volumes couldn't be backed up with the following error:

time="2022-10-13T12:42:00Z" level=warning msg="Epoll wait failed : interrupted system call" backup=velero/test3 cmd=/plugins/velero-blockstore-openebs logSource="/go/src/github.com/openebs/velero-plugin/pkg/clouduploader/server.go
:302" pluginName=velero-blockstore-openebs

<... can be 50+ similar log entries and then ...>

time="2022-10-13T12:42:00Z" level=info msg="1 errors encountered backup up item" backup=velero/test3 logSource="pkg/backup/backup.go:413" name=registry-server-7d5466494d-84fj5
time="2022-10-13T12:42:00Z" level=error msg="Error backing up item" backup=velero/test3 error="error taking snapshot of volume: rpc error: code = Unavailable desc = error reading from server: EOF" logSource="pkg/backup/backup.go:4
17" name=registry-server-7d5466494d-84fj5

What did you expect to happen:
I expect the backup to be successful

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

  • kubectl logs deployment/velero -n velero -> here
  • kubectl logs deployment/maya-apiserver -n openebs -> N/A
  • velero backup describe <backupname>
    Name:         test3
    Namespace:    velero
    Labels:       velero.io/storage-location=local-backup
    Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.2
                  velero.io/source-cluster-k8s-major-version=1
                  velero.io/source-cluster-k8s-minor-version=25
    
    Phase:  PartiallyFailed (run `velero backup logs test3` for more information)
    
    Errors:    4
    Warnings:  165
    
    Namespaces:
      Included:  *
      Excluded:  <none>
    
    Resources:
      Included:        *
      Excluded:        <none>
      Cluster-scoped:  auto
    
    Label selector:  <none>
    
    Storage Location:  local-backup
    
    Velero-Native Snapshot PVs:  auto
    
    TTL:  720h0m0s
    
    Hooks:  <none>
    
    Backup Format Version:  1.1.0
    
    Started:    2022-10-13 13:41:29 +0100 BST
    Completed:  2022-10-13 13:42:38 +0100 BST
    
    Expiration:  2022-11-12 12:41:28 +0000 GMT
    
    Total items to be backed up:  767
    Items backed up:              767
    
    Velero-Native Snapshots:  1 of 5 snapshots completed successfully (specify --details for more information)
    
    CSI Volume Snapshots: <none included>
    
  • velero backup logs <backupname> -> here

Anything else you would like to add:
Currently migrating a 25 nodes cluster to k8s. This is the initial setup/test before definitive migration with just 5 nodes, self-hosted microk8s cluster running cStor

Environment:

  • Velero version (use velero version):
    Client:
            Version: v1.9.2
            Git commit: -
    Server:
            Version: v1.9.2
    
  • Velero features (use velero client config get features): features: EnableCSI
  • Velero-plugin version: 3.3.0
  • OpenEBS version: 3.3.0
  • Kubernetes version (use kubectl version): 1.25
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: 5 nodes, home made from consumer products, bought over the years so nodes are slightly different
    • Intel i7 or i9 from 8th to 10th Gen
    • 16GB to 32GB
    • 2 nodes have 3x 1TB SSD for cStor
  • OS (e.g. from /etc/os-release):
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
@NicoG60 NicoG60 added the Bug issue/pr is a bug/fix to existing feature label Oct 13, 2022
@NicoG60
Copy link
Author

NicoG60 commented Oct 17, 2022

Quick update, I've tried to target AWS directly instead of MinIO and I got the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug issue/pr is a bug/fix to existing feature
Projects
None yet
Development

No branches or pull requests

1 participant