Recovering a PBS Backup

Sometimes you think you’re doing everything right: you copy your Proxmox Backup Server (PBS) datastore to a USB disk, mount it in your PBS VM, and try to add it back. Then you’re hit with cryptic errors like:

TASK ERROR: unable to open existing chunk store path "/mnt/usb/.chunks" - permissions or owner not correct

Or worse:

EMLINK: Too many links

This post explains why this happens, what we learned the hard way, and the exact steps to reliably recover a PBS datastore from removable storage.

The Problem

PBS is extremely strict about datastores:

  • They must reside on a POSIX filesystem that supports extended attributes and directories with tens of thousands of subdirs.
  • Permissions must exactly match PBS’s expectations (backup:backup user, restrictive modes).
  • .chunks must be private to the backup user, and root directories can’t be world-writable.

When we first tried:

  • Mounting the USB via Samba → failed, CIFS doesn’t support required attributes.
  • Mounting via NFS → failed, same problem.
  • Using a disk formatted with ext4 → failed with EMLINK: Too many links, because the ext4 volume wasn’t created with large_dir/dir_nlink features, so PBS couldn’t create 65k subdirs in .chunks.

Bottom line: ext4 defaults and network filesystems aren’t suitable for PBS datastores.

The Solution

The winning recipe:

  1. Attach a new virtual disk to the PBS VM from the Proxmox host.
  2. Format it as XFS (robust defaults, handles huge directories easily).
  3. Copy your old PBS datastore (.chunks, ct, vm, etc.) onto the XFS partition.
  4. Create a new datastore directory with PBS itself, then swap in your copied data.
  5. Register the datastore in PBS, and voilà — backups reappear in the GUI.

Step-by-Step Guide

1. Add a new virtual disk

In Proxmox VE, attach a new disk to your PBS VM. Assume it shows up as /dev/sdc.

2. Format as XFS

Inside the PBS VM:

mkfs.xfs -m crc=1 -n ftype=1 /dev/sdc1
mkdir -p /mnt/sdc
mount /dev/sdc1 /mnt/sdc

Make it persistent:

blkid /dev/sdc1
# add to /etc/fstab
UUID=<uuid-from-blkid>  /mnt/sdc  xfs  defaults  0  2

3. Copy your datastore

Suppose the old USB datastore is mounted at /mnt/usb. Copy everything over:

rsync -aHAX --numeric-ids /mnt/usb/ /mnt/sdc/

Now /mnt/sdc should contain .chunks, ct, vm, etc.

4. Seed a new datastore

Let PBS create the correct structure and perms:

mkdir /mnt/sdc/_new
chown backup:backup /mnt/sdc/_new
chmod 0750 /mnt/sdc/_new

proxmox-backup-manager datastore create recovered /mnt/sdc/_new

PBS will create /mnt/sdc/_new/.chunks with the right defaults.

5. Replace .chunks and indexes

Stop PBS temporarily:

systemctl stop proxmox-backup-proxy

Replace PBS’s empty .chunks with your real one:

mv /mnt/sdc/_new/.chunks /mnt/sdc/_new/.chunks.ref
mv /mnt/sdc/.chunks /mnt/sdc/_new/
mv /mnt/sdc/ct /mnt/sdc/_new/
mv /mnt/sdc/vm /mnt/sdc/_new/

Fix ownership and permissions:

chown -R backup:backup /mnt/sdc/_new
chmod 0750 /mnt/sdc/_new
chmod 0750 /mnt/sdc/_new/.chunks
find /mnt/sdc/_new/.chunks -type d -exec chmod 0700 {} +
find /mnt/sdc/_new/.chunks -type f -exec chmod 0600 {} +

Restart PBS:

systemctl start proxmox-backup-proxy

6. Verify in PBS

Check from the CLI:

proxmox-backup-manager datastore list

And open the PBS WebUI — your backups should be visible again 🎉.

Lessons Learned

  • PBS is picky by design — it enforces strict ownership and perms to guarantee security.
  • ext4 needs special features (large_dir, dir_nlink) to host PBS .chunks; most defaults don’t have them.
  • XFS is the safe default for PBS datastores. Just use it.
  • Always let PBS initialize the datastore first, then copy your existing data into its structure.

Final Thoughts

Recovering PBS backups taught us two key things:

  1. Don’t fight PBS’s permission checks — instead, mirror its exact defaults.
  2. Use a filesystem that supports PBS’s needs without surprises (XFS).

Now we can safely move PBS datastores around, and if disaster strikes, we know how to bring them back.