backupdb and ext4 i_version/generation xattributes

[warner]

I recently learned that several linux filesystems can track version/generation numbers for local files. We could use this information in the backupdb to improve the speed+reliability of detecting files that have not been modified since the last time we did a backup.

lsattr -v FOO.txt shows a "version/generation number", and probably works for even old ext2 filesystems.

​this and ​this talk about "mounting a filesystem with i_version support", and suggests that the following ext4 extended-attributes will become available:

• file.crtime - actual file creation time
• file.i_generation - inode generation number
• file.i_version ("directories only") - inode data version number

It's not yet clear to me what information is really available, or how one might get to it (especially from python), but this ticket is to remind me that "tahoe backup" would be a lot better if we could quickly and reliably determine that a file had not changed. The filesize+timestamp heuristic is useful, but it'd be nice to be able to do better. A real generation number would be ideal, if the kernel promises to update it reliably. A kernel-maintained hash of the filesystem contents would be great too (it would let us detect renames without reading the file contents).

[davidsarah]

Replying to warner:

It's not yet clear to me what information is really available, or how one might get to it (especially from python), [...]

​The patch you linked to would have made this info accessible via "extended attributes" for ext4 filesystems, but it doesn't look like that patch was accepted.

In general, anything you can do from C, you can do from Python using ctypes (if the FFI overhead is not an issue, as it probably isn't in this case).

[randombit]

It may be hard to use this safely, though: on my local XFS filesystems, and remote ZFS mounts, lsattr -v returns a number, but doesn't update it on writes. It fails (bad ioctl for device type) on a tmpfs, which is far safer. It's not clear to me where the values reported on XFS and ZFS filesystems are coming from (it's not the inode number, at least). Trying to set the values (using chattr -v) also fails on these filesystems.

I like the idea of a kernel maintained per-file hash cache. I may play with this in my copious free time.

[davidsarah]

Replying to randombit:

It may be hard to use this safely, though: on my local XFS filesystems, and remote ZFS mounts, lsattr -v returns a number, but doesn't update it on writes. It fails (bad ioctl for device type) on a tmpfs, which is far safer.

We could potentially test whether this attribute is working correctly in a given directory subtree, by updating a test file and seeing whether it changes.