tahoe backup loops on recursive links

[francois]

francois@korn:~$ mkdir /tmp/test-backup/
francois@korn:~$ ln -s . /tmp/test-backup/test
francois@korn:~$ tahoe backup -v /tmp/test-backup tahoe:tmp
/usr/lib/pymodules/python2.6/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated
  import struct, sets, time
processing /tmp/test-backup, olddircap None
processing /tmp/test-backup/test, olddircap None
processing /tmp/test-backup/test/test, olddircap None
processing /tmp/test-backup/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
processing /tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test, olddircap None
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 8, in <module>
    load_entry_point('allmydata-tahoe==1.5.0', 'console_scripts', 'tahoe')()
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/runner.py", line 91, in run
    rc = runner(sys.argv[1:])
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/runner.py", line 78, in runner
    rc = cli.dispatch[command](so)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/cli.py", line 456, in backup
    rc = tahoe_backup.backup(options)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 370, in backup
    return bu.run()
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 212, in run
    new_backup_dircap = self.process(options.from_dir, latest_backup_dircap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 262, in process
    newchilddircap = self.process(childpath, oldchildcap)
  File "/usr/lib/pymodules/python2.6/allmydata/scripts/tahoe_backup.py", line 268, in process
    raise BackupProcessingError("Cannot backup this file %r" % childpath)
allmydata.scripts.tahoe_backup.BackupProcessingError: Cannot backup this file '/tmp/test-backup/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test/test'
francois@korn:~$ 

[zooko]

#104 (does cp -r work as expected?) was about this issue as well as a couple of others. Moving the loop issue from #104 to here.

[warner]

As #641 points out, if we could either skip symlinks or record them as symlinks (instead of following them), this would go away.

Francois: it looks like your patch only catches cycles of length one, right? I.e. if someone did ln -s .. foo or ln -s ../.. foo, then "tahoe backup" would still get stuck in a loop, yeah?

The more general solution (without fixing #641) would involve keeping a full list of every directory we'd ever started to process, and comparing the target of each symlinky child against that list, and bailing if it was found. That sounds like a lot of memory.

[davidsarah]

Replying to zooko:

#104 (does cp -r work as expected?) was about this issue as well as a couple of others.

I don't think it's the same issue. This is only about recursive symlinks in the local filesystem. cp -r needs to handle loops both in the Tahoe and local filesystems. As Brian says, the only way to do that correctly in general is to keep track of the mutable directory storage indices (for the Tahoe fs) or the inode numbers (for the local fs) that you've already seen.

[francois]

Replying to warner:

Francois: it looks like your patch only catches cycles of length one, right? I.e. if someone did ln -s .. foo or ln -s ../.. foo, then "tahoe backup" would still get stuck in a loop, yeah?

Yes, Brian, you're right, this fix is wrong. Can someone with the necessary trac permissions delete this patch from the ticket ?

[warner]

eh, I think it's fine to leave it there: no harm in having well-labeled experiments lying around :)

I don't know how to fix this in general. I suspect it will require having explicit support for symlinks in tahoe (#641), at least the ability to copy the symlink target name into a tahoe file-like object and then copy it back out again. A short-term fix (again #641, also #729) would be to let you skip over symlinks.. that's what I've been doing for my personal backups.

[warner]

e769bbb6dd41d40c fixes this, by skipping over all symlinks (with a warning).