[tahoe-dev] find out how much your files converge with your friends's

[Jeremy Fitzhardinge]

zooko wrote:
> On Aug 19, 2008, at 13:01 PM, Jeremy Fitzhardinge wrote:
>
>   
>> I generated this with:
>>
>> $ dupfilefind -p -I /proc,/sys,/dev,/tmp,/var/tmp > /tmp/dupfiles.out
>>     
>
> Hey Jeremy:
>
> I'm sorry to say that when I tried this I learned that my  
> implementation of -I / --ignore-dirs was utterly broken.  I fixed  
> that, added avoidance of infinite loop in case of symlink recursion,  
> and uploaded dupfilefind v1.4.0.
>
> http://pypi.python.org/pypi/dupfilefind
> http://allmydata.org/trac/dupfilefind
>
> You can install it with "easy_install -U dupfilefind".

This fell into a cpu-bound infinite loop.  When I ^C'd it, it printed:

$ dupfilefind -p -I /proc,/sys,/dev,/tmp,/var/tmp / > /tmp/dupfiles.out
^CTraceback (most recent call last):
  File "/usr/bin/dupfilefind", line 8, in <module>
    load_entry_point('dupfilefind==1.4.1', 'console_scripts', 'dupfilefind')()
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/cmdline_dupfilefind.py", line 36, in main
    return dupfilefind.dffem(len(args.verbose), args.ignore_dirs.split(','), args.hard_link_them, args.delete_them, args.min_size, args.profiles, args.dir)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 23, in dffem
    return _dffem(VERBOSITY, IGNOREDIRS, HARDLINKEM, DELETEEM, MINSIZE, PROFILES, DIRS, d1)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 86, in _dffem
    visit(os.path.abspath(dirpath), dirnames, fnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 80, in visit
    visit(idirpath, idirnames, ifnames)
  File "/usr/lib/python2.5/site-packages/dupfilefind-1.4.1-py2.5.egg/dupfilefind/dff.py", line 79, in visit
    for (idirpath, idirnames, ifnames,) in os.walk(os.path.join(dirpath, dirname)):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 304, in walk
    for x in walk(path, topdown, onerror):
  File "/usr/lib/python2.5/os.py", line 286, in walk
    names = listdir(top)
KeyboardInterrupt
exit status 1