"tahoe cp $DIRCAP/$PATH $LOCAL" raises AttributeError

[zooko]

tahoe cp --help suggests that I try:

You can also use a dircap as either FROM or TO target:

tahoe cp
URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html
./ # copy Zooko's wiki page to a local file

But when I try it, I get an exception:

$ time ~/playground/allmydata/tahoe/trunk-hashedformat/bin/tahoe cp -r --node-url http://testgrid.allmydata.org:3567 URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html foo
Traceback (most recent call last):
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/support/bin/tahoe", line 8, in <module>
    load_entry_point('allmydata-tahoe==1.4.1-r3989', 'console_scripts', 'tahoe')()
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/runner.py", line 91, in run
    rc = runner(sys.argv[1:])
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/runner.py", line 78, in runner
    rc = cli.dispatch[command](so)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/cli.py", line 429, in cp
    rc = tahoe_cp.copy(options)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 759, in copy
    return Copier().do_copy(options)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 444, in do_copy
    self.try_copy()
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 501, in try_copy
    return self.copy_to_directory(sources, target)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 661, in copy_to_directory
    self.copy_files_to_target(self.targetmap[target], target)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 692, in copy_files_to_target
    self.copy_file_into(source, name, target)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 737, in copy_file_into
    target.put_file(name, f)
  File "/data2/homezooko/playground/allmydata/tahoe/trunk-hashedformat/src/allmydata/scripts/tahoe_cp.py", line 163, in put_file
    pathname = os.path.join(self.pathname, name)
  File "/usr/lib/python2.5/posixpath.py", line 60, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

[kevan]

Testing:

copying a filecap to a local directory

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:CHK:lmwmniv6kzf2amxuoikl6ncmqi:42izq3rtixijq7bs5s3pjcyv4st2hgzuzfqxfjaifbb4mo62adyq:3:10:83 adir
Traceback (most recent call last):
  File "/Users/kacarstensen/Documents/code/tahoe-761/support/bin/tahoe", line 8, in <module>
    load_entry_point('allmydata-tahoe==1.5.0-r4054', 'console_scripts', 'tahoe')()
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/runner.py", line 91, in run
    rc = runner(sys.argv[1:])
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/runner.py", line 78, in runner
    rc = cli.dispatch[command](so)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/cli.py", line 436, in cp
    rc = tahoe_cp.copy(options)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 759, in copy
    return Copier().do_copy(options)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 444, in do_copy
    self.try_copy()
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 501, in try_copy
    return self.copy_to_directory(sources, target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 661, in copy_to_directory
    self.copy_files_to_target(self.targetmap[target], target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 692, in copy_files_to_target
    self.copy_file_into(source, name, target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 737, in copy_file_into
    target.put_file(name, f)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 163, in put_file
    pathname = os.path.join(self.pathname, name)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/posixpath.py", line 65, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

copying a file by means of a dircap and a path to a local directory

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html adir
Traceback (most recent call last):
  File "/Users/kacarstensen/Documents/code/tahoe-761/support/bin/tahoe", line 8, in <module>
    load_entry_point('allmydata-tahoe==1.5.0-r4054', 'console_scripts', 'tahoe')()
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/runner.py", line 91, in run
    rc = runner(sys.argv[1:])
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/runner.py", line 78, in runner
    rc = cli.dispatch[command](so)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/cli.py", line 436, in cp
    rc = tahoe_cp.copy(options)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 759, in copy
    return Copier().do_copy(options)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 444, in do_copy
    self.try_copy()
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 501, in try_copy
    return self.copy_to_directory(sources, target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 661, in copy_to_directory
    self.copy_files_to_target(self.targetmap[target], target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 692, in copy_files_to_target
    self.copy_file_into(source, name, target)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 737, in copy_file_into
    target.put_file(name, f)
  File "/Users/kacarstensen/Documents/code/tahoe-761/src/allmydata/scripts/tahoe_cp.py", line 163, in put_file
    pathname = os.path.join(self.pathname, name)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/posixpath.py", line 65, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

copying a filecap to a local file that doesn't exist

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:CHK:lmwmniv6kzf2amxuoikl6ncmqi:42izq3rtixijq7bs5s3pjcyv4st2hgzuzfqxfjaifbb4mo62adyq:3:10:83 newfile
Success: file copied

copying a file by means of a dircap and a path to a local file that doesn't exist

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html wiki.html
Success: file copied

copying a filecap to a local file that does exist

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:CHK:lmwmniv6kzf2amxuoikl6ncmqi:42izq3rtixijq7bs5s3pjcyv4st2hgzuzfqxfjaifbb4mo62adyq:3:10:83 wiki.html
Success: file copied

copying a file by means of a dircap and a path to a local file that does exist.

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html newfile
Success: file copied

copying a dircap to a local directory

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp --recursive URI:DIR2:f7q633wsw7x2fkqf3vkrdl35je:u3bcw323sqhev5lhem5qkgcwrelp63rsroh7qnzz5haorhlmqdxq thedir
Success: files copied

(it copies the contents of the directory on the grid to thedir)

copying a dircap to a local directory that doesn't exist

macbook-3:tahoe-761 kacarstensen$ ./bin/tahoe cp --recursive URI:DIR2:f7q633wsw7x2fkqf3vkrdl35je:u3bcw323sqhev5lhem5qkgcwrelp63rsroh7qnzz5haorhlmqdxq newdir
Success: files copied

(don't worry -- the files and directories I use for testing were, when not taken from tahoe cp --help, generated for the purposes of the testing, so I'm not leaking anything by not censoring caps)

From what I gather, those two errors are happening because the code in tahoe_cp.py​ isn't attempting to deduce a destination filename within the directory to use when copying. It doesn't have a lot to go on in the first case (since the filecap doesn't know what its name might be), but it seems like it should be able to infer that wiki.html should be the destination name in the second case, and continue. I have a patch that implements this behavior (complaining in the first failure case, working in the second), but I want to test it more thoroughly before I upload it.

[kevan]

These changes passed all of the tests that I could think to throw at them. Comments?

[warner]

One comment about the original issue that zooko described: the help text suggested running "cp DIRCAP/filename ./" (note that the target is "./"), but the failing example actually ran "cp -r DIRCAP/filename filename2" (note the -r and "filename2" target). The command in the help text works correctly.

Kevan: thanks for the comprehensive testing! If I might compress the cases down to a brief list, I think it looks like this:

• cp -r DIRCAP/filename1 filename2 : EXCEPTION (zooko's example)
• cp FILECAP dirname : EXCEPTION
• cp DIRCAP/filename dirname : EXCEPTION
• cp FILECAP filename : OK
• cp DIRCAP/filename filename : OK
• cp FILECAP filename : OK (overwrite)
• cp DIRCAP/filename filename : OK (overwrite)
• cp -r FILECAP dirname : OK
• cp -r DIRCAP dirname : OK

The behavior of plain old /bin/cp, with or without -r, is pretty ugly, and there are a lot of variables: source has-or-does-not-have trailing slash, dest trailing slash, dest not-exists/is-file/is-dir, -r-or-not. Other programs that try to look kind of like /bin/cp (like rsync and scp) have slightly different semantics.

I like your conclusions, that the first case ("cp FILECAP dirname") should complain cleanly, and the second case ("cp DIRCAP/filename dirname") should result in dirname/filename being created.

I don't know what "cp -r FILECAP dirname" should do (it seems plausible that -r should assert that the source argument is a directory), but I think the current behavior is acceptable. Given that, it seems like zooko's "cp -r DIRCAP/filename1 filename2" ought to behave just like the non-recursive "cp DIRCAP/filename1 filename2".

Anyways, those are my feelings. I'll go review your patch now.

[warner]

The patch looks great. I've applied it in 74974b27fe044438, along with an additional test (of the cp -r DIRCAP/filename1 filename2 case), and some cleanups to have it only put files in the per-test subdirectory (instead of directly under _trial_temp/).

thanks!