tahoe cp copies a source directory's children rather than the directory itself

[zooko]

I had two entries in my local filesystem, a file named "22brain.html" and a directory named "22brain_files", which was full of other files and subdirectories. I ran {{tahoe cp 22brain* tahoe:22brain/}}}. Afterward I expected to find tahoe:22brain/22brain.html and tahoe:22brain/22brain_files/, but instead I found the contents of the local "22brain_files" directory copied directly into the tahoe:22brain directory. This breaks all the links in the "22brain.html" file and means I need to delete this directory and try again.

[zooko]

I see that the same thing happens if I run tahoe cp -r 22brain_files tahoe:22brain/.

See also #705 ("tahoe mv" unlinks the target even when it is a directory).

[davidsarah]

zooko forgot he had filed this ticket :-) and wrote in the duplicate ticket #1854 :

$ ls -al
total 1740
drwxrwxr-x 3 zooko zooko   4096 Nov 11 04:56 .
drwxrwxr-x 3 zooko zooko   4096 Nov 11 04:52 ..
-rw-rw-r-- 1 zooko zooko  80898 Aug 21 19:23 276a292eebb111e186fe22000a1c9ebd_7.jpg
-rw-rw-r-- 1 zooko zooko  96881 Nov 10 03:48 causes-of-death-2010.png
-rw-rw-r-- 1 zooko zooko  20450 Nov 11 04:52 hackers-2012-slides.html
-rw-rw-r-- 1 zooko zooko  11458 Nov 10 14:57 hackers-2012-slides.rst
-rw-rw-r-- 1 zooko zooko 948821 Nov 10 03:46 Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf
-rw-rw-r-- 1 zooko zooko 113191 Nov 10 11:51 Neal-2008-Table4.png
-rw-rw-r-- 1 zooko zooko 117367 Nov 10 10:48 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf
-rw-rw-r-- 1 zooko zooko 148251 Nov 10 10:37 Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf
-rw-rw-r-- 1 zooko zooko 219436 Nov 10 10:42 Neal-2010-Table2.png
drwxrwxr-x 4 zooko zooko   4096 Nov 10 14:04 ui
$ time tahoe cp -r * URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a:
Success: files copied

real    0m26.085s
user    0m1.003s
sys     0m0.140s
HACK zompu:~/book/gitworld/presentations/hackers-2012$ tahoe ls -l URI:DIR2-MDMF:qda3lf42dkxclwbuh5h7qwygne:obhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a:
-r--  80898 Nov 11 04:58                                                                  276a292eebb111e186fe22000a1c9ebd_7.jpg
-r-- 948821 Nov 11 04:58                                                       Murphy-2012-Deaths__Preliminary_Data_For_2010.pdf
-r-- 113191 Nov 11 04:58                                                                                    Neal-2008-Table4.png
-r-- 117367 Nov 11 04:58 Neal-2008-The_ketogenic_diet_for_the_treatment_of_childhood_epilepsy__a_randomised_controlled_trial.pdf
-r-- 148251 Nov 11 04:58                                               Neal-2010-Efficacy_Of_Dietary_Treatments_For_Epilepsy.pdf
-r-- 219436 Nov 11 04:58                                                                                    Neal-2010-Table2.png
-r--  96881 Nov 11 04:58                                                                                causes-of-death-2010.png
drwx      - Nov 11 04:56                                                                                                 default
-r--  20450 Nov 11 04:58                                                                                hackers-2012-slides.html
-r--  11458 Nov 11 04:58                                                                                 hackers-2012-slides.rst
drwx      - Nov 11 04:56                                                                                             small-white

This is wrong, causing my slides to be unable to load their CSS when viewed from Tahoe-LAFS (e.g.it looks like ​https://zooko.com/uri/URI%3ADIR2-MDMF-RO%3Appnrefnrnovjpoiv3jirjnpoim%3Aobhqprvm6hafvarzzssrawgazx6p6tgopi4fslirhelg7xqyfr6a/hackers-2012-slides.html when it should look like this: ​https://zooko.com/uri/URI:DIR2-MDMF-RO:nmrgdbrmwdta2bfzke4pfttsei:vlijvwz4kncumxkob5um45sni7btqelu4q5oltcl5nqxwjrif2iq/hackers-2012-slides.html).

The result should have been that the directory-and-file structure in the tahoe-lafs dir matched that in the local dir.

[davidsarah]

It isn't clear from the bug description or from comment:3 whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether all subdirectories will be flattened.

[markberger]

Replying to davidsarah:

It isn't clear from the bug description or from comment:3 whether it is just that a directory specified in the command line won't be created as a directory in the destination, or whether all subdirectories will be flattened.

From my testing it appears that only the top level directories are flattened:

Here is my example folder on the local drive:

tahoe-cp-test/
    readme.txt
    pics/
        rio.jpg
        seattle/
            seattle.jpg

When in the directory tahoe-cp-test/, the command tahoe cp -r * tahoe:tahoe-cp-test/ will output the following structure on the grid:

tahoe-cp-test/
    readme.txt
    rio.jpg
    seattle/
        seattle.jpg

This error appears to only occur when you are in the directory you want to upload. cp -r ~/tahoe-cp-test tahoe:tahoe-cp-test uploads the files with the correct structure.

[markberger]

This error doesn't occur when ./ is supplied instead of *.

I believe this ticket ultimately arises from tahoe's interface design and not necessarily from a bug in the code. Because tahoe cp can take multiple arguments, * is supplying both files and directories. The CLI breaks down the list of arguments into individual copy actions, so each folder and file can be viewed to have its own tahoe cp call.

In the instance of tahoe cp -r example_dir/ tahoe:example_dir/, the contents of example_dir/ should be placed into tahoe:example_dir/ without preserving the top level directory (by this I mean we want tahoe:example_dir/contents, not tahoe:example_dir/example_dir/contents).

However with this ticket, we want the opposite behavior. By supplying *, we are telling tahoe to copy multiple directories to the grid, but in this instance we want to retain the top level directory (we want tahoe to create tahoe:example_dir/example_dir/ if example_dir/ is in *).

In order to have the desired behavior outlined in this ticket, we need to distinguish between these two scenarios. I believe the current behavior makes sense, although it is counter intuitive to what users probably expect. I'm not sure how we can distinguish between these two behaviors while keeping support for multiple arguments since it seems like * is populated by the console (I could be wrong about this).

[daira]

Wildcards are indeed expanded by the Unix shell. I don't think that's the problem, though; the problem is that tahoe cp is doing the wrong thing with individual source arguments that are directories. Compare with Unix cp:

$ mkdir foo
$ mkdir bar
$ cp foo bar
cp: omitting directory `foo'
$ ls
bar  foo
$ cp -R foo bar
$ ls bar
foo

I.e. cp -R foo bar should create a copy of foo itself, not the children of foo, under bar. (It doesn't matter whether or not foo is specified with a trailing slash.)

[markberger]

Thanks for clearing that up Daira! I'll start working on this.

[markberger]

Here is the pull request: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/56

[daira]

Needs more work, removing 'review-needed'.

[daira]

Reviewed, +1. I added #2047 for a possible cleanup that can be done separately.

[daira]

Still needs docs and a NEWS entry noting the incompatible change.

[zooko]

We have inconsistent policies about NEWS at this point. I prefer the Twisted process, where each ticket that makes a change comes with a NEWS entry. Brian prefers — last time we talked about it, IIRC — for the Released Manager to read back through the closed tickets and write a NEWS entry for each one.

But the missing docs should definitely be fixed for this ticket. Removing the reviewed keyword.

[markberger]

I couldn't find any existing documentation about the expected behavior for tahoe cp -r so I added an example to the docs.

[daira]

warner will rebase this.

[Brian Warner <warner@…>]

In b88d8b0b3ea345dec3833fdef0a98326c30572f8/trunk:

Merge branch markberger/712-tahoe-cp-doesnt-copy-dir

Closes ticket:712 (trac).
Closes tahoe-lafs/tahoe-lafs#56 (github PR).

[daira]

This fix may have caused a regression: #2329