Uploading files with http put

Wed Nov 27 01:20:27 UTC 2019

Never mind the email below. I found this explanation for what’s going on in the Tahoe REST API documentation:

https://tahoe-lafs.readthedocs.io/en/tahoe-lafs-1.12.1/frontends/webapi.html#writing-uploading-a-file

I think I have  a good enough understanding of how this works so that I can make asynchronous uploads work in my Nextcloud plugin with any changes to Tahoe-LAFS.

                Bruce T

From: Bruce Thompson <brucet.cisco at gmail.com>
Date: Tuesday, November 26, 2019 at 5:52 PM
To: Jean-Paul Calderone <jean-paul+tahoe-dev at leastauthority.com>
Cc: "tahoe-dev at tahoe-lafs.org" <tahoe-dev at tahoe-lafs.org>
Subject: Re: Uploading files with http put

Jean-Paul et al,

I went through the Tahoe client code today to determine exactly what Tahoe does when you do an upload of a single file to a Tahoe directory (tahoe cp localfile alias:path)”.

I do not understand how the Tahoe client creates the URL that it does the HTTP Put to. I tried a test with the following command:

“tahoe cp /tmp/bar node76:foo”

In this command, the target of the cp is a Tahoe directory named “foo” which is contained in alias “node76:”. I was expecting that the Tahoe cp command would end up creating a URL for the http put by combining the writecap URI of the “foo” directory with the filename “bar”. However, this does not seem to be the case.

Here is the call path from phyCharm where the HTTP put is actually sent:

Here are the parameters to do_http from pycharm:

There is no Tahoe write-cap URI of the “foo” directory in the path of this call.

How does this work????

                Bruce T

From: Bruce Thompson <brucet.cisco at gmail.com>
Date: Tuesday, November 26, 2019 at 12:07 PM
To: Jean-Paul Calderone <jean-paul+tahoe-dev at leastauthority.com>
Cc: "tahoe-dev at tahoe-lafs.org" <tahoe-dev at tahoe-lafs.org>
Subject: Re: Uploading files with http put

Jean-Paul,

Does that mean you want to be able to do an HTTP PUT to start an upload and then poll the upload status with HTTP GET?

I checked the Nextcloud code to see how it handles HTTP Put. It looks like Nextcloud files_external applications that use HTTP (like the Amazon S3 plugin) use a combination of the icewinds php stream wrapper and guzzle http to monitor a file upload via http put. Guzzle http is a php package that support asynchronous http operation. The S3 plugin uses a combination of the icewinds php stream wrapper and guzzle http to monitor the number of bytes transferred from a local file to the web server via HTTP put.

Using the method I described above, all I would need is a way to do a synchronous HTTP put of a local file to a URL representing an immutable file resource on the Tahoe-LAFS server. 

Note that this approach is extremely complicated in php but is likely doable.

If not, what does your ideal interaction between Nextcloud and Tahoe-LAFS look like?

The simplest approach would be to make progress functionality available via the tahoe CLI.

One implementation approach would be to have the Tahoe “cp” CLI send progress indications to stdout as it is copying. The progress indications could be in terms of bytes transferred, % complete, etc. If this type of functionality was available, then I could simple use this php code snippet to monitor progress in PHP:

https://stackoverflow.com/questions/45953/php-execute-a-background-process#45966

                Bruce T

From: Jean-Paul Calderone <jean-paul+tahoe-dev at leastauthority.com>
Date: Tuesday, November 26, 2019 at 8:26 AM
To: Bruce Thompson <brucet.cisco at gmail.com>
Cc: "tahoe-dev at tahoe-lafs.org" <tahoe-dev at tahoe-lafs.org>
Subject: Re: Uploading files with http put

On Tue, Nov 26, 2019 at 9:23 AM brucet <brucet.cisco at gmail.com> wrote:

Jean-Paul,

Can you elaborate on what you mean by "asynchronous"?

PHP does not directly support asynchronous operations so I used the term loosely. 

Nextcloud includes a stream package that does callbacks on php streams. This is a link to the icewinds php stream wrapper which is included in the nextcloud code:

https://github.com/icewind1991/Streams

This library supports callbacks to provide updated status in the web UI when performing reads / writes on a php stream. This functionality works with http put / get. I am using this functionality for file downloads from Tahoe-LAFS by using http get. I would like to do the same for file uploads to Tahoe-LAFS by using http put.

Does that mean you want to be able to do an HTTP PUT to start an upload and then poll the upload status with HTTP GET?

If not, what does your ideal interaction between Nextcloud and Tahoe-LAFS look like?

                Bruce T

From: Jean-Paul Calderone <jean-paul+tahoe-dev at leastauthority.com>
Date: Monday, November 25, 2019 at 6:38 PM
To: Bruce Thompson <brucet.cisco at gmail.com>
Cc: "tahoe-dev at tahoe-lafs.org" <tahoe-dev at tahoe-lafs.org>
Subject: Re: Uploading files with http put

Can you elaborate on what you mean by "asynchronous"?

On Mon, Nov 25, 2019 at 7:28 PM brucet <brucet.cisco at gmail.com> wrote:

Never saw a response to this. Does anyone have any pointer of where to look for an answer to my question below?

Thanks,

                Bruce T

From: Bruce Thompson <brucet.cisco at gmail.com>
Date: Friday, November 22, 2019 at 9:08 AM
To: "tahoe-dev at tahoe-lafs.org" <tahoe-dev at tahoe-lafs.org>
Subject: Uploading files with http put

I am writing a Tahoe-LAFS plugin for Nextcloud and close to having it completed. The plugin puts Tahoe-LAFS on even footing with other cloud storage systems that have been integrated with Nextcloud such as: Amazon S3 and Swift.

The problem I am having is I need to support asynchronous uploading of local files to Tahoe-LAFS. I have not found a way to do this using the “tahoe cp” command. However, if I use an http put to perform the upload then I can support asynchronous uploads in Nextcloud.

I am using the “tahoe” command from the plugin so I can use any tahoe command that completes in a relatively short period of time. Is there a simple way for me to create a placeholder tahoe URI that I can use to perform an http put with to perform an upload?

Thanks in advance,

                Bruce T

_______________________________________________
tahoe-dev mailing list
tahoe-dev at tahoe-lafs.org
https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20191126/f54994cd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 286805 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20191126/f54994cd/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 112389 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20191126/f54994cd/attachment-0003.png>