[tahoe-lafs-trac-stream] [tahoe-lafs] #822: Web API should use a more reliable, out-of-band means of reporting errors (such as a server connection being lost) during a download

[tahoe-lafs]

#822: Web API should use a more reliable, out-of-band means of reporting errors
(such as a server connection being lost) during a download
-------------------------------+-------------------------------------------
     Reporter:  davidsarah     |      Owner:
         Type:  defect         |     Status:  new
     Priority:  major          |  Milestone:  soon
    Component:  code-          |    Version:  1.5.0
  frontend-web                 |   Keywords:  integrity error http download
   Resolution:                 |
Launchpad Bug:                 |
-------------------------------+-------------------------------------------
Changes (by zooko):

 * priority:  critical => major


Old description:

> The discussion of bug #698 (which turned out to be a Firefox bug) turned
> up a potential integrity problem that can occur if a server connection is
> lost in the middle of a download via the WUI:
>
> http://allmydata.org/trac/tahoe/ticket/698#comment:1
>
> > The first thing that comes to mind is that a server connection could
> have been lost in the middle of the download (in this case, after we've
> retrieved the UEB and some of the hashes, but before we've retrieved the
> first data block). The web server has to commit to success (200) or
> failure (404 or 500 or something) before it starts sending any of the
> plaintext, but it doesn't want to store the entire file either. So it
> bases the HTTP response code upon the initial availability of k servers,
> and hopes they'll stick around for the whole download.
>
> > When we get a "late failure" (i.e. one of the servers disconnects in
> the middle), the webapi doesn't have a lot of choices. At the moment, it
> emits a brief error message (attached to whatever partial content has
> already been written out), then drops the HTTP connection, and hopes that
> the client is observant enough to notice that the number of received
> bytes does not match the previously-sent Content-Length header, and then
> announce an error on the client side.
>
> > If the application doing the fetch (perhaps the browser, perhaps
> tiddywiki itself?) doesn't strictly check the Content-Length header, then
> it could get partial content without an error message.
>
> > There are two directions to fix this:
>
> >    * change the webapi to use "Chunked Encoding", basically delivering
> data one segment at a time, possibly giving the server a chance to emit
> an error header in between segments: this would let us respond better to
> these errors
>
> >    * fix the other download-should-be-better tickets (#193, #287) to
> tolerate lost servers better, which might reduce the rate at which these
> errors occur
>
> As pointed out in http://allmydata.org/pipermail/tahoe-
> dev/2009-May/001724.html , it is possible that the length so far plus the
> length of the error message, coincidentally equals the expected file
> length. So even for a web client that diligently checks the Content-
> Length, there might not be enough information to detect an error. An
> attacker might try to force this situation (I don't know what their
> chance of success would be, but probably much higher than trying to
> attack the crypto).
>
> In any case, the WUI is currently using in-band error reporting, which is
> problematic because the error message will be treated as data of whatever
> format the client thinks the content has. This is an integrity issue
> because the download from the gateway to the client has no cryptographic
> integrity checking.
>
> To close this bug, find and implement some way to make typical web
> clients reliably report an error when a download fails part-way through.
> Alternatively, prove that it isn't possible, and document this as an
> inherent limitation of the WUI.

New description:

 The discussion of bug #698 (which turned out to be a Firefox bug) turned
 up a potential integrity problem that can occur if a server connection is
 lost in the middle of a download via the WUI:

 http://allmydata.org/trac/tahoe/ticket/698#comment:1

 > The first thing that comes to mind is that a server connection could
 have been lost in the middle of the download (in this case, after we've
 retrieved the UEB and some of the hashes, but before we've retrieved the
 first data block). The web server has to commit to success (200) or
 failure (404 or 500 or something) before it starts sending any of the
 plaintext, but it doesn't want to store the entire file either. So it
 bases the HTTP response code upon the initial availability of k servers,
 and hopes they'll stick around for the whole download.

 > When we get a "late failure" (i.e. one of the servers disconnects in the
 middle), the webapi doesn't have a lot of choices. At the moment, it emits
 a brief error message (attached to whatever partial content has already
 been written out), then drops the HTTP connection, and hopes that the
 client is observant enough to notice that the number of received bytes
 does not match the previously-sent Content-Length header, and then
 announce an error on the client side.

 > If the application doing the fetch (perhaps the browser, perhaps
 tiddywiki itself?) doesn't strictly check the Content-Length header, then
 it could get partial content without an error message.

 > There are two directions to fix this:

 >    * change the webapi to use "Chunked Encoding", basically delivering
 data one segment at a time, possibly giving the server a chance to emit an
 error header in between segments: this would let us respond better to
 these errors

 >    * fix the other download-should-be-better tickets (#193, #287) to
 tolerate lost servers better, which might reduce the rate at which these
 errors occur

 As pointed out in http://allmydata.org/pipermail/tahoe-
 dev/2009-May/001724.html , it is possible that the length so far plus the
 length of the error message, coincidentally equals the expected file
 length. So even for a web client that diligently checks the Content-
 Length, there might not be enough information to detect an error. An
 attacker might try to force this situation (I don't know what their chance
 of success would be, but probably much higher than trying to attack the
 crypto).

 In any case, the WUI is currently using in-band error reporting, which is
 problematic because the error message will be treated as data of whatever
 format the client thinks the content has. This is an integrity issue
 because the download from the gateway to the client has no cryptographic
 integrity checking.

 To close this bug, find and implement some way to make typical web clients
 reliably report an error when a download fails part-way through.
 Alternatively, prove that it isn't possible, and document this as an
 inherent limitation of the WUI.

--

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/822#comment:10>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage