[tahoe-lafs-trac-stream] [tahoe-lafs] #1124: upload is unhappy even though the shares are already distributed

[tahoe-lafs]

#1124: upload is unhappy even though the shares are already distributed
-------------------------+-------------------------------------------------
     Reporter:  zooko    |      Owner:
         Type:  defect   |     Status:  new
     Priority:  major    |  Milestone:  1.11.0
    Component:  code-    |    Version:  1.7.0
  peerselection          |   Keywords:  upload availability reliability
   Resolution:           |  unfinished-business servers-of-happiness
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by markberger):

 * cc: mjberger@… (added)


Old description:

> Here's a test case that I added to
> [source:src/allmydata/test/test_upload.py]:
> {{{
>     def test_problem_layout_0123_03_2_1(self):
>         # Zooko stumbled on this case when investigating #1118.
>         self.basedir = self.mktemp()
>         d = self._setup_and_upload(k=2, n=4)
>
>         # server 0: shares 0, 1, 2, 3
>         # server 1: shares 0, 3
>         # server 2: share 1
>         # server 3: share 2
>         # With this layout, an upload should just be satisfied that the
> current distribution is good enough, right?
>         def _setup(ign):
>             self._add_server_with_share(server_number=0,
> share_number=None)
>             self._add_server_with_share(server_number=1, share_number=0)
>             self._add_server_with_share(server_number=2, share_number=1)
>             self._add_server_with_share(server_number=3, share_number=2)
>             # Copy shares
>             self._copy_share_to_server(3, 1)
>             client = self.g.clients[0]
>             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
>             return client
>
>         d.addCallback(_setup)
>         d.addCallback(lambda client:
>             client.upload(upload.Data("data" * 10000, convergence="")))
>         return d
> }}}
> When I run it, it fails like this:
> {{{
> [ERROR]:
> allmydata.test.test_upload.EncodingParameters.test_problem_layout_ticket_1118
>
> Traceback (most recent call last):
>   File "/Users/zooko/playground/tahoe-
> lafs/trunk/src/allmydata/immutable/upload.py", line 510, in _got_response
>     return self._loop()
>   File "/Users/zooko/playground/tahoe-
> lafs/trunk/src/allmydata/immutable/upload.py", line 363, in _loop
>     self._get_progress_message()))
> allmydata.interfaces.UploadUnhappinessError: shares could be placed on
> only 3 server(s) such that any 2 of them have enough shares to recover
> the file, but we were asked to place shares on at least 4 such servers.
> (placed all 4 shares, want to place shares on at least 4 servers such
> that any 2 of them have enough shares to recover the file, sent 4 queries
> to 4 peers, 4 queries placed some shares, 0 placed none (of which 0
> placed none due to the server being full and 0 placed none due to an
> error))
> }}}
> Why does the upload not succeed?
> I added debugprints and here are some of them:
> {{{
> 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
> xgru5adv: alreadygot=(0, 1, 2, 3), allocated=()
> 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
> ysbz4st7: alreadygot=(0, 3), allocated=()
> 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
> b3llgpww: alreadygot=(2,), allocated=(1,)
> 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
> rvsry4kn: alreadygot=(1,), allocated=(0,)
> 2010-07-18 09:54:15-0600 [-] server selection unsuccessful for
> <Tahoe2PeerSelector for upload t2uvf>: shares could be placed on only 3
> server(s) such that any 2 of them have enough shares to recover the file,
> but we were asked to place shares on at least 4 such servers.: placed all
> 4 shares, want to place shares on at least 4 servers such that any 2 of
> them have enough shares to recover the file, sent 4 queries to 4 peers, 4
> queries placed some shares, 0 placed none (of which 0 placed none due to
> the server being full and 0 placed none due to an error): sh0: rvsry4kn,
> sh1: b3llgpww+rvsry4kn, sh2: b3llgpww, sh3: ysbz4st7+xgru5adv
> }}}

New description:

 Here's a test case that I added to
 [source:src/allmydata/test/test_upload.py]:
 {{{
     def test_problem_layout_0123_03_2_1(self):
         # Zooko stumbled on this case when investigating #1118.
         self.basedir = self.mktemp()
         d = self._setup_and_upload(k=2, n=4)

         # server 0: shares 0, 1, 2, 3
         # server 1: shares 0, 3
         # server 2: share 1
         # server 3: share 2
         # With this layout, an upload should just be satisfied that the
 current distribution is good enough, right?
         def _setup(ign):
             self._add_server_with_share(server_number=0,
 share_number=None)
             self._add_server_with_share(server_number=1, share_number=0)
             self._add_server_with_share(server_number=2, share_number=1)
             self._add_server_with_share(server_number=3, share_number=2)
             # Copy shares
             self._copy_share_to_server(3, 1)
             client = self.g.clients[0]
             client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
             return client

         d.addCallback(_setup)
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
         return d
 }}}
 When I run it, it fails like this:
 {{{
 [ERROR]:
 allmydata.test.test_upload.EncodingParameters.test_problem_layout_ticket_1118

 Traceback (most recent call last):
   File "/Users/zooko/playground/tahoe-
 lafs/trunk/src/allmydata/immutable/upload.py", line 510, in _got_response
     return self._loop()
   File "/Users/zooko/playground/tahoe-
 lafs/trunk/src/allmydata/immutable/upload.py", line 363, in _loop
     self._get_progress_message()))
 allmydata.interfaces.UploadUnhappinessError: shares could be placed on
 only 3 server(s) such that any 2 of them have enough shares to recover the
 file, but we were asked to place shares on at least 4 such servers.
 (placed all 4 shares, want to place shares on at least 4 servers such that
 any 2 of them have enough shares to recover the file, sent 4 queries to 4
 peers, 4 queries placed some shares, 0 placed none (of which 0 placed none
 due to the server being full and 0 placed none due to an error))
 }}}
 Why does the upload not succeed?
 I added debugprints and here are some of them:
 {{{
 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
 xgru5adv: alreadygot=(0, 1, 2, 3), allocated=()
 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
 ysbz4st7: alreadygot=(0, 3), allocated=()
 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
 b3llgpww: alreadygot=(2,), allocated=(1,)
 2010-07-18 09:54:15-0600 [-] response to allocate_buckets() from peer
 rvsry4kn: alreadygot=(1,), allocated=(0,)
 2010-07-18 09:54:15-0600 [-] server selection unsuccessful for
 <Tahoe2PeerSelector for upload t2uvf>: shares could be placed on only 3
 server(s) such that any 2 of them have enough shares to recover the file,
 but we were asked to place shares on at least 4 such servers.: placed all
 4 shares, want to place shares on at least 4 servers such that any 2 of
 them have enough shares to recover the file, sent 4 queries to 4 peers, 4
 queries placed some shares, 0 placed none (of which 0 placed none due to
 the server being full and 0 placed none due to an error): sh0: rvsry4kn,
 sh1: b3llgpww+rvsry4kn, sh2: b3llgpww, sh3: ysbz4st7+xgru5adv
 }}}

--

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1124#comment:15>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage