#814 closed defect (fixed)

v1.4.1 storage servers sending a negative number for maximum-immutable-share-size?

Reported by: zooko Owned by:
Priority: major Milestone: 1.6.0
Component: code-storage Version: 1.4.1
Keywords: easy reviewed Cc: david-sarah@…
Launchpad Bug:


I'm looking at the logs of allmydata.com user gar5 and I see this. I believe the storage servers in question are tahoe-lafs v1.4.1 storage servers operated by allmydata.com:

05:16:27.571 [3121]: connected to pfavfmv3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5412811776L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.572 [3122]: connected to 5ws3m43h, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5369373696L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.573 [3123]: connected to hndn3zdv, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5419058176L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.573 [3124]: connected to gsulyyiv, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5401027584L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
05:16:27.574 [3125]: connected to 7yun2nc3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5405422592L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}

Change History (15)

comment:1 Changed at 2009-10-15T17:58:38Z by zooko

Hm, but in the same flog incident file, other 1.4.1 servers report positive numbers:

local#118354 11:38:18.121: connected to f7342k6g, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 283901230080L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118355 11:38:18.122: connected to 5n3rnbx3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 267428404224L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118356 11:38:18.123: connected to p3677neu, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 274421722112L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118357 11:38:18.124: connected to ipxyurtk, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 270720273408L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118358 11:38:18.125: connected to yeelbkr2, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 276353047552L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}
local#118359 11:38:18.126: connected to jtobcyqt, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': 270300511232L, 'tolerates-immutab
le-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'}

comment:2 Changed at 2009-10-15T22:24:30Z by warner

I'd suspect a problem with the how-much-space-is-left calculation, since I think the maximum-immutable-share-size value that each server reports is actually min(format_support, unreserved_space_left). If the server is overfull and there's not a clamp at zero, you could get this sort of thing.

I also vaguely remember fixing this problem at some point, maybe after 1.4.1?

comment:3 Changed at 2009-10-25T18:10:22Z by zooko

Hm, it looks like this bug is still in place in trunk: storage/server.py#L209. How to test it? Maybe monkeypatch in a fake os.statvfs()?

And how do clients currently behave when storage servers tell them that they have negative "maximum-immutable-share-size"? immutable/upload.py#L187 They just compare whether it is >= the space that they need, so they work fine when it reports a negative number. However, it seems better if they would report 0 instead of a negative number.

comment:4 Changed at 2009-10-27T02:32:12Z by zooko

  • Keywords easy added

comment:5 Changed at 2009-10-27T02:50:11Z by zooko

If you like this ticket you might also like #637 (support "keep this much disk space free" on Windows as well as other platforms).

comment:6 Changed at 2009-10-27T04:47:32Z by davidsarah

  • Cc david-sarah@… added
  • Milestone changed from undecided to 1.6.0
  • Owner set to davidsarah
  • Status changed from new to assigned

The current code for get_available_space in storage/server.py subtracts the 'reserved_space' value from the free space available to non-root users, without clamping to zero. I had already spotted that and changed it to clamp to zero in the (unfinished) patch for #637.

It is also possible for the maximum-immutable-share-size to be 264 if os.statvfs is not supported (or if neither os.statvfs nor win32api.GetDiskFreeSpaceEx? is supported after #637 is fixed). Will 264 be JSON-encoded properly?

comment:7 follow-up: Changed at 2009-10-27T05:05:41Z by davidsarah

Just to confirm that we have the right explanation, what was the reserved_space set to on gar5?

comment:8 in reply to: ↑ 7 Changed at 2009-10-27T05:07:27Z by davidsarah

Replying to davidsarah:

Just to confirm that we have the right explanation, what was the reserved_space set to on gar5?

I mean, on the servers listed in the log. (I misread gar5 as a server name.)

comment:9 Changed at 2009-10-27T05:24:35Z by zooko

Well for example this one: 05:16:27.571 [3121]: connected to pfavfmv3, version {'http://allmydata.org/tahoe/protocols/storage/v1': {'maximum-immutable-share-size': -5412811776L, 'tolerates-immutable-read-overrun': True, 'delete-mutable-shares-with-zero-length-writev': True}, 'application-version': 'tahoe-server/1.4.1'} Has reserved_space = 10G in its tahoe.cfg file.

comment:10 Changed at 2009-10-28T03:17:05Z by davidsarah

That makes sense (if reserved_space had been less than 5412811776 bytes, i.e. ~5.4 GB, it would have excluded this explanation).

comment:11 Changed at 2009-11-06T07:47:14Z by davidsarah

This should be fixed by the latest patch attached to #637.

comment:12 Changed at 2009-11-07T04:06:59Z by zooko

  • Keywords review added

comment:13 Changed at 2009-11-08T03:33:48Z by davidsarah

  • Owner davidsarah deleted
  • Status changed from assigned to new

comment:14 Changed at 2009-11-21T06:06:58Z by davidsarah

  • Keywords reviewed added; review removed

This was reviewed with #637.

comment:15 Changed at 2009-11-30T20:05:35Z by warner

  • Resolution set to fixed
  • Status changed from new to closed

This has been subsumed by #637, which was just committed, so this can be closed.

Note: See TracTickets for help on using tickets.