| 121 | | === storage index count === |
| 122 | | |
| 123 | | ext3 (on tahoebs1) refuses to create more than 32000 subdirectories in a |
| 124 | | single parent directory. In 0.5.1, this appears as a limit on the number of |
| 125 | | buckets (one per storage index) that any !StorageServer can hold. A simple |
| 126 | | nested directory structure will work around this.. the following code would |
| 127 | | let us manage 33.5G shares (see #150). |
| 128 | | |
| 129 | | {{{ |
| 130 | | from idlib import b2a |
| 131 | | os.path.join(b2a(si[:2]), b2a(si[2:4]), b2a(si)) |
| 132 | | }}} |
| 133 | | |
| 134 | | This limitation is independent of problems of memory use and lookup speed. |
| 135 | | Once the number of buckets is large, the filesystem may take a long time (and |
| 136 | | multiple disk seeks) to determine if a bucket is present or not. The |
| 137 | | provisioning page suggests how frequently these lookups will take place, and |
| 138 | | we can compare this against the time each one will take to see if we can keep |
| 139 | | up or not. If and when necessary, we'll move to a more sophisticated storage |
| 140 | | server design (perhaps with a database to locate shares). |
| 141 | | |
| 142 | | I was unable to measure a consistent slowdown resulting from having 30000 |
| 143 | | buckets in a single storage server. |
| 144 | | |
