[tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed

[tahoe-lafs]

#329: dirnodes could cache encrypted/serialized entries for speed
-------------------------------+--------------------------------------------
     Reporter:  warner         |       Owner:  kevan    
         Type:  enhancement    |      Status:  closed   
     Priority:  minor          |   Milestone:  undecided
    Component:  code-dirnodes  |     Version:  0.8.0    
   Resolution:  fixed          |    Keywords:  dirnode  
Launchpad_bug:                 |  
-------------------------------+--------------------------------------------

Comment(by zooko):

 Now that I've applied your actual optimization patch, time to unpack and
 repack a 4096-entry directory dropped substantially.  However, time to
 pack and time to unpack also changed, which suggests that my benchmark
 script is accidentally using your cache, or that extraneous factors on my
 system are screwing up the results or something.

 Before {{{optimizations.txt}}}:

 {{{
 benchmarking <function unpack at 0x1b78320>
 N:      64, time: best:    0.08,   2th-best:    0.09, ave:    0.10,   2th-
 worst:    0.10, worst:    0.15 (of      5), reps/s:      9, ave rate:
 632
 N:     256, time: best:    0.35,   2th-best:    0.37, ave:    0.38,   2th-
 worst:    0.39, worst:    0.40 (of      5), reps/s:      2, ave rate:
 675
 N:    1024, time: best:    1.40,   2th-best:    1.42, ave:    1.63,   2th-
 worst:    1.74, worst:    1.88 (of      5), reps/s:      0, ave rate:
 629
 N:    4096, time: best:    5.29,   2th-best:    5.90, ave:    6.16,   2th-
 worst:    6.36, worst:    7.15 (of      5), reps/s:      0, ave rate:
 665
 benchmarking <function pack at 0x1b782a8>
 N:      64, time: best:    0.09,   2th-best:    0.09, ave:    0.09,   2th-
 worst:    0.09, worst:    0.09 (of      5), reps/s:     11, ave rate:
 705
 N:     256, time: best:    0.36,   2th-best:    0.36, ave:    0.39,   2th-
 worst:    0.37, worst:    0.51 (of      5), reps/s:      2, ave rate:
 650
 N:    1024, time: best:    1.38,   2th-best:    1.43, ave:    1.51,   2th-
 worst:    1.51, worst:    1.80 (of      5), reps/s:      0, ave rate:
 676
 N:    4096, time: best:    5.22,   2th-best:    5.77, ave:    6.08,   2th-
 worst:    6.33, worst:    6.79 (of      5), reps/s:      0, ave rate:
 674
 benchmarking <function unpack_and_repack at 0x1b78398>
 N:      64, time: best:    0.10,   2th-best:    0.10, ave:    0.14,   2th-
 worst:    0.18, worst:    0.19 (of      5), reps/s:      7, ave rate:
 473
 N:     256, time: best:    0.36,   2th-best:    0.37, ave:    0.39,   2th-
 worst:    0.40, worst:    0.42 (of      5), reps/s:      2, ave rate:
 664
 N:    1024, time: best:    1.41,   2th-best:    1.42, ave:    1.46,   2th-
 worst:    1.48, worst:    1.52 (of      5), reps/s:      0, ave rate:
 702
 N:    4096, time: best:    5.22,   2th-best:    5.58, ave:    5.81,   2th-
 worst:    6.09, worst:    6.20 (of      5), reps/s:      0, ave rate:
 705
 }}}

 after {{{optimizations.txt}}}:

 {{{
 benchmarking <function unpack at 0x1b78398>
 N:      64, time: best:    0.05,   2th-best:    0.05, ave:    0.06,   2th-
 worst:    0.06, worst:    0.07 (of      5), reps/s:     17, ave rate:
 1091
 N:     256, time: best:    0.21,   2th-best:    0.21, ave:    0.23,   2th-
 worst:    0.24, worst:    0.27 (of      5), reps/s:      4, ave rate:
 1093
 N:    1024, time: best:    0.83,   2th-best:    0.84, ave:    0.88,   2th-
 worst:    0.87, worst:    1.00 (of      5), reps/s:      1, ave rate:
 1170
 N:    4096, time: best:    3.29,   2th-best:    3.35, ave:    3.54,   2th-
 worst:    3.58, worst:    4.09 (of      5), reps/s:      0, ave rate:
 1156
 benchmarking <function pack at 0x1b78320>
 N:      64, time: best:    0.06,   2th-best:    0.06, ave:    0.06,   2th-
 worst:    0.06, worst:    0.06 (of      5), reps/s:     17, ave rate:
 1106
 N:     256, time: best:    0.22,   2th-best:    0.22, ave:    0.22,   2th-
 worst:    0.22, worst:    0.23 (of      5), reps/s:      4, ave rate:
 1158
 N:    1024, time: best:    0.85,   2th-best:    0.85, ave:    0.86,   2th-
 worst:    0.87, worst:    0.88 (of      5), reps/s:      1, ave rate:
 1188
 N:    4096, time: best:    3.27,   2th-best:    3.28, ave:    3.36,   2th-
 worst:    3.38, worst:    3.60 (of      5), reps/s:      0, ave rate:
 1218
 benchmarking <function unpack_and_repack at 0x1b78410>
 N:      64, time: best:    0.06,   2th-best:    0.06, ave:    0.07,   2th-
 worst:    0.09, worst:    0.10 (of      5), reps/s:     13, ave rate:
 881
 N:     256, time: best:    0.22,   2th-best:    0.22, ave:    0.28,   2th-
 worst:    0.26, worst:    0.46 (of      5), reps/s:      3, ave rate:
 929
 N:    1024, time: best:    0.85,   2th-best:    0.90, ave:    1.00,   2th-
 worst:    1.02, worst:    1.34 (of      5), reps/s:      0, ave rate:
 1022
 N:    4096, time: best:    3.25,   2th-best:    3.33, ave:    3.49,   2th-
 worst:    3.67, worst:    3.74 (of      5), reps/s:      0, ave rate:
 1172
 }}}

 I've applied your patches plus one from me to fix a conflict between one
 of my optimizations and one of yours.

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/329#comment:26>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid