flogtool scalability/performance

[zooko]

I'm trying to diagnose problems on the current tahoe production grid, and here are some observations:

1. The logs.pickle file is 2 GB in size, and can't be web-viewed as flogtool web-viewer logs.pickle eventually dies after running out of RAM.

2. 7zipping logs.pickle takes 15 minutes and results in a logs.pickle.7z of 115 MB. (I chose 7zip because I didn't want to use up too much CPU on the tahoe production grid server, and 7zip is one of the most CPU-efficient among the compressors that also give good ratio on a large file.)

3. flogtool dump logs.pickle &> logs.dump.txt on my Macbook Pro takes 30 minutes and results in a logs.dump.txt of size 415 MB.

So for even a little-used and small grid like the current tahoe production grid, flogtool has some scalability/performance problems.

By the way, in order to estimate how much redundancy there is in the resulting text log file compared to the pickle, I 7zipped the logs.dump.txt, which resulted in a logs.dump.txt.7z of size 40 MB.