replace all use of pickles with JSON

[davidsarah]

The ​pickle format is specific to Python. Loading pickles allows arbitrary code execution (by design) and has been subject to ​memory corruption bugs.

The security exposure in Tahoe-LAFS is in practice not too bad because we only use pickles as private state, and it could be argued that a storage server has security problems anyway if an attacker can write to the filesystem under its node directory. Still, the potential for memory corruption is not nice.

We currently read and write pickles:

• in PickleStatsGatherer at src/allmydata/stats.py#L245​
• in ShareCrawler in src/allmydata/storage/crawler.py​
• in LeaseCheckingCrawler (subclass of ShareCrawler) in src/allmydata/storage/expirer.py​
• in misc/operations_helpers/cpu-watcher.tac​

If all of these uses of pickles were simply replaced with JSON, the state of crawls in progress at the time of the upgrade would be lost. This seems acceptable to me; I don't see any need to support resuming an interrupted crawl from a pickle written by a previous version.

See also #1280 and #561.

[davidsarah]

Twisted's plugin system ​also uses pickles.

[davidsarah]

Replying to davidsarah:

it could be argued that a storage server has security problems anyway if an attacker can write to the filesystem under its node directory.

Not least because they could change the .tac file, which also allows arbitrary code execution.

[warner]

PickleStatsGatherer is gone, replaced by JSONStatsGatherer, in c9047b1 (which is after tahoe-lafs-1.11.0, and should be in 1.12.0).