silent failure of bin/tahoe when root partition is out of space?

[davidsarah]

Kiba on IRC reported that after following quickstart.html from a darcs checkout, bin/tahoe printed nothing for any command (including bin/tahoe --version-and-path). Our theory is that this is due to lack of space on the root partition. The output of python2 -v bin/tahoe --version-and-path included some warnings that .pyc files could not be generated (presumably due to lack of space).

[davidsarah]

Output of 'python2 -v bin/tahoe --version-and-path' on Kiba's ArchLinux? system

[davidsarah]

It's not clear whether Tahoe code is responsible, or one of our dependencies, or whether Python in general fails silently in low disk space situations.

The earliest we can print a warning without major contortions would be in src/allmydata/__init__.py​. It would be best to import as little as possible before doing the check, which we could do by moving fileutil.get_disk_stats and get_available_space into their own file (since src/allmydata/util/fileutil.py​ imports code from twisted and pycryptopp). If we apply the patch on #1274, then we can check disk space without importing pywin32 on Windows.

OTOH, without knowing precisely what fails, we don't know which path to check. We could check the partition containing allmydata.__file__, and/or the one containing sys.executable, I suppose.

[davidsarah]

Replying to davidsarah:

It's not clear whether Tahoe code is responsible, or one of our dependencies, or whether Python in general fails silently in low disk space situations.

I should have read attachment:output.txt​ more carefully. It must be Python, since the failure occurs before any Tahoe code or dependencies have been imported.

That means we can't fix this in Tahoe.

[zooko]

I guess the right thing to do is open a bug report with ​http://bugs.python.org and cross-link with this ticket...

[zooko]

Actually, if the problem is triggered by python's failed attempt to write .pyc files, then perhaps we could work-around the problem by setting the blessed PYTHONDONTWRITEBYTECODE=1 environment variable, or passing the equivalent -B option to python (only available in python >= 2.6.

Kiba: could you please run your filesystem out of space again and reproduce this problem after setting PYTHONDONTWRITEBYTECODE=1?

We should consider using that setting for other reasons -- stale .pyc files that are left behind when their .py file has been mv'ed or rm'ed are a source of confusion, and the .pyc files can contain file locations which are stale because the .pyc and .py were both moved since the .pyc was generated. Also, the presence of .pyc's might encourage someone to delete or omit the corresponding .py's, thus making it harder for people to get access to the running source code. Finally, the time spent to write .pyc files is a performance penalty for startup. I wouldn't be surprised if PYTHONDONTWRITEBYTECODE=1 doesn't improve startup performance even when .pyc files are already present, too.

[zooko]

See also #871 which is more about what happens when the disk fills up after the server has started.