#2290 closed defect (fixed)

allmydata.test.test_util.FileUtil.test_disk_stats sometimes fails on Travis

Reported by: daira Owned by: Brian Warner <warner@…>
Priority: normal Milestone: undecided
Component: code Version: 1.10.0
Keywords: tests travis Cc:
Launchpad Bug:

Description

https://travis-ci.org/travis-tahoe/tahoe-lafs/jobs/34190813

[FAIL]
Traceback (most recent call last):

File "/home/travis/build/travis-tahoe/tahoe-lafs/src/allmydata/test/test_util.py", line 522, in test_disk_stats
  self.failUnless(disk['used'] > 0, disk['used'])

File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/twisted/trial/_synctest.py", line 308, in assertTrue
  raise self.failureException(msg)

twisted.trial.unittest.FailTest: 0

allmydata.test.test_util.FileUtil.test_disk_stats

I don't understand why this failed on Python 2.6.9 but succeeded (https://travis-ci.org/travis-tahoe/tahoe-lafs/jobs/34190811) on the Python 2.7.8 run of the same build. Also, a previous build with identical source (https://travis-ci.org/travis-tahoe/tahoe-lafs/jobs/34187776) passed on Python 2.6.9.

(Travis isn't enabled yet for the main repo, but this is an identical test repo.)

Change History (15)

comment:1 Changed at 2014-09-12T16:11:51Z by warner

I caught an instance of this with extra debug prints turned on, to dump the return value from statvfs

https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/35096649

allmydata.test.test_util.FileUtil.test_disk_stats ...
 STATVFS posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=5242880, f_bfree=5242880, f_bavail=5242880, f_files=2621440, f_ffree=2621439, f_favail=2621439, f_flag=4096, f_namemax=255)
 STATVFS posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=5242880, f_bfree=5242880, f_bavail=5242880, f_files=2621440, f_ffree=2621439, f_favail=2621439, f_flag=4096, f_namemax=255)
 GET_DISK_STATS {'free_for_root': 21474836480, 'avail': 21474828288, 'free_for_nonroot': 21474836480, 'total': 21474836480, 'used': 0}

Traceback (most recent call last):
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_util.py", line 523, in test_disk_stats
    self.failUnless(disk['used'] > 0, disk['used'])
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/twisted/trial/_synctest.py", line 308, in assertTrue
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0

[FAILURE]

From what I can tell, every call to the os.statvfs in get_disk_stats() in this particular run returned the same thing.

In the passing test runs, os.statvfs() returns varying values. The 2.7 job that happened on the same git hash got:

allmydata.test.test_util.FileUtil.test_disk_stats ... 
STATVFS posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=31457280, f_bfree=25937477, f_bavail=25937477, f_files=24741045, f_ffree=24172188, f_favail=24172188, f_flag=4096, f_namemax=255)
STATVFS posix.statvfs_result(f_bsize=4096, f_frsize=4096, f_blocks=31457280, f_bfree=25937477, f_bavail=25937477, f_files=24741045, f_ffree=24172188, f_favail=24172188, f_flag=4096, f_namemax=255)
GET_DISK_STATS {'free_for_root': 106239905792, 'avail': 106239897600, 'free_for_nonroot': 106239905792, 'total': 128849018880, 'used': 22609113088}
Last edited at 2014-09-12T16:24:27Z by warner (previous) (diff)

comment:2 Changed at 2014-09-12T19:50:35Z by warner

#2298 was a dup.

After some more investigation, it appears that Travis-CI workers sometimes report fake-looking os.statvfs() values. I've filed an issue to investigate:

https://github.com/travis-ci/travis-ci/issues/2788

On the job that failed, every single os.statvfs() call for the whole 20-minute test suite claimed that the 20 GiB disk was empty (f_bfree == f_blocks). On passing jobs, f_blocks reported a 120 GiB disk (or larger) and the number of free blocks rose and fell over the course of the test run.

My hunch is that some of the travis workers are using a virtual disk that happens to return bogus or simulated data to os.statvfs(). Daira and I decided that it'd be ok to relax this test (to accept disk["used"] >= 0). If the Travis folks find an easy solution, we can tighten the test back up again.

comment:3 Changed at 2014-09-12T20:00:29Z by Brian Warner <warner@…>

  • Owner set to Brian Warner <warner@…>
  • Resolution set to fixed
  • Status changed from new to closed

In 38668c9e35a9f27123f05d3726e12263248996e6/trunk:

test_disk_stats: tolerate used==0 for a Travis worker bug

Some Travis-CI workers report persistently empty disks, causing spurious
test failures. It's not really that important to assert used>0, so this
relaxes the test.

Closes ticket:2290

comment:4 Changed at 2014-09-12T20:02:42Z by Brian Warner <warner@…>

In 38668c9e35a9f27123f05d3726e12263248996e6/trunk:

test_disk_stats: tolerate used==0 for a Travis worker bug

Some Travis-CI workers report persistently empty disks, causing spurious
test failures. It's not really that important to assert used>0, so this
relaxes the test.

Closes ticket:2290

comment:5 Changed at 2014-09-29T15:46:09Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:6 Changed at 2014-09-29T16:23:20Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:7 Changed at 2014-09-29T16:24:11Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:8 Changed at 2014-09-29T17:02:16Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:9 Changed at 2014-09-29T17:19:32Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:10 Changed at 2014-09-29T17:51:00Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:11 Changed at 2014-09-30T17:26:54Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:12 Changed at 2014-09-30T17:40:10Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:13 Changed at 2014-09-30T18:04:36Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:14 Changed at 2014-09-30T19:06:33Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

comment:15 Changed at 2014-10-02T00:34:29Z by Brian Warner <warner@…>

In e80f0753471948419466ad943d659b2a0f9e91a8/trunk:

travis: remove the os.statvfs debug command

According to https://github.com/travis-ci/travis-ci/issues/2788 (which
was WONTFIXED), roughly half the travis-ci buildserver fleet runs off
tmpfs volumes, which have different notions of space-free and space-used
than real filesystems. We've adapted our tests to avoid relying upon
space-used>0, and no longer need this call to investigate the problem.

refs ticket:2290

Note: See TracTickets for help on using tickets.