[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2787: intermittent "Address Already In Use" error during tests

[Tahoe-LAFS]

#2787: intermittent "Address Already In Use" error during tests
--------------------------+------------------------
 Reporter:  warner        |          Owner:  warner
     Type:  defect        |         Status:  new
 Priority:  normal        |      Milestone:  soon
Component:  code-network  |        Version:  1.11.0
 Keywords:                |  Launchpad Bug:
--------------------------+------------------------
 I'm seeing occasional errors during tests like this:

 {{{
   File
 "/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py",
 line 662, in test_system_v2_server
     return self.do_system_test()
   File
 "/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py",
 line 378, in do_system_test
     self.create_tub()
   File
 "/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py",
 line 321, in create_tub
     tub.listenOn("tcp:%d" % portnum)
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/foolscap/pb.py", line 514, in listenOn
     l.setServiceParent(self)
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/application/service.py", line 188, in setServiceParent
     self.parent.addService(self)
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/application/service.py", line 309, in addService
     service.privilegedStartService()
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/application/service.py", line 278, in
 privilegedStartService
     service.privilegedStartService()
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/application/internet.py", line 113, in
 privilegedStartService
     self._port = self._getPort()
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/application/internet.py", line 141, in _getPort
     'listen%s' % (self.method,))(*self.args, **self.kwargs)
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/internet/posixbase.py", line 478, in listenTCP
     p.startListening()
   File "/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-
 packages/twisted/internet/tcp.py", line 984, in startListening
     raise CannotListenError(self.interface, self.port, le)
 twisted.internet.error.CannotListenError: Couldn't listen on any:49299:
 [Errno 48] Address already in use.
 [ERROR]
 }}}

 I'm still tracing this down, but it looks like iputil.py
 `allocate_tcp_port()` (which I wrote for Foolscap and copied over a few
 months ago) is sometimes giving us port numbers that are actually already
 in use. Those ports are coming from the kernel (we do a bind(port=0) and
 then ask what port got allocated).

 One problem that I know about is that we're binding the test port to
 127.0.0.1, and using SO_REUSEADDR, and the combination of those two might
 make the kernel think it's ok to give us a port that's already bound to
 something *other* than 127.0.0.1. But in some tests, replacing that with
 0.0.0.0 didn't help: I was still given ports that are already in use.

 I have to experiment some more to figure out what's going on. I think in
 the long run, `allocate_tcp_port()` might need to actually try to listen
 on the port, and if that fails, grab a different one.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2787>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage