id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
2787	"intermittent ""Address Already In Use"" error during tests"	warner	warner	"I'm seeing occasional errors during tests like this:

{{{
  File ""/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py"", line 662, in test_system_v2_server
    return self.do_system_test()
  File ""/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py"", line 378, in do_system_test
    self.create_tub()
  File ""/Users/warner/stuff/tahoe/tahoe/src/allmydata/test/test_introducer.py"", line 321, in create_tub
    tub.listenOn(""tcp:%d"" % portnum)
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/foolscap/pb.py"", line 514, in listenOn
    l.setServiceParent(self)
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/application/service.py"", line 188, in setServiceParent
    self.parent.addService(self)
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/application/service.py"", line 309, in addService
    service.privilegedStartService()
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/application/service.py"", line 278, in privilegedStartService
    service.privilegedStartService()
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/application/internet.py"", line 113, in privilegedStartService
    self._port = self._getPort()
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/application/internet.py"", line 141, in _getPort
    'listen%s' % (self.method,))(*self.args, **self.kwargs)
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/internet/posixbase.py"", line 478, in listenTCP
    p.startListening()
  File ""/Users/warner/stuff/tahoe/tahoe/.tox/py27/lib/python2.7/site-packages/twisted/internet/tcp.py"", line 984, in startListening
    raise CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:49299: [Errno 48] Address already in use.
[ERROR]
}}}

I'm still tracing this down, but it looks like iputil.py `allocate_tcp_port()` (which I wrote for Foolscap and copied over a few months ago) is sometimes giving us port numbers that are actually already in use. Those ports are coming from the kernel (we do a bind(port=0) and then ask what port got allocated).

One problem that I know about is that we're binding the test port to 127.0.0.1, and using SO_REUSEADDR, and the combination of those two might make the kernel think it's ok to give us a port that's already bound to something *other* than 127.0.0.1. But in some tests, replacing that with 0.0.0.0 didn't help: I was still given ports that are already in use.

I have to experiment some more to figure out what's going on. I think in the long run, `allocate_tcp_port()` might need to actually try to listen on the port, and if that fails, grab a different one.
"	defect	closed	normal	soon	code-network	1.11.0	wontfix