#3603 closed defect (fixed)

Start porting allmydata.scripts to Python 3

Reported by: jaraco Owned by: jaraco
Priority: normal Milestone: Support Python 3
Component: unknown Version: n/a
Keywords: review-needed Cc:
Launchpad Bug:

Description (last modified by itamarst)

Pull Request 975

Specifically covers allmydata.test.cli.test_create, allmydata.test.cli.test_runner, allmydata.scripts.create_node.

Change History (18)

comment:1 Changed at 2021-01-30T15:56:40Z by jaraco

I learned that for the scripts, even after adding them to 'PORTED_MODULES', none of the tests fail, so even syntax errors aren't caught by the test suite. I'll need to find another way to ascertain the routines necessary to validate an effective port.

comment:2 Changed at 2021-01-30T16:06:14Z by jaraco

I found at least one failure mode that can be fixed:

$ .tox/py36/bin/python -m allmydata.scripts.runner
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/allmydata/scripts/runner.py", line 251, in <module>
    run()
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/allmydata/scripts/runner.py", line 182, in run
    assert sys.version_info < (3,), u"Tahoe-LAFS does not run under Python 3. Please use Python 2.7.x."
AssertionError: Tahoe-LAFS does not run under Python 3. Please use Python 2.7.x.

comment:3 Changed at 2021-01-30T16:23:36Z by jaraco

After 5bf2b09b8, I've run some basic tests on runner, and it appears to be working with -q and -V and properly dispatching commands, so I'm prepared to declare that module as ported.

comment:4 Changed at 2021-01-30T17:09:37Z by jaraco

Attempting to run create-client results in an error:

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner create-client
/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:182: UserWarning: Support for Python 3 is experimental. Use at your own risk.
  warnings.warn("Support for Python 3 is experimental. Use at your own risk.")
Traceback (most recent call last):
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/create_node.py", line 469, in create_client
    return create_node(config)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/create_node.py", line 405, in create_node
    precondition(isinstance(basedir, unicode), basedir)
builtins.NameError: name 'unicode' is not defined

comment:5 Changed at 2021-01-30T17:21:55Z by jaraco

With 36b7fdaecf, create-client now succeeds.

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner create-client
Node created in '/Users/jaraco/.tahoe'
 Please add introducers to '/Users/jaraco/.tahoe/private/introducers.yaml'!
 The node cannot connect to a grid without it.
 Please set [node]nickname= in '/Users/jaraco/.tahoe/tahoe.cfg'

comment:6 Changed at 2021-01-30T17:27:37Z by jaraco

Run command fails with:

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner run
'tahoe run' in '/Users/jaraco/.tahoe'
running node in '/Users/jaraco/.tahoe'
2021-01-30T12:26:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 20.3.0 (/Users/jaraco/code/public/tahoe-lafs/.tox/py36/bin/python 3.6.6) starting up.
2021-01-30T12:26:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.selectreactor.SelectReactor.
2021-01-30T12:26:29-0500 [-] Foolscap logging initialized
2021-01-30T12:26:29-0500 [-] Note to developers: twistd.log does not receive very much.
2021-01-30T12:26:29-0500 [-] Use 'flogtool tail -c NODEDIR/private/logport.furl' instead
2021-01-30T12:26:29-0500 [-] and read docs/logging.rst
2021-01-30T12:26:32-0500 [-] Main loop terminated.
2021-01-30T12:26:32-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.

Unknown error
Traceback (most recent call last):
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 239, in create_client
    _client_factory=_client_factory,
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 290, in create_client_from_config
    storage_broker,
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 669, in __init__
    self.init_web(webport) # strports string
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 1024, in init_web
    staticdir,
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/webish.py", line 208, in __init__
    self.buildServer(webport, tempdir, nodeurl_path, staticdir)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/webish.py", line 228, in buildServer
    s = strports.service(webport, self.site)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/application/strports.py", line 40, in service
    endpoints.serverFromString(reactor, description), factory)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1731, in serverFromString
    nameOrPlugin, args, kw = _parseServer(description, None)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1649, in _parseServer
    getPlugins(IStreamServerEndpointStringParser), endpointType
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1665, in _matchPluginToPrefix
    raise ValueError("Unknown endpoint type: '%s'" % (endpointType,))
builtins.ValueError: Unknown endpoint type: 'b'tcp'

comment:7 Changed at 2021-01-30T17:33:14Z by jaraco

Looking at the config file, it looks like the config generated by create-client has some unwanted bytes literals:

[node]
nickname = b''
reveal-IP-address = true
web.port = b'tcp:3456:interface=127.0.0.1'

comment:8 Changed at 2021-01-30T17:48:55Z by jaraco

In 57cb88638, I've addressed that issue by opening the config file using io.open with encoding='utf-8' and then writing text.

comment:9 Changed at 2021-01-30T17:52:20Z by jaraco

  • Description modified (diff)
  • Status changed from new to assigned

comment:10 Changed at 2021-01-31T15:38:08Z by jaraco

Attempting to run an invalid command, the rendering doesn't look right:

$ .tox/py36/bin/python -m allmydata.scripts.runner start out
...
/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:  "UsageError('Unknown command: start',)"

Compare that to what's expected to be rendered on Python 2:

/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:  Unknown command: start

comment:11 Changed at 2021-01-31T15:54:14Z by jaraco

The issue traces to this line, where the message fails to be "decoded" when it's already text.

comment:13 Changed at 2021-01-31T16:14:54Z by jaraco

Fixed in 621ae58ab.

comment:14 Changed at 2021-01-31T16:25:33Z by jaraco

Running test_runner on Python 3 (tox -e py36 -- allmydata.test.test_runner) reveals many failures. That might be a good module to port next.

comment:15 Changed at 2021-02-10T22:02:36Z by jaraco

  • Keywords review-needed added

comment:16 Changed at 2021-02-10T22:03:09Z by jaraco

It occurs to me that the PR at its current state is probably in a good place for review. It doesn't close this issue, but at least addresses some concerns.

comment:17 Changed at 2021-03-17T19:23:16Z by itamarst

  • Description modified (diff)
  • Summary changed from Port allmydata.scripts to Python 3 to Start porting allmydata.scripts to Python 3

comment:18 Changed at 2021-03-17T20:05:28Z by GitHub <noreply@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In dddcff09/trunk:

Merge pull request #975 from tahoe-lafs/3603.scripts

Port scripts.create_node to Python 3

Fixes ticket:3603

Note: See TracTickets for help on using tickets.