Port the node.py module to Python 3

[rpatterson]

To improve the Python 3 compatibility, port one of the core modules, ./src/allmydata/node.py, to Python 3 compatibility while preserving Python 2 compatibility.

[Ross Patterson <me@…>]

In b2332b5/trunk:

fix(py3): Duplicate section name that py3 catches

Python 3 raises an exception that Python 2 doesn't and as such exposes malformed cfg/ini
contents used in the tests.

The test output diff shows that this mostly converts test DuplicateSectionError errors
to test errors with different exceptions which is reasonable for the type of fix this
is. It does seem to reduce coverage which I'm guessing is because the malformed
contents were triggering error handling code paths that aren't triggered now, but I
haven't confirmed that. I would think that to cover those cases we should write tests
that do that explicitly rather than accidentally.

`diff
--- ../../.tox/make-test-py3-all-old.log 2020-10-04 15:13:09.670578482 -0700
+++ ../../.tox/make-test-py3-all-new.log 2020-10-04 15:16:34.054975263 -0700
@@ -1835,7 +1835,7 @@

raise self.failureException(msg)

twisted.trial.unittest.FailTest?: ['allmydata', 'allmydata.main', 'allm[5873 chars]try'] != set() : Some unported modules remain:
Ported files: 96 / 292

-Ported lines: 27978 / 93480
+Ported lines: 27978 / 93482

allmydata.test.test_python3.Python3PortingEffortTests.test_finished_porting

@@ -2166,11 +2166,11 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 685, in test_disabled_but_helper

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 687, in test_disabled_but_helper

yield client.create_client(basedir)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in exit

self._expectedName, reason.getTraceback()),

-twisted.trial.unittest.FailTest?: configparser.DuplicateSectionError? raised instead of ValueError?:
+twisted.trial.unittest.FailTest?: builtins.NameError? raised instead of ValueError?:

Traceback (most recent call last):

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks

_inlineCallbacks(None, g, status)

@@ -2178,12 +2178,12 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 685, in test_disabled_but_helper

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 687, in test_disabled_but_helper

yield client.create_client(basedir)

--- <exception caught here> ---

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 685, in test_disabled_but_helper

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 687, in test_disabled_but_helper

yield client.create_client(basedir)

-configparser.DuplicateSectionError?: While reading from '/home/rpatterson/src/work/sfu/tahoe-lafs/_trial_temp/test_node/test_disabled_but_helper/tahoe.cfg' [line 10]: section 'node' already exists
+builtins.NameError?: name 'unicode' is not defined

allmydata.test.test_node.ClientNotListening?.test_disabled_but_helper

@@ -2194,11 +2194,11 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 668, in test_disabled_but_storage

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 670, in test_disabled_but_storage

yield client.create_client(basedir)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in exit

self._expectedName, reason.getTraceback()),

-twisted.trial.unittest.FailTest?: configparser.DuplicateSectionError? raised instead of ValueError?:
+twisted.trial.unittest.FailTest?: builtins.NameError? raised instead of ValueError?:

Traceback (most recent call last):

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks

_inlineCallbacks(None, g, status)

@@ -2206,12 +2206,12 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 668, in test_disabled_but_storage

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 670, in test_disabled_but_storage

yield client.create_client(basedir)

--- <exception caught here> ---

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 668, in test_disabled_but_storage

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 670, in test_disabled_but_storage

yield client.create_client(basedir)

-configparser.DuplicateSectionError?: While reading from '/home/rpatterson/src/work/sfu/tahoe-lafs/_trial_temp/test_node/test_disabled_but_storage/tahoe.cfg' [line 10]: section 'node' already exists
+builtins.NameError?: name 'unicode' is not defined

allmydata.test.test_node.ClientNotListening?.test_disabled_but_storage

@@ -2222,7 +2222,7 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 739, in test_create_client_invalid_config

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config

yield client.create_client(self.basedir)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in exit

self._expectedName, reason.getTraceback()),

@@ -2234,10 +2234,10 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 739, in test_create_client_invalid_config

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config

yield client.create_client(self.basedir)

--- <exception caught here> ---

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 739, in test_create_client_invalid_config

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config

yield client.create_client(self.basedir)

builtins.TypeError?: startswith first arg must be str or a tuple of str, not bytes

@@ -2250,7 +2250,7 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 702, in test_port_none_introducer

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 704, in test_port_none_introducer

yield create_introducer(basedir)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in exit

self._expectedName, reason.getTraceback()),

@@ -2262,10 +2262,10 @@

result = result.throwExceptionIntoGenerator(g)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator

return g.throw(self.type, self.value, self.tb)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 702, in test_port_none_introducer

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 704, in test_port_none_introducer

yield create_introducer(basedir)

--- <exception caught here> ---

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 702, in test_port_none_introducer

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 704, in test_port_none_introducer

yield create_introducer(basedir)

File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/introducer/server.py", line 76, in create_introducer

tor_provider,

@@ -7432,23 +7432,17 @@

===============================================================================
[ERROR]
Traceback (most recent call last):

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 655, in test_disabled

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 657, in test_disabled

n = yield client.create_client(basedir)

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 243, in create_client
• config = read_config(basedir, u"client.port")
• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 219, in read_config
• _valid_config=_valid_config(),
• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/node.py", line 187, in read_config
• parser = configutil.get_config(config_fname)
• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/util/configutil.py", line 50, in get_config
• config.readfp(f)
• File "/usr/lib/python3.6/configparser.py", line 764, in readfp
• self.read_file(fp, source=filename)
• File "/usr/lib/python3.6/configparser.py", line 718, in read_file
• self._read(f, source)
• File "/usr/lib/python3.6/configparser.py", line 1067, in _read
• lineno)

-configparser.DuplicateSectionError?: While reading from '/home/rpatterson/src/work/sfu/tahoe-lafs/_trial_temp/test_node/test_disabled/tahoe.cfg' [line 10]: section 'node' already exists
+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
+ result = g.send(result)
+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 285, in create_client_from_config
+ introducer_clients = create_introducer_clients(config, main_tub, _introducer_factory)
+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 519, in create_introducer_clients
+ introducer_cache_filepath,
+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/introducer/client.py", line 29, in init
+ assert type(nickname) is unicode
+builtins.NameError?: name 'unicode' is not defined

allmydata.test.test_node.ClientNotListening?.test_disabled
===============================================================================

@@ -7456,7 +7450,7 @@

Traceback (most recent call last):

File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 577, in test_listen_on_zero

t = FakeTub?()

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 549, in init

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 548, in init

self.tubID = base64.b32encode("foo")

File "/usr/lib/python3.6/base64.py", line 154, in b32encode

s = memoryview(s).tobytes()

@@ -7466,9 +7460,9 @@

===============================================================================
[ERROR]
Traceback (most recent call last):

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 605, in test_multiple_ports

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 606, in test_multiple_ports

t = FakeTub?()

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 549, in init

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 548, in init

self.tubID = base64.b32encode("foo")

File "/usr/lib/python3.6/base64.py", line 154, in b32encode

s = memoryview(s).tobytes()

@@ -7478,9 +7472,9 @@

===============================================================================
[ERROR]
Traceback (most recent call last):

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 624, in test_tor_i2p_listeners

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 626, in test_tor_i2p_listeners

t = FakeTub?()

• File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 549, in init

+ File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 548, in init

self.tubID = base64.b32encode("foo")

File "/usr/lib/python3.6/base64.py", line 154, in b32encode

s = memoryview(s).tobytes()

@@ -8781,11 +8775,11 @@

src/allmydata/immutable/downloader/init.py 7 1 2 1 78% 12, 11->12
src/allmydata/immutable/downloader/common.py 14 1 2 1 88% 12, 11->12
src/allmydata/immutable/downloader/fetcher.py 147 1 56 1 99% 12, 11->12

-src/allmydata/immutable/downloader/finder.py 143 9 40 5 92% 11, 88-89, 108, 115, 170-173, 10->11, 87->88, 107->108, 113->115, 165->exit
+src/allmydata/immutable/downloader/finder.py 143 11 40 5 91% 11, 88-89, 108, 115, 170-173, 231-232, 10->11, 87->88, 107->108, 113->115, 165->exit

src/allmydata/immutable/downloader/node.py 282 9 66 9 94% 11, 170-172, 224-235, 271, 10->11, 43->exit, 60->68, 165->169, 169->170, 256->exit, 270->271, 517->exit, 537->exit
src/allmydata/immutable/downloader/segmentation.py 118 1 22 1 99% 11, 10->11

-src/allmydata/immutable/downloader/share.py 453 15 154 8 96% 11, 210, 248-250, 354, 430-437, 518, 722-725, 862, 10->11, 209->210, 239->248, 352->354, 515->518, 660->663, 710->722, 762->exit
-src/allmydata/immutable/downloader/status.py 154 9 36 1 95% 11, 223, 226, 230, 232, 234, 274, 285, 287, 10->11
+src/allmydata/immutable/downloader/share.py 453 23 154 11 94% 11, 210, 239-250, 289-290, 354, 399-400, 430-437, 518, 722-725, 775, 862, 10->11, 209->210, 222->239, 288->289, 352->354, 397->399, 515->518, 660->663, 710->722, 762->exit, 774->775
+src/allmydata/immutable/downloader/status.py 154 12 36 1 93% 11, 65-67, 223, 226, 230, 232, 234, 274, 285, 287, 10->11

src/allmydata/immutable/encode.py 421 13 124 10 95% 114, 197-200, 278-280, 405, 412, 462, 509, 562, 687, 102->104, 113->114, 195->197, 404->405, 411->412, 461->462, 499->509, 505->511, 561->562, 685->687
src/allmydata/immutable/filenode.py 196 48 30 5 72% 77-78, 83, 85, 88, 94-101, 104-124, 127-172, 254, 258, 315, 40->42, 224->227, 227->229, 251->254, 257->258
src/allmydata/immutable/happiness_upload.py 214 1 132 3 99% 15, 13->15, 213->211, 280->279

@@ -8808,7 +8802,7 @@

src/allmydata/mutable/repairer.py 57 37 18 0 29% 13, 15, 17, 19, 29-34, 65-71, 74-126, 129-131
src/allmydata/mutable/retrieve.py 489 123 120 33 71% 46, 48, 50, 52, 56, 58, 60, 62, 64, 89-90, 133, 186-193, 204-208, 211-212, 224-226, 231, 240, 251, 312, 318, 344-354, 377, 385-386, 399-400, 425-434, 490, 501, 515-516, 529-540, 564-578, 591-592, 629-630, 653-654, 674-675, 681-682, 698, 712-729, 758-760, 765, 772-774, 790-792, 871, 883, 909-910, 919-941, 965-966, 981-994, 999-1005, 129->133, 167->169, 169->171, 201->204, 223->224, 230->231, 237->239, 239->240, 243->247, 249->251, 309->312, 317->318, 376->377, 381->385, 391->394, 396->399, 424->425, 489->490, 499->501, 514->515, 590->591, 628->629, 652->653, 673->674, 677->687, 680->681, 687->694, 694->698, 755->764, 764->765, 868->871, 880->883, 964->965
src/allmydata/mutable/servermap.py 612 240 186 26 56% 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 74, 130-139, 142, 148, 159-161, 175, 177, 183, 186-199, 206, 213, 217-220, 231, 234-238, 243-252, 255-259, 315, 328-350, 358-363, 370-372, 379, 429, 433, 443-447, 495, 498, 506-508, 514-516, 569-570, 603-611, 623-638, 718-721, 732-741, 792, 796, 803-804, 850-851, 872-874, 911-915, 929-945, 961-975, 982-999, 1003-1013, 1043-1045, 1050-1052, 1060-1064, 1069-1070, 1093-1100, 1106-1186, 1214-1215, 1232-1233, 313->315, 427->429, 432->433, 439->443, 459->461, 493->495, 497->498, 504->506, 509->514, 566->569, 597->603, 687->exit, 702->exit, 710->exit, 717->718, 727->732, 759->exit, 791->792, 795->796, 869->872, 1039->1043, 1047->1050, 1059->1060, 1066->1069, 1092->1093, 1213->1214

-src/allmydata/node.py 388 84 146 34 75% 120, 132, 190, 241, 243-245, 278, 284, 294-295, 303-306, 315, 320, 339, 341, 361, 393-396, 422, 449, 453, 490, 493, 500, 511-512, 548, 566, 574, 581, 583, 590-591, 601, 612, 629-633, 679, 681, 738-741, 747, 756, 764, 792-805, 808-809, 814-815, 825-846, 189->190, 240->241, 242->243, 277->278, 314->315, 319->320, 338->339, 340->341, 360->361, 391->393, 421->422, 448->449, 451->453, 489->490, 492->493, 499->500, 510->511, 537->535, 547->548, 565->566, 573->574, 580->581, 582->583, 589->590, 600->601, 611->612, 622->629, 673->679, 680->681, 737->738, 746->747, 763->764, 821->830, 823->821
+src/allmydata/node.py 388 84 146 33 75% 120, 132, 190, 241, 243-245, 278, 284, 294-295, 303-306, 315, 320, 339, 341, 361, 393-396, 422, 449, 453, 490, 493, 500, 511-512, 548, 566, 574, 581, 583, 590-591, 601, 612, 629-633, 679, 681, 738-741, 747, 756, 764, 792-805, 808-809, 814-815, 825-846, 189->190, 240->241, 242->243, 277->278, 314->315, 319->320, 338->339, 340->341, 360->361, 391->393, 421->422, 448->449, 451->453, 489->490, 492->493, 499->500, 510->511, 547->548, 565->566, 573->574, 580->581, 582->583, 589->590, 600->601, 611->612, 622->629, 673->679, 680->681, 737->738, 746->747, 763->764, 821->830, 823->821

src/allmydata/nodemaker.py 97 23 38 10 70% 49, 61, 66, 70, 81, 94, 107-115, 130-138, 141-150, 57->61, 65->66, 69->70, 79->81, 86->95, 90->94, 104->107, 124->exit, 129->130, 129->133
src/allmydata/scripts/admin.py 51 20 2 1 60% 9-14, 25, 28, 31-37, 40-46, 57, 59, 61-66, 56->57
src/allmydata/scripts/backupdb.py 146 91 14 1 36% 84-91, 94-96, 99, 103, 106, 111-114, 117-119, 122, 125, 128, 176-221, 231-242, 245-263, 266-272, 308-324, 327-333, 336-341, 306->308

@@ -8843,7 +8837,7 @@

src/allmydata/storage/common.py 24 1 4 2 89% 11, 10->11, 36->39
src/allmydata/storage/crawler.py 222 1 64 3 99% 16, 13->16, 96->99, 496->505
src/allmydata/storage/expirer.py 240 1 81 2 99% 9, 7->9, 250->exit

-src/allmydata/storage/immutable.py 198 1 48 5 98% 12, 11->12, 142->140, 155->160, 187->197, 273->exit
+src/allmydata/storage/immutable.py 198 2 48 6 97% 12, 101, 11->12, 100->101, 142->140, 155->160, 187->197, 273->exit

src/allmydata/storage/lease.py 35 1 4 1 95% 12, 11->12
src/allmydata/storage/mutable.py 289 8 90 6 96% 12, 162, 252, 289, 362-367, 11->12, 160->162, 247->252, 307->311, 354->362, 448->exit
src/allmydata/storage/server.py 371 9 120 9 96% 13, 92, 222, 243, 329, 349, 375, 422-423, 10->13, 91->92, 221->222, 241->243, 289->300, 317->329, 325->305, 346->349, 374->375

@@ -8905,7 +8899,7 @@

src/allmydata/windows/fixups.py 133 133 54 0 0% 1-237
src/allmydata/windows/registry.py 42 42 12 0 0% 1-77
------------------------------------------------------------------------------------------------

-TOTAL 27477 11786 8244 602 54%
+TOTAL 27477 11800 8244 605 54%

18 files skipped due to complete coverage.
+ '[' '!' -z 1 ']'

`

Trac: refs #3455

[Ross Patterson <me@…>]

In 95f2d53f/trunk:

chore(refs #3455) Address feedback, BBB comments

​https://github.com/tahoe-lafs/tahoe-lafs/pull/845#issuecomment-704469561