Changeset 609bd81 in trunk


Ignore:
Timestamp:
2022-01-25T15:42:04Z (3 years ago)
Author:
Itamar Turner-Trauring <itamar@…>
Branches:
master
Children:
2583236
Parents:
0ad31e3 (diff), 6480e0c (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.
Message:

Merge remote-tracking branch 'origin/master' into 3855-immutable-http-apis-part-1

Files:
1 added
1 deleted
15 edited
2 moved

Legend:

Unmodified
Added
Removed

  • TabularUnified .circleci/config.yml

    r0ad31e3 r609bd81  
    5050          {}
    5151
    52       # Just one Python 3.6 configuration while the port is in-progress.
    53       - "python36":
     52      # Test against Python 3:
     53      - "python37":
    5454          {}
    5555
    … …  
    119119      - "build-image-pypy27-buster":
    120120          <<: *DOCKERHUB_CONTEXT
    121       - "build-image-python36-ubuntu":
     121      - "build-image-python37-ubuntu":
    122122          <<: *DOCKERHUB_CONTEXT
    123123
    … …  
    380380
    381381
    382   python36:
     382  python37:
    383383    <<: *UBUNTU_18_04
    384384    docker:
    385385      - <<: *DOCKERHUB_AUTH
    386         image: "tahoelafsci/ubuntu:18.04-py3"
     386        image: "tahoelafsci/ubuntu:18.04-py3.7"
    387387        user: "nobody"
    388388
    … …  
    393393      # reporter.
    394394      TAHOE_LAFS_TRIAL_ARGS: "--reporter=subunitv2-file"
    395       TAHOE_LAFS_TOX_ENVIRONMENT: "py36"
     395      TAHOE_LAFS_TOX_ENVIRONMENT: "py37"
    396396
    397397
    … …  
    512512    docker:
    513513      - <<: *DOCKERHUB_AUTH
    514         image: "docker:17.05.0-ce-git"
     514        # CircleCI build images; https://github.com/CircleCI-Public/cimg-base
     515        # for details.
     516        image: "cimg/base:2022.01"
    515517
    516518    environment:
    … …  
    578580
    579581
    580   build-image-python36-ubuntu:
     582  build-image-python37-ubuntu:
    581583    <<: *BUILD_IMAGE
    582584
    … …  
    584586      DISTRO: "ubuntu"
    585587      TAG: "18.04"
    586       PYTHON_VERSION: "3"
     588      PYTHON_VERSION: "3.7"
    587589
    588590

  • TabularUnified .github/workflows/ci.yml

    r0ad31e3 r609bd81  
    4040        python-version:
    4141          - 2.7
    42           - 3.6
    4342          - 3.7
    4443          - 3.8
    4544          - 3.9
    4645        include:
    47           # On macOS don't bother with 3.6-3.8, just to get faster builds.
     46          # On macOS don't bother with 3.7-3.8, just to get faster builds.
    4847          - os: macos-10.15
    4948            python-version: 2.7
    … …  
    182181        python-version:
    183182          - 2.7
    184           - 3.6
     183          - 3.7
    185184          - 3.9
    186185        include:
    187           # On macOS don't bother with 3.6, just to get faster builds.
     186          # On macOS don't bother with 3.7, just to get faster builds.
    188187          - os: macos-10.15
    189188            python-version: 2.7

  • TabularUnified Makefile

    r0ad31e3 r609bd81  
    3636        tox --develop -e codechecks
    3737# Run all the test environments in parallel to reduce run-time
    38         tox --develop -p auto -e 'py27,py36,pypy27'
     38        tox --develop -p auto -e 'py27,py37,pypy27'
    3939.PHONY: test-venv-coverage
    4040## Run all tests with coverage collection and reporting.
    … …  
    5252## Run all tests under Python 3
    5353test-py3-all: .tox/create-venvs.log
    54         tox --develop -e py36 allmydata
     54        tox --develop -e py37 allmydata
    5555
    5656# This is necessary only if you want to automatically produce a new

  • TabularUnified NEWS.rst

    r0ad31e3 r609bd81  
    66
    77.. towncrier start line
     8Release 1.17.1 (2022-01-07)
     9'''''''''''''''''''''''''''
     10
     11Bug Fixes
     12---------
     13
     14- Fixed regression on Python 3 causing the JSON version of the Welcome page to sometimes produce a 500 error (`#3852 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3852>`_)
     15- Fixed regression on Python 3 where JSON HTTP POSTs failed to be processed. (`#3854 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3854>`_)
     16
     17
     18Misc/Other
     19----------
     20
     21- `#3848 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3848>`_, `#3849 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3849>`_, `#3850 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3850>`_, `#3856 <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3856>`_
    822
    923

  • TabularUnified docs/Installation/install-tahoe.rst

    r0ad31e3 r609bd81  
    2929
    30304. Start PowerShell and enter the following command to verify python installation::
    31    
     31
    3232    python --version
    3333
    34345. Enter the following command to install Tahoe-LAFS::
    35    
     35
    3636    pip install tahoe-lafs
    3737
    38386. Verify installation by checking for the version::
    39    
     39
    4040    tahoe --version
    4141
    … …  
    5757
    58582. Install Tahoe-LAFS using pip::
    59    
     59
    6060    pip install tahoe-lafs
    6161
    62623. Verify installation by checking for the version::
    63    
     63
    6464    tahoe --version
    6565
    66 If you are looking to hack on the source code or run pre-release code, we recommend you install Tahoe-LAFS on a `virtualenv` instance. To learn more, see :doc:`install-on-linux`.   
     66If you are looking to hack on the source code or run pre-release code, we recommend you install Tahoe-LAFS on a `virtualenv` instance. To learn more, see :doc:`install-on-linux`.
    6767
    6868You can always write to the `tahoe-dev mailing list <https://lists.tahoe-lafs.org/mailman/listinfo/tahoe-dev>`_ or chat on the `Libera.chat IRC <irc://irc.libera.chat/%23tahoe-lafs>`_ if you are not able to get Tahoe-LAFS up and running on your deployment.

  • TabularUnified docs/release-checklist.rst

    r0ad31e3 r609bd81  
    7171
    7272- update "docs/known_issues.rst" if appropriate
    73 - update "docs/Installation/install-tahoe.rst" references to the new release
    7473- Push the branch to github
    7574- Create a (draft) PR; this should trigger CI (note that github
    … …  
    108107    - tox -e deprecations,upcoming-deprecations
    109108
     109- clone to a clean, local checkout (to avoid extra files being included in the release)
     110
     111    - cd /tmp
     112    - git clone /home/meejah/src/tahoe-lafs
     113
    110114- build tarballs
    111115
    … …  
    154158- secure-copy all release artifacts to the download area on the
    155159  tahoe-lafs.org host machine. `~source/downloads` on there maps to
    156   https://tahoe-lafs.org/downloads/ on the Web.
    157 - scp dist/*1.15.0* username@tahoe-lafs.org:/home/source/downloads
     160  https://tahoe-lafs.org/downloads/ on the Web:
     161
     162    - scp dist/*1.15.0* username@tahoe-lafs.org:/home/source/downloads
     163
    158164- the following developers have access to do this:
    159165
    … …  
    161167  - meejah
    162168  - warner
     169
     170Push the signed tag to the main repository:
     171
     172- git push origin tahoe-lafs-1.17.1
    163173
    164174For the actual release, the tarball and signature files need to be

  • TabularUnified nix/tahoe-lafs.nix

    r0ad31e3 r609bd81  
    88}:
    99python.pkgs.buildPythonPackage rec {
    10   # Most of the time this is not exactly the release version (eg 1.17.0).
     10  # Most of the time this is not exactly the release version (eg 1.17.1).
    1111  # Give it a `post` component to make it look newer than the release version
    1212  # and we'll bump this up at the time of each release.
    … …  
    2121  # it is excluded from the source tree by default.  When it is included, the
    2222  # package tends to be frequently spuriously rebuilt.
    23   version = "1.17.0.post1";
     23  version = "1.17.1.post1";
    2424  name = "tahoe-lafs-${version}";
    2525  src = lib.cleanSourceWith {

  • TabularUnified relnotes.txt

    r0ad31e3 r609bd81  
    1 ANNOUNCING Tahoe, the Least-Authority File Store, v1.17.0
     1ANNOUNCING Tahoe, the Least-Authority File Store, v1.17.1
    22
    3 The Tahoe-LAFS team is pleased to announce version 1.17.0 of
     3The Tahoe-LAFS team is pleased to announce version 1.17.1 of
    44Tahoe-LAFS, an extremely reliable decentralized storage
    55system. Get it with "pip install tahoe-lafs", or download a
    … …  
    1616  https://tahoe-lafs.readthedocs.org/en/latest/about.html
    1717
    18 The previous stable release of Tahoe-LAFS was v1.16.0, released on
    19 October 19, 2021.
     18The previous stable release of Tahoe-LAFS was v1.17.0, released on
     19December 6, 2021.
    2020
    21 This release fixes several security issues raised as part of an audit
    22 by Cure53. We developed fixes for these issues in a private
    23 repository. Shortly after this release, public tickets will be updated
    24 with further information (along with, of course, all the code).
     21This release fixes two Python3-releated regressions and 4 minor bugs.
    2522
    26 There is also OpenMetrics support now and several bug fixes.
    27 
    28 In all, 46 issues have been fixed since the last release.
    29 
    30 Please see ``NEWS.rst`` for a more complete list of changes.
     23Please see ``NEWS.rst`` [1] for a complete list of changes.
    3124
    3225
    … …  
    6760all versions since v1.0.
    6861
    69 Network connections are limited by the Introducer protocol in
    70 use. If the Introducer is running v1.10 or v1.11, then servers
    71 from this release (v1.12) can serve clients of all versions
    72 back to v1.0 . If it is running v1.12, then they can only
    73 serve clients back to v1.10. Clients from this release can use
    74 servers back to v1.10, but not older servers.
     62Network connections are limited by the Introducer protocol in use. If
     63the Introducer is running v1.10 or v1.11, then servers from this
     64release can serve clients of all versions back to v1.0 . If it is
     65running v1.12 or higher, then they can only serve clients back to
     66v1.10. Clients from this release can use servers back to v1.10, but
     67not older servers.
    7568
    7669Except for the new optional MDMF format, we have not made any
    … …  
    8073to build such an infrastructure in the future.
    8174
    82 This is the twenty-first release in the version 1 series. This
     75This is the twenty-second release in the version 1 series. This
    8376series of Tahoe-LAFS will be actively supported and maintained
    8477for the foreseeable future, and future versions of Tahoe-LAFS
    … …  
    140133ACKNOWLEDGEMENTS
    141134
    142 This is the eighteenth release of Tahoe-LAFS to be created
     135This is the nineteenth release of Tahoe-LAFS to be created
    143136solely as a labor of love by volunteers. Thank you very much
    144137to the team of "hackers in the public interest" who make
    … …  
    148141on behalf of the Tahoe-LAFS team
    149142
    150 December 6, 2021
     143January 7, 2022
    151144Planet Earth
    152145
    153146
    154 [1] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.0/NEWS.rst
     147[1] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.1/NEWS.rst
    155148[2] https://github.com/tahoe-lafs/tahoe-lafs/blob/master/docs/known_issues.rst
    156149[3] https://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects
    157 [4] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.0/COPYING.GPL
    158 [5] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.0/COPYING.TGPPL.rst
    159 [6] https://tahoe-lafs.readthedocs.org/en/tahoe-lafs-1.17.0/INSTALL.html
     150[4] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.1/COPYING.GPL
     151[5] https://github.com/tahoe-lafs/tahoe-lafs/blob/tahoe-lafs-1.17.1/COPYING.TGPPL.rst
     152[6] https://tahoe-lafs.readthedocs.org/en/tahoe-lafs-1.17.1/INSTALL.html
    160153[7] https://lists.tahoe-lafs.org/mailman/listinfo/tahoe-dev
    161154[8] https://tahoe-lafs.org/trac/tahoe-lafs/roadmap

  • TabularUnified setup.py

    r0ad31e3 r609bd81  
    381381      packages=find_packages('src') + ['allmydata.test.plugins'],
    382382      classifiers=trove_classifiers,
    383       # We support Python 2.7, and we're working on support for 3.6 (the
    384       # highest version that PyPy currently supports).
    385       python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
     383      # We support Python 2.7, and Python 3.7 or later.
     384      python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*",
    386385      install_requires=install_requires,
    387386      extras_require={
    … …  
    414413              "junitxml",
    415414              "tenacity",
    416               "paramiko",
     415              # Pin old version until
     416              # https://github.com/paramiko/paramiko/issues/1961 is fixed.
     417              "paramiko < 2.9",
    417418              "pytest-timeout",
    418419              # Does our OpenMetrics endpoint adhere to the spec:

  • TabularUnified src/allmydata/test/web/test_root.py

    r0ad31e3 r609bd81  
    1212
    1313import time
     14import json
    1415
    1516from urllib.parse import (
    … …  
    2526from twisted.application import service
    2627from testtools.twistedsupport import succeeded
    27 from twisted.internet.defer import inlineCallbacks
     28from twisted.internet.defer import (
     29    inlineCallbacks,
     30    succeed,
     31)
    2832
    2933from ...storage_client import (
    … …  
    3135    StorageFarmBroker,
    3236)
    33 from ...web.root import RootElement
     37from ...web.root import (
     38    RootElement,
     39    Root,
     40)
    3441from ...util.connection_status import ConnectionStatus
     42from ...crypto.ed25519 import (
     43    create_signing_keypair,
     44)
    3545from allmydata.web.root import URIHandler
    3646from allmydata.client import _Client
    … …  
    4858from ..common import (
    4959    SyncTestCase,
     60    AsyncTestCase,
    5061)
    5162
    … …  
    139150        self.assertThat(item.slotData.get("version"), Equals(""))
    140151        self.assertThat(item.slotData.get("nickname"), Equals(""))
     152
     153
     154class RenderRoot(AsyncTestCase):
     155
     156    @inlineCallbacks
     157    def test_root_json(self):
     158        """
     159        The 'welcome' / root page renders properly with ?t=json when some
     160        servers show None for available_space while others show a
     161        valid int
     162
     163        See also https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3852
     164        """
     165        ann = {
     166            "anonymous-storage-FURL": "pb://w2hqnbaa25yw4qgcvghl5psa3srpfgw3@tcp:127.0.0.1:51309/vucto2z4fxment3vfxbqecblbf6zyp6x",
     167            "permutation-seed-base32": "w2hqnbaa25yw4qgcvghl5psa3srpfgw3",
     168        }
     169        srv0 = NativeStorageServer(b"server_id0", ann, None, {}, EMPTY_CLIENT_CONFIG)
     170        srv0.get_connection_status = lambda: ConnectionStatus(False, "summary0", {}, 0, 0)
     171
     172        srv1 = NativeStorageServer(b"server_id1", ann, None, {}, EMPTY_CLIENT_CONFIG)
     173        srv1.get_connection_status = lambda: ConnectionStatus(False, "summary1", {}, 0, 0)
     174        # arrange for this server to have some valid available space
     175        srv1.get_available_space = lambda: 12345
     176
     177        class FakeClient(_Client):
     178            history = []
     179            stats_provider = object()
     180            nickname = ""
     181            nodeid = b"asdf"
     182            _node_public_key = create_signing_keypair()[1]
     183            introducer_clients = []
     184            helper = None
     185
     186            def __init__(self):
     187                service.MultiService.__init__(self)
     188                self.storage_broker = StorageFarmBroker(
     189                    permute_peers=True,
     190                    tub_maker=None,
     191                    node_config=EMPTY_CLIENT_CONFIG,
     192                )
     193                self.storage_broker.test_add_server(b"test-srv0", srv0)
     194                self.storage_broker.test_add_server(b"test-srv1", srv1)
     195
     196        root = Root(FakeClient(), now_fn=time.time)
     197
     198        lines = []
     199
     200        req = DummyRequest(b"")
     201        req.fields = {}
     202        req.args = {
     203            b"t": [b"json"],
     204        }
     205
     206        # for some reason, DummyRequest is already finished when we
     207        # try to add a notifyFinish handler, so override that
     208        # behavior.
     209
     210        def nop():
     211            return succeed(None)
     212        req.notifyFinish = nop
     213        req.write = lines.append
     214
     215        yield root.render(req)
     216
     217        raw_js = b"".join(lines).decode("utf8")
     218        js = json.loads(raw_js)
     219        servers = js["servers"]
     220        self.assertEquals(len(servers), 2)
     221        self.assertIn(
     222            {
     223                "connection_status": "summary0",
     224                "nodeid": "server_id0",
     225                "last_received_data": 0,
     226                "version": None,
     227                "available_space": None,
     228                "nickname": ""
     229            },
     230            servers
     231        )
     232        self.assertIn(
     233            {
     234                "connection_status": "summary1",
     235                "nodeid": "server_id1",
     236                "last_received_data": 0,
     237                "version": None,
     238                "available_space": 12345,
     239                "nickname": ""
     240            },
     241            servers
     242        )

  • TabularUnified src/allmydata/test/web/test_web.py

    r0ad31e3 r609bd81  
    821821        d = self.GET("/?t=json")
    822822        def _check(res):
     823            """
     824            Check that the results are correct.
     825            We can't depend on the order of servers in the output
     826            """
    823827            decoded = json.loads(res)
    824             expected = {
    825                 u'introducers': {
    826                     u'statuses': [],
     828            self.assertEqual(decoded['introducers'], {u'statuses': []})
     829            actual_servers = decoded[u"servers"]
     830            self.assertEquals(len(actual_servers), 2)
     831            self.assertIn(
     832                {
     833                    u"nodeid": u'other_nodeid',
     834                    u'available_space': 123456,
     835                    u'connection_status': u'summary',
     836                    u'last_received_data': 30,
     837                    u'nickname': u'other_nickname \u263b',
     838                    u'version': u'1.0',
    827839                },
    828                 u'servers': sorted([
    829                     {u"nodeid": u'other_nodeid',
    830                      u'available_space': 123456,
    831                      u'connection_status': u'summary',
    832                      u'last_received_data': 30,
    833                      u'nickname': u'other_nickname \u263b',
    834                      u'version': u'1.0',
    835                     },
    836                     {u"nodeid": u'disconnected_nodeid',
    837                      u'available_space': 123456,
    838                      u'connection_status': u'summary',
    839                      u'last_received_data': 35,
    840                      u'nickname': u'disconnected_nickname \u263b',
    841                      u'version': u'1.0',
    842                     },
    843                 ], key=lambda o: sorted(o.items())),
    844             }
    845             self.assertEqual(expected, decoded)
     840                actual_servers
     841            )
     842            self.assertIn(
     843                {
     844                    u"nodeid": u'disconnected_nodeid',
     845                    u'available_space': 123456,
     846                    u'connection_status': u'summary',
     847                    u'last_received_data': 35,
     848                    u'nickname': u'disconnected_nickname \u263b',
     849                    u'version': u'1.0',
     850                },
     851                actual_servers
     852            )
     853
    846854        d.addCallback(_check)
    847855        return d

  • TabularUnified src/allmydata/test/web/test_webish.py

    r0ad31e3 r609bd81  
    9191        self._fields_test(b"GET", {}, b"", Equals(None))
    9292
    93     def test_form_fields(self):
     93    def test_form_fields_if_filename_set(self):
    9494        """
    9595        When a ``POST`` request is received, form fields are parsed into
    96         ``TahoeLAFSRequest.fields``.
     96        ``TahoeLAFSRequest.fields`` and the body is bytes (presuming ``filename``
     97        is set).
    9798        """
    9899        form_data, boundary = multipart_formdata([
    … …  
    121122            ),
    122123        )
     124
     125    def test_form_fields_if_name_is_file(self):
     126        """
     127        When a ``POST`` request is received, form fields are parsed into
     128        ``TahoeLAFSRequest.fields`` and the body is bytes when ``name``
     129        is set to ``"file"``.
     130        """
     131        form_data, boundary = multipart_formdata([
     132            [param(u"name", u"foo"),
     133             body(u"bar"),
     134            ],
     135            [param(u"name", u"file"),
     136             body(u"some file contents"),
     137            ],
     138        ])
     139        self._fields_test(
     140            b"POST",
     141            {b"content-type": b"multipart/form-data; boundary=" + bytes(boundary, 'ascii')},
     142            form_data.encode("ascii"),
     143            AfterPreprocessing(
     144                lambda fs: {
     145                    k: fs.getvalue(k)
     146                    for k
     147                    in fs.keys()
     148                },
     149                Equals({
     150                    "foo": "bar",
     151                    "file": b"some file contents",
     152                }),
     153            ),
     154        )
     155
     156    def test_form_fields_require_correct_mime_type(self):
     157        """
     158        The body of a ``POST`` is not parsed into fields if its mime type is
     159        not ``multipart/form-data``.
     160
     161        Reproducer for https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3854
     162        """
     163        data = u'{"lalala": "lolo"}'
     164        data = data.encode("utf-8")
     165        self._fields_test(b"POST", {"content-type": "application/json"},
     166                          data, Equals(None))
    123167
    124168

  • TabularUnified src/allmydata/web/root.py

    r0ad31e3 r609bd81  
    298298        return json.dumps(result, indent=1) + "\n"
    299299
    300 
    301300    def _describe_known_servers(self, broker):
    302         return sorted(list(
     301        return list(
    303302            self._describe_server(server)
    304303            for server
    305304            in broker.get_known_servers()
    306         ), key=lambda o: sorted(o.items()))
    307 
     305        )
    308306
    309307    def _describe_server(self, server):

  • TabularUnified src/allmydata/webish.py

    r0ad31e3 r609bd81  
    115115            self.args = parse_qs(argstring, 1)
    116116
    117         if self.method == b'POST':
     117        content_type = (self.requestHeaders.getRawHeaders("content-type") or [""])[0]
     118        if self.method == b'POST' and content_type.split(";")[0] in ("multipart/form-data", "application/x-www-form-urlencoded"):
    118119            # We use FieldStorage here because it performs better than
    119120            # cgi.parse_multipart(self.content, pdict) which is what

  • TabularUnified tox.ini

    r0ad31e3 r609bd81  
    99python =
    1010    2.7: py27-coverage,codechecks
    11     3.6: py36-coverage
    1211    3.7: py37-coverage,typechecks,codechecks3
    1312    3.8: py38-coverage
    … …  
    1918
    2019[tox]
    21 envlist = typechecks,codechecks,codechecks3,py{27,36,37,38,39}-{coverage},pypy27,pypy3,integration,integration3
     20envlist = typechecks,codechecks,codechecks3,py{27,37,38,39}-{coverage},pypy27,pypy3,integration,integration3
    2221minversion = 2.4
    2322
    … …  
    5251     certifi
    5352     # VCS hooks support
    54      py36,!coverage: pre-commit
     53     py37,!coverage: pre-commit
    5554
    5655# We add usedevelop=False because testing against a true installation gives
Note: See TracChangeset for help on using the changeset viewer.