Changeset a0a39d3 in trunk

Timestamp:

2020-11-26T02:27:49Z (5 years ago)

Author:

meejah <meejah@…>

Branches:

master

Children:

4450a7a4

Parents:

da0fe23 (diff), fba386c (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.

Message:

Merge remote-tracking branch 'origin/master' into 2916.grid-manager-proposal.5

Files:

• .github/workflows/ci.yml (modified) (4 diffs)
• docs/INSTALL.rst (modified) (9 diffs)
• docs/how_to_make_a_tahoe-lafs_release.org (deleted)
• docs/release-checklist.rst (added)
• docs/windows.rst (modified) (3 diffs)
• integration/util.py (modified) (1 diff)
• newsfragments/1549.installation (added)
• newsfragments/3477.minor (added)
• newsfragments/3478.minor (added)
• newsfragments/3497.installation (added)
• newsfragments/3502.minor (added)
• newsfragments/3509.bugfix (added)
• newsfragments/3510.bugfix (added)
• newsfragments/3513.minor (added)
• newsfragments/3517.minor (added)
• newsfragments/3518.removed (added)
• newsfragments/3537.minor (added)
• nix/overlays.nix (modified) (1 diff)
• nix/twisted.nix (added)
• setup.py (modified) (1 diff)
• src/allmydata/client.py (modified) (4 diffs)
• src/allmydata/introducer/client.py (modified) (3 diffs)
• src/allmydata/mutable/layout.py (modified) (2 diffs)
• src/allmydata/mutable/repairer.py (modified) (1 diff)
• src/allmydata/mutable/retrieve.py (modified) (3 diffs)
• src/allmydata/mutable/servermap.py (modified) (9 diffs)
• src/allmydata/node.py (modified) (7 diffs)
• src/allmydata/scripts/runner.py (modified) (3 diffs)
• src/allmydata/test/cli/test_cli.py (modified) (1 diff)
• src/allmydata/test/common.py (modified) (2 diffs)
• src/allmydata/test/no_network.py (modified) (1 diff)
• src/allmydata/test/test_client.py (modified) (2 diffs)
• src/allmydata/test/test_introducer.py (modified) (13 diffs)
• src/allmydata/test/test_log.py (modified) (2 diffs)
• src/allmydata/test/test_runner.py (modified) (5 diffs)
• src/allmydata/test/test_version.py (deleted)
• src/allmydata/test/web/test_introducer.py (modified) (1 diff)
• src/allmydata/test/web/test_web.py (modified) (2 diffs)
• src/allmydata/test/web/test_webish.py (modified) (6 diffs)
• src/allmydata/util/_python3.py (modified) (2 diffs)
• src/allmydata/util/log.py (modified) (2 diffs)
• src/allmydata/version_checks.py (deleted)
• src/allmydata/web/introweb.py (modified) (2 diffs)
• src/allmydata/web/root.py (modified) (2 diffs)
• src/allmydata/webish.py (modified) (6 diffs)

.github/workflows/ci.yml ¶

@@ -22 +22 @@
     steps:
 
-      # Get vcpython27 on Windows + Python 2.7, to build zfec
+      # Get vcpython27 on Windows + Python 2.7, to build netifaces
       # extension.  See https://chocolatey.org/packages/vcpython27 and
       # https://github.com/crazy-max/ghaction-chocolatey
@@ -79 +79 @@
     steps:
 
+      # Get vcpython27 for Windows + Python 2.7, to build netifaces
+      # extension.  See https://chocolatey.org/packages/vcpython27 and
+      # https://github.com/crazy-max/ghaction-chocolatey
+      - name: Install MSVC 9.0 for Python 2.7 [Windows]
+        if: matrix.os == 'windows-latest' && matrix.python-version == '2.7'
+        uses: crazy-max/ghaction-chocolatey@v1
+        with:
+          args: install vcpython27
+
       - name: Install Tor [Ubuntu]
         if: matrix.os == 'ubuntu-latest'
@@ -92 +101 @@
         with:
           args: install tor
-
-      - name: Install MSVC 9.0 for Python 2.7 [Windows]
-        if: matrix.os == 'windows-latest' && matrix.python-version == '2.7'
-        uses: crazy-max/ghaction-chocolatey@v1
-        with:
-          args: install vcpython27
 
       - name: Check out Tahoe-LAFS sources
@@ -142 +145 @@
     steps:
 
-      # Get vcpython27 on Windows + Python 2.7, to build zfec
+      # Get vcpython27 for Windows + Python 2.7, to build netifaces
       # extension.  See https://chocolatey.org/packages/vcpython27 and
       # https://github.com/crazy-max/ghaction-chocolatey

docs/INSTALL.rst ¶

@@ -40 +40 @@
 
 If you are on a Mac, you can either follow these instructions, or use the
-pre-packaged bundle described in :doc:`OS-X`. The Tahoe project hosts
-pre-compiled "wheels" for all dependencies, so use the ``--find-links=``
-option described below to avoid needing a compiler.
+pre-packaged bundle described in :doc:`OS-X`.
 
 Many Linux distributions include Tahoe-LAFS packages. Debian and Ubuntu users
@@ -55 +53 @@
 
 If you don't use a pre-packaged copy of Tahoe, you can build it yourself.
-You'll need Python2.7, pip, and virtualenv. On unix-like platforms, you will
-need a C compiler, the Python development headers, and some libraries
-(libffi-dev and libssl-dev).
+You'll need Python2.7, pip, and virtualenv.
+Tahoe-LAFS depends on some libraries which require a C compiler to build.
+However, for many platforms, PyPI hosts already-built packages of libraries.
+
+If there is no already-built package for your platform,
+you will need a C compiler,
+the Python development headers,
+and some libraries (libffi-dev and libssl-dev).
 
 On a modern Debian/Ubuntu-derived distribution, this command will get you
@@ -65 +68 @@
 
 On OS-X, install pip and virtualenv as described below. If you want to
-compile the dependencies yourself (instead of using ``--find-links`` to take
-advantage of the pre-compiled ones we host), you'll also need to install
+compile the dependencies yourself, you'll also need to install
 Xcode and its command-line tools.
 
@@ -151 +153 @@
  New python executable in ~/venv/bin/python2.7
  Installing setuptools, pip, wheel...done.
- 
+
  % venv/bin/pip install -U pip setuptools
  Downloading/unpacking pip from https://pypi.python.org/...
  ...
  Successfully installed pip setuptools
- 
+
  % venv/bin/pip install tahoe-lafs
  Collecting tahoe-lafs
@@ -162 +164 @@
  Installing collected packages: ...
  Successfully installed ...
- 
+
  % venv/bin/tahoe --version
  tahoe-lafs: 1.14.0
  foolscap: ...
- 
+
  %
-
-On OS-X, instead of ``pip install tahoe-lafs``, use this command to take
-advantage of the hosted pre-compiled wheels::
-
- venv/bin/pip install --find-links=https://tahoe-lafs.org/deps tahoe-lafs
-
 
 Install From a Source Tarball
@@ -183 +179 @@
  New python executable in ~/venv/bin/python2.7
  Installing setuptools, pip, wheel...done.
- 
+
  % venv/bin/pip install https://tahoe-lafs.org/downloads/tahoe-lafs-1.14.0.tar.bz2
  Collecting https://tahoe-lafs.org/downloads/tahoe-lafs-1.14.0.tar.bz2
@@ -189 +185 @@
  Installing collected packages: ...
  Successfully installed ...
- 
+
  % venv/bin/tahoe --version
  tahoe-lafs: 1.14.0
@@ -214 +210 @@
 
  % git clone https://github.com/tahoe-lafs/tahoe-lafs.git
- 
+
  % cd tahoe-lafs
- 
+
  % virtualenv venv
- 
+
  % venv/bin/pip install --editable .[test]
  Obtaining file::~/tahoe-lafs
  ...
  Successfully installed ...
- 
+
  % venv/bin/tahoe --version
  tahoe-lafs: 1.14.0.post34.dev0
@@ -283 +279 @@
   ...
  Ran 1186 tests in 423.179s
- 
+
  PASSED (skips=7, expectedFailures=3, successes=1176)
  __________________________ summary ___________________________________

docs/windows.rst ¶

@@ -34 +34 @@
 into this virtualenv::
 
-    PS C:\Users\me> venv\Scripts\pip install --find-links=https://tahoe-lafs.org/deps/ tahoe-lafs
+    PS C:\Users\me> venv\Scripts\pip install tahoe-lafs
     Collecting tahoe-lafs
     ...
@@ -70 +70 @@
 git checkout)::
 
-    $ venv\Scripts\pip install --find-links=https://tahoe-lafs.org/deps/ .
+    $ venv\Scripts\pip install .
 
 If you're planning to hack on the source code, you might want to add
@@ -78 +78 @@
 ------------
 
-Tahoe-LAFS depends upon several packages that use compiled C code
-(such as zfec). This code must be built separately for each platform
-(Windows, OS-X, and different flavors of Linux).
-
-Pre-compiled "wheels" of all Tahoe's dependencies are hosted on the
-tahoe-lafs.org website in the ``deps/`` directory. The ``--find-links=``
-argument (used in the examples above) instructs ``pip`` to look at that URL
-for dependencies. This should avoid the need for anything to be compiled
-during the install.
+Tahoe-LAFS depends upon several packages that use compiled C code (such as zfec).
+This code must be built separately for each platform (Windows, OS-X, and different flavors of Linux).
+Fortunately, this is now done by upstream packages for most platforms.
+The result is that a C compiler is usually not required to install Tahoe-LAFS.

integration/util.py ¶

@@ -258 +258 @@
             config_path = join(node_dir, 'tahoe.cfg')
             config = get_config(config_path)
-            set_config(config, 'node', 'log_gatherer.furl', flog_gatherer)
+            set_config(
+                config,
+                u'node',
+                u'log_gatherer.furl',
+                flog_gatherer.decode("utf-8"),
+            )
             write_config(config_path, config)
         created_d.addCallback(created)

nix/overlays.nix ¶

@@ -16 +16 @@
       # is too old.
       pyutil = python-super.callPackage ./pyutil.nix { };
+
+      # Need a newer version of Twisted, too.
+      twisted = python-super.callPackage ./twisted.nix { };
     };
   };

setup.py ¶

@@ -99 +99 @@
     #   specified by Twisted[conch].  Since this would be the *whole point* of
     #   an sftp extra in Tahoe-LAFS, there is no point in having one.
-    "Twisted[tls,conch] >= 18.4.0",
+    # * Twisted 19.10 introduces Site.getContentFile which we use to get
+    #   temporary upload files placed into a per-node temporary directory.
+    "Twisted[tls,conch] >= 19.10.0",
 
     "PyYAML >= 3.11",

src/allmydata/client.py ¶

@@ -39 +39 @@
     hashutil, base32, pollmixin, log, idlib,
     yamlutil, configutil,
+    fileutil,
 )
 from allmydata.util.encodingutil import get_filesystem_encoding
@@ -520 +521 @@
             str(allmydata.__full_version__),
             str(_Client.OLDEST_SUPPORTED_VERSION),
-            list(node.get_app_versions()),
             partial(_sequencer, config),
             introducer_cache_filepath,
@@ -1071 +1071 @@
         self._key_generator.set_default_keysize(keysize)
 
+    def _get_tempdir(self):
+        """
+        Determine the path to the directory where temporary files for this node
+        should be written.
+
+        :return bytes: The path which will exist and be a directory.
+        """
+        tempdir_config = self.config.get_config("node", "tempdir", "tmp")
+        if isinstance(tempdir_config, bytes):
+            tempdir_config = tempdir_config.decode('utf-8')
+        tempdir = self.config.get_config_path(tempdir_config)
+        if not os.path.exists(tempdir):
+            fileutil.make_dirs(tempdir)
+        return tempdir
+
     def init_web(self, webport):
         self.log("init_web(webport=%s)", args=(webport,))
@@ -1078 +1093 @@
         staticdir_config = self.config.get_config("node", "web.static", "public_html")
         staticdir = self.config.get_config_path(staticdir_config)
-        ws = WebishServer(self, webport, nodeurl_path, staticdir)
+        ws = WebishServer(
+            self,
+            webport,
+            self._get_tempdir(),
+            nodeurl_path,
+            staticdir,
+        )
         ws.setServiceParent(self)
 

src/allmydata/introducer/client.py ¶

@@ -25 +25 @@
     def __init__(self, tub, introducer_furl,
                  nickname, my_version, oldest_supported,
-                 app_versions, sequencer, cache_filepath):
+                 sequencer, cache_filepath):
         self._tub = tub
         self.introducer_furl = introducer_furl
@@ -33 +33 @@
         self._my_version = my_version
         self._oldest_supported = oldest_supported
-        self._app_versions = app_versions
         self._sequencer = sequencer
         self._cache_filepath = cache_filepath
 
-        self._my_subscriber_info = {
-            "version": 0,
-            "nickname": self._nickname,
-            "app-versions": self._app_versions,
-            "my-version": self._my_version,
-            "oldest-supported": self._oldest_supported,
-        }
+        self._my_subscriber_info = { "version": 0,
+                                     "nickname": self._nickname,
+                                     "app-versions": [],
+                                     "my-version": self._my_version,
+                                     "oldest-supported": self._oldest_supported,
+                                     }
 
         self._outbound_announcements = {} # not signed
@@ -192 +190 @@
                   # publish(), each time we make a change
                   "nickname": self._nickname,
-                  "app-versions": self._app_versions,
+                  "app-versions": [],
                   "my-version": self._my_version,
                   "oldest-supported": self._oldest_supported,

src/allmydata/mutable/layout.py ¶

@@ +1 @@
+"""
+Ported to Python 3.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+
+from future.utils import PY2
+if PY2:
+    # Omit dict so Python 3 changes don't leak into API callers on Python 2.
+    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, list, object, range, str, max, min  # noqa: F401
 from past.utils import old_div
 
@@ -1745 +1757 @@
 
     def _read(self, readvs, force_remote=False):
-        unsatisfiable = list(filter(lambda x: x[0] + x[1] > len(self._data), readvs))
+        unsatisfiable = [x for x in readvs if x[0] + x[1] > len(self._data)]
         # TODO: It's entirely possible to tweak this so that it just
         # fulfills the requests that it can, and not demand that all

src/allmydata/mutable/repairer.py ¶

@@ +1 @@
+"""
+Ported to Python 3.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+
+from future.utils import PY2
+if PY2:
+    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
 
 from zope.interface import implementer

src/allmydata/mutable/retrieve.py ¶

@@ -1 @@
-from past.builtins import unicode
+"""
+Ported to Python 3.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+
+from future.utils import PY2
+if PY2:
+    # Don't import bytes and str, to prevent API leakage
+    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, dict, list, object, range, max, min  # noqa: F401
 
 import time
@@ -750 +761 @@
         blockhashes = dict(enumerate(blockhashes))
         self.log("the reader gave me the following blockhashes: %s" % \
-                 blockhashes.keys())
+                 list(blockhashes.keys()))
         self.log("the reader gave me the following sharehashes: %s" % \
-                 sharehashes.keys())
+                 list(sharehashes.keys()))
         bht = self._block_hash_trees[reader.shnum]
 
@@ -909 +920 @@
 
     def notify_server_corruption(self, server, shnum, reason):
-        if isinstance(reason, unicode):
+        if isinstance(reason, str):
             reason = reason.encode("utf-8")
         storage_server = server.get_storage_server()

src/allmydata/mutable/servermap.py ¶

@@ +1 @@
+"""
+Ported to Python 3.
+"""
 from __future__ import print_function
-
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import unicode_literals
+
+from future.utils import PY2
+if PY2:
+    # Doesn't import str to prevent API leakage on Python 2
+    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, max, min  # noqa: F401
 from past.builtins import unicode
 
@@ -189 +199 @@
         print("servermap:", file=out)
 
-        for ( (server, shnum), (verinfo, timestamp) ) in self._known_shares.items():
+        for ( (server, shnum), (verinfo, timestamp) ) in list(self._known_shares.items()):
             (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
              offsets_tuple) = verinfo
@@ -227 +237 @@
         timestamp) tuples."""
         versionmap = DictOfSets()
-        for ( (server, shnum), (verinfo, timestamp) ) in self._known_shares.items():
+        for ( (server, shnum), (verinfo, timestamp) ) in list(self._known_shares.items()):
             versionmap.add(verinfo, (shnum, server, timestamp))
         return versionmap
@@ -246 +256 @@
         versionmap = self.make_versionmap()
         all_shares = {}
-        for verinfo, shares in versionmap.items():
+        for verinfo, shares in list(versionmap.items()):
             s = set()
             for (shnum, server, timestamp) in shares:
@@ -272 +282 @@
         versionmap = self.make_versionmap()
         bits = []
-        for (verinfo, shares) in versionmap.items():
+        for (verinfo, shares) in list(versionmap.items()):
             vstr = self.summarize_version(verinfo)
             shnums = set([shnum for (shnum, server, timestamp) in shares])
@@ -283 +293 @@
         versionmap = self.make_versionmap()
         recoverable_versions = set()
-        for (verinfo, shares) in versionmap.items():
+        for (verinfo, shares) in list(versionmap.items()):
             (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
              offsets_tuple) = verinfo
@@ -299 +309 @@
 
         unrecoverable_versions = set()
-        for (verinfo, shares) in versionmap.items():
+        for (verinfo, shares) in list(versionmap.items()):
             (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
              offsets_tuple) = verinfo
@@ -333 +343 @@
         unrecoverable = set()
         highest_recoverable_seqnum = -1
-        for (verinfo, shares) in versionmap.items():
+        for (verinfo, shares) in list(versionmap.items()):
             (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
              offsets_tuple) = verinfo
@@ -668 +678 @@
         ds = []
 
-        for shnum,datav in datavs.items():
+        for shnum,datav in list(datavs.items()):
             data = datav[0]
             reader = MDMFSlotReadProxy(ss,

src/allmydata/node.py ¶

@@ -20 +20 @@
 import types
 import errno
-import tempfile
 from base64 import b32decode, b32encode
 
@@ -29 +28 @@
 from twisted.application import service
 from twisted.python.failure import Failure
-from twisted.python.compat import nativeString
-from foolscap.api import Tub, app_versions
+from foolscap.api import Tub
+
 import foolscap.logging.log
-from allmydata.version_checks import get_package_versions, get_package_versions_string
+
 from allmydata.util import log
 from allmydata.util import fileutil, iputil
-from allmydata.util.assertutil import _assert
 from allmydata.util.fileutil import abspath_expanduser_unicode
 from allmydata.util.encodingutil import get_filesystem_encoding, quote_output
 from allmydata.util import configutil
+
+from . import (
+    __full_version__,
+)
 
 def _common_valid_config():
@@ -80 +82 @@
     })
 
-# Add our application versions to the data that Foolscap's LogPublisher
-# reports.
-for thing, things_version in list(get_package_versions().items()):
-    app_versions.add_version(thing, things_version)
-
 # group 1 will be addr (dotted quad string), group 3 if any will be portnum (string)
 ADDR_RE = re.compile("^([1-9][0-9]*\.[1-9][0-9]*\.[1-9][0-9]*\.[1-9][0-9]*)(:([1-9][0-9]*))?$")
@@ -233 +230 @@
         """
     return _Config(parser, portnumfile, basedir, fname, write_new_config)
-
-
-def get_app_versions():
-    """
-    :returns: dict of versions important to Foolscap
-    """
-    return dict(app_versions.versions)
 
 
@@ -833 +823 @@
         self._tor_provider = tor_provider
 
-        self.init_tempdir()
-
         self.create_log_tub()
         self.logSource = "Node"
@@ -852 +840 @@
             self.control_tub.setServiceParent(self)
 
-        self.log("Node constructed. " + get_package_versions_string())
+        self.log("Node constructed. " + __full_version__)
         iputil.increase_rlimits()
 
@@ -860 +848 @@
         """
         return len(self.tub.getListeners()) > 0
-
-    def init_tempdir(self):
-        """
-        Initialize/create a directory for temporary files.
-        """
-        tempdir_config = self.config.get_config("node", "tempdir", "tmp")
-        if isinstance(tempdir_config, bytes):
-            tempdir_config = tempdir_config.decode('utf-8')
-        tempdir = self.config.get_config_path(tempdir_config)
-        if not os.path.exists(tempdir):
-            fileutil.make_dirs(tempdir)
-        tempfile.tempdir = nativeString(tempdir)
-        # this should cause twisted.web.http (which uses
-        # tempfile.TemporaryFile) to put large request bodies in the given
-        # directory. Without this, the default temp dir is usually /tmp/,
-        # which is frequently too small.
-        temp_fd, test_name = tempfile.mkstemp()
-        _assert(os.path.dirname(test_name) == tempdir, test_name, tempdir)
-        os.close(temp_fd)  # avoid leak of unneeded fd
 
     # pull this outside of Node's __init__ too, see:

src/allmydata/scripts/runner.py ¶

@@ -8 +8 @@
 from twisted.internet import defer, task, threads
 
-from allmydata.version_checks import get_package_versions_string
 from allmydata.scripts.common import get_default_nodedir
 from allmydata.scripts import debug, create_node, cli, \
@@ -20 +19 @@
 )
 
+from .. import (
+    __full_version__,
+)
+
 _default_nodedir = get_default_nodedir()
 
@@ -78 +81 @@
 
     def opt_version(self):
-        print(get_package_versions_string(debug=True), file=self.stdout)
+        print(__full_version__, file=self.stdout)
         self.no_command_needed = True
 
-    def opt_version_and_path(self):
-        print(get_package_versions_string(show_paths=True, debug=True), file=self.stdout)
-        self.no_command_needed = True
+    opt_version_and_path = opt_version
 
     opt_eliot_destination = opt_eliot_destination

src/allmydata/test/cli/test_cli.py ¶

@@ -1267 +1267 @@
         stdout = StringIO()
         self.failUnlessRaises(SystemExit, self.parse, ["--version"], stdout)
-        self.failUnlessIn(allmydata.__appname__ + ":", stdout.getvalue())
+        self.failUnlessIn(allmydata.__full_version__, stdout.getvalue())
         # but "tahoe SUBCOMMAND --version" should be rejected
         self.failUnlessRaises(usage.UsageError, self.parse,

src/allmydata/test/common.py ¶

@@ -111 +111 @@
     my_version = attr.ib()
     oldest_supported = attr.ib()
-    app_versions = attr.ib()
     sequencer = attr.ib()
     cache_filepath = attr.ib()
@@ -1152 +1151 @@
     * trial-compatible mktemp method
     * unittest2-compatible assertRaises helper
-    * Automatic cleanup of tempfile.tempdir mutation (pervasive through the
-      Tahoe-LAFS test suite).
+    * Automatic cleanup of tempfile.tempdir mutation (once pervasive through
+      the Tahoe-LAFS test suite, perhaps gone now but someone should verify
+      this).
     """
     def setUp(self):

src/allmydata/test/no_network.py ¶

@@ -362 +362 @@
         """
         if self._setup_errors:
-            raise self._setup_errors[0].value
+            self._setup_errors[0].raiseException()
 
     @defer.inlineCallbacks

src/allmydata/test/test_client.py ¶

@@ -42 +42 @@
 from allmydata.node import OldConfigError, UnescapedHashError, create_node_dir
 from allmydata.frontends.auth import NeedRootcapLookupScheme
-from allmydata.version_checks import (
-    get_package_versions_string,
-)
 from allmydata import client
 from allmydata.storage_client import (
@@ -622 +619 @@
         self.failUnless("." in str(allmydata.__full_version__),
                         "non-numeric version in '%s'" % allmydata.__version__)
-        all_versions = get_package_versions_string()
-        self.failUnless(allmydata.__appname__ in all_versions)
         # also test stats
         stats = c.get_stats()

src/allmydata/test/test_introducer.py ¶

@@ -156 +156 @@
     def test_create(self):
         ic = IntroducerClient(None, "introducer.furl", u"my_nickname",
-                              "my_version", "oldest_version", {}, fakeseq,
+                              "my_version", "oldest_version", fakeseq,
                               FilePath(self.mktemp()))
         self.failUnless(isinstance(ic, IntroducerClient))
@@ -189 +189 @@
         ic1 = IntroducerClient(None,
                                "introducer.furl", u"my_nickname",
-                               "ver23", "oldest_version", {}, fakeseq,
+                               "ver23", "oldest_version", fakeseq,
                                FilePath(self.mktemp()))
         # we use a second client just to create a different-looking
@@ -195 +195 @@
         ic2 = IntroducerClient(None,
                                "introducer.furl", u"my_nickname",
-                               "ver24","oldest_version",{}, fakeseq,
+                               "ver24","oldest_version",fakeseq,
                                FilePath(self.mktemp()))
         announcements = []
@@ -299 +299 @@
         ic1 = IntroducerClient(None,
                                "introducer.furl", u"my_nickname",
-                               "ver23", "oldest_version", {}, realseq,
+                               "ver23", "oldest_version", realseq,
                                FilePath(self.mktemp()))
         furl1 = "pb://62ubehyunnyhzs7r6vdonnm2hpi52w6y@127.0.0.1:36106/gydnp"
@@ -397 +397 @@
         tub2.setServiceParent(self.parent)
         c = IntroducerClient(tub2, ifurl,
-                             u"nickname", "version", "oldest", {}, fakeseq,
+                             u"nickname", "version", "oldest", fakeseq,
                              FilePath(self.mktemp()))
         furl1 = "pb://onug64tu@127.0.0.1:123/short" # base32("short")
@@ -478 +478 @@
                                  NICKNAME % str(i),
                                  "version", "oldest",
-                                 {"component": "component-v1"}, fakeseq,
+                                 fakeseq,
                                  FilePath(self.mktemp()))
             received_announcements[c] = {}
@@ -738 +738 @@
         introducer = IntroducerService()
         tub = introducer_furl = None
-        app_versions = {"whizzy": "fizzy"}
         client_v2 = IntroducerClient(tub, introducer_furl, NICKNAME % u"v2",
-                                     "my_version", "oldest", app_versions,
+                                     "my_version", "oldest",
                                      fakeseq, FilePath(self.mktemp()))
         #furl1 = "pb://62ubehyunnyhzs7r6vdonnm2hpi52w6y@127.0.0.1:0/swissnum"
@@ -752 +751 @@
         s0 = subs[0]
         self.failUnlessEqual(s0.service_name, "storage")
-        self.failUnlessEqual(s0.app_versions, app_versions)
         self.failUnlessEqual(s0.nickname, NICKNAME % u"v2")
         self.failUnlessEqual(s0.version, "my_version")
@@ -761 +759 @@
         introducer = IntroducerService()
         tub = introducer_furl = None
-        app_versions = {"whizzy": "fizzy"}
         client_v2 = IntroducerClient(tub, introducer_furl, u"nick-v2",
-                                     "my_version", "oldest", app_versions,
+                                     "my_version", "oldest",
                                      fakeseq, FilePath(self.mktemp()))
         furl1 = "pb://62ubehyunnyhzs7r6vdonnm2hpi52w6y@127.0.0.1:0/swissnum"
@@ -777 +774 @@
         self.assertThat(a[0].canary, Is(canary0))
         self.failUnlessEqual(a[0].index, ("storage", public_key_str))
-        self.failUnlessEqual(a[0].announcement["app-versions"], app_versions)
         self.failUnlessEqual(a[0].nickname, u"nick-v2")
         self.failUnlessEqual(a[0].service_name, "storage")
@@ -855 +851 @@
         yield flushEventualQueue()
         ic2 = IntroducerClient(None, "introducer.furl", u"my_nickname",
-                               "my_version", "oldest_version", {}, fakeseq,
+                               "my_version", "oldest_version", fakeseq,
                                ic._cache_filepath)
         announcements = {}
@@ -955 +951 @@
         listenOnUnused(tub)
         c = IntroducerClient(tub, self.introducer_furl,
-                             u"nickname-client", "version", "oldest", {},
+                             u"nickname-client", "version", "oldest",
                              fakeseq, FilePath(self.mktemp()))
         announcements = {}
@@ -1028 +1024 @@
             "0.0.0",
             "1.2.3",
-            {},
             (0, u"i am a nonce"),
             "invalid",

src/allmydata/test/test_log.py ¶

@@ -10 +10 @@
 from __future__ import print_function
 
-from future.utils import PY2
+from future.utils import PY2, native_str
 if PY2:
     from builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
@@ -155 +155 @@
         self.assertEqual([m[2] for m in self.messages],
                          ["grand", "par1", "par2", "msg1", "msg1"])
+
+    def test_native_string_keys(self):
+        """Keyword argument keys are all native strings."""
+        class LoggingObject17(tahoe_log.PrefixingLogMixin):
+            pass
+
+        obj = LoggingObject17()
+        # Native string by default:
+        obj.log(hello="world")
+        # Will be Unicode on Python 2:
+        obj.log(**{"my": "message"})
+        for message in self.messages:
+            for k in message[-1].keys():
+                self.assertIsInstance(k, native_str)

src/allmydata/test/test_runner.py ¶

@@ -13 +13 @@
 from twisted.internet.defer import (
     inlineCallbacks,
-    returnValue,
     DeferredList,
 )
@@ -21 +20 @@
 )
 from allmydata.util import fileutil, pollmixin
-from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output, \
-    get_filesystem_encoding
+from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output
 from allmydata.test import common_util
-from allmydata.version_checks import normalized_version
 import allmydata
-from allmydata import __appname__
 from .common_util import parse_cli, run_cli
 from .cli_node_api import (
@@ -59 +55 @@
 
 class RunBinTahoeMixin(object):
-
-    @inlineCallbacks
-    def find_import_location(self):
-        res = yield self.run_bintahoe(["--version-and-path"])
-        out, err, rc_or_sig = res
-        self.assertEqual(rc_or_sig, 0, res)
-        lines = out.splitlines()
-        tahoe_pieces = lines[0].split()
-        self.assertEqual(tahoe_pieces[0], "%s:" % (__appname__,), (tahoe_pieces, res))
-        returnValue(tahoe_pieces[-1].strip("()"))
-
     def run_bintahoe(self, args, stdin=None, python_options=[], env=None):
         command = sys.executable
@@ -87 +72 @@
 
 class BinTahoe(common_util.SignalMixin, unittest.TestCase, RunBinTahoeMixin):
-    @inlineCallbacks
-    def test_the_right_code(self):
-        # running "tahoe" in a subprocess should find the same code that
-        # holds this test file, else something is weird
-        test_path = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
-        bintahoe_import_path = yield self.find_import_location()
-
-        same = (bintahoe_import_path == test_path)
-        if not same:
-            msg = ("My tests and my 'tahoe' executable are using different paths.\n"
-                   "tahoe: %r\n"
-                   "tests: %r\n"
-                   "( according to the test source filename %r)\n" %
-                   (bintahoe_import_path, test_path, srcfile))
-
-            if (not isinstance(rootdir, unicode) and
-                rootdir.decode(get_filesystem_encoding(), 'replace') != rootdir):
-                msg += ("However, this may be a false alarm because the import path\n"
-                        "is not representable in the filesystem encoding.")
-                raise unittest.SkipTest(msg)
-            else:
-                msg += "Please run the tests in a virtualenv that includes both the Tahoe-LAFS library and the 'tahoe' executable."
-                self.fail(msg)
-
-    def test_path(self):
-        d = self.run_bintahoe(["--version-and-path"])
-        def _cb(res):
-            out, err, rc_or_sig = res
-            self.failUnlessEqual(rc_or_sig, 0, str(res))
-
-            # Fail unless the __appname__ package is *this* version *and*
-            # was loaded from *this* source directory.
-
-            required_verstr = str(allmydata.__version__)
-
-            self.failIfEqual(required_verstr, "unknown",
-                             "We don't know our version, because this distribution didn't come "
-                             "with a _version.py and 'setup.py update_version' hasn't been run.")
-
-            srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
-            info = repr((res, allmydata.__appname__, required_verstr, srcdir))
-
-            appverpath = out.split(')')[0]
-            (appverfull, path) = appverpath.split('] (')
-            (appver, comment) = appverfull.split(' [')
-            (branch, full_version) = comment.split(': ')
-            (app, ver) = appver.split(': ')
-
-            self.failUnlessEqual(app, allmydata.__appname__, info)
-            norm_ver = normalized_version(ver)
-            norm_required = normalized_version(required_verstr)
-            self.failUnlessEqual(norm_ver, norm_required, info)
-            self.failUnlessEqual(path, srcdir, info)
-            self.failUnlessEqual(branch, allmydata.branch)
-            self.failUnlessEqual(full_version, allmydata.full_version)
-        d.addCallback(_cb)
-        return d
-
     def test_unicode_arguments_and_output(self):
         tricky = u"\u2621"
@@ -166 +93 @@
         def _cb(res):
             out, err, rc_or_sig = res
-            self.failUnlessEqual(rc_or_sig, 0, str(res))
-            self.failUnless(out.startswith(allmydata.__appname__+':'), str(res))
+            self.assertEqual(rc_or_sig, 0, str(res))
+            self.assertTrue(out.startswith(allmydata.__appname__ + '/'), str(res))
         d.addCallback(_cb)
         return d

src/allmydata/test/web/test_introducer.py ¶

@@ -128 +128 @@
             self,
             soup,
-            u"%s: %s" % (allmydata.__appname__, allmydata.__version__),
+            allmydata.__full_version__,
         )
         assert_soup_has_text(self, soup, u"no peers!")

src/allmydata/test/web/test_web.py ¶

@@ -7 +7 @@
 from bs4 import BeautifulSoup
 
+from twisted.python.filepath import (
+    FilePath,
+)
 from twisted.application import service
 from twisted.internet import defer
@@ -317 +320 @@
         self.clock = Clock()
         self.fakeTime = 86460 # 1d 0h 1m 0s
-        self.ws = webish.WebishServer(self.s, "0", staticdir=self.staticdir,
-                                      clock=self.clock, now_fn=lambda:self.fakeTime)
+        tempdir = FilePath(self.mktemp())
+        tempdir.makedirs()
+        self.ws = webish.WebishServer(
+            self.s,
+            "0",
+            tempdir=tempdir.path,
+            staticdir=self.staticdir,
+            clock=self.clock,
+            now_fn=lambda:self.fakeTime,
+        )
         self.ws.setServiceParent(self.s)
         self.webish_port = self.ws.getPortnum()

src/allmydata/test/web/test_webish.py ¶

@@ -5 +5 @@
 from uuid import (
     uuid4,
+)
+from errno import (
+    EACCES,
+)
+from io import (
+    BytesIO,
+)
+
+from hypothesis import (
+    given,
+)
+from hypothesis.strategies import (
+    integers,
 )
 
@@ -13 +26 @@
     MatchesAll,
     Not,
-)
-
+    IsInstance,
+    HasLength,
+)
+
+from twisted.python.runtime import (
+    platform,
+)
 from twisted.python.filepath import (
     FilePath,
@@ -31 +49 @@
 from ...webish import (
     TahoeLAFSRequest,
-    tahoe_lafs_site,
+    TahoeLAFSSite,
 )
 
@@ -97 +115 @@
 class TahoeLAFSSiteTests(SyncTestCase):
     """
-    Tests for the ``Site`` created by ``tahoe_lafs_site``.
+    Tests for ``TahoeLAFSSite``.
     """
     def _test_censoring(self, path, censored):
@@ -113 +131 @@
         logPath = self.mktemp()
 
-        site = tahoe_lafs_site(Resource(), logPath=logPath)
+        site = TahoeLAFSSite(self.mktemp(), Resource(), logPath=logPath)
         site.startFactory()
 
@@ -170 +188 @@
             b"/uri?uri=[CENSORED]",
         )
+
+    def _create_request(self, tempdir):
+        """
+        Create and return a new ``TahoeLAFSRequest`` hooked up to a
+        ``TahoeLAFSSite``.
+
+        :param bytes tempdir: The temporary directory to give to the site.
+
+        :return TahoeLAFSRequest: The new request instance.
+        """
+        site = TahoeLAFSSite(tempdir.path, Resource(), logPath=self.mktemp())
+        site.startFactory()
+
+        channel = DummyChannel()
+        channel.site = site
+        request = TahoeLAFSRequest(channel)
+        return request
+
+    @given(integers(min_value=0, max_value=1024 * 1024 - 1))
+    def test_small_content(self, request_body_size):
+        """
+        A request body smaller than 1 MiB is kept in memory.
+        """
+        tempdir = FilePath(self.mktemp())
+        request = self._create_request(tempdir)
+        request.gotLength(request_body_size)
+        self.assertThat(
+            request.content,
+            IsInstance(BytesIO),
+        )
+
+    def _large_request_test(self, request_body_size):
+        """
+        Assert that when a request with a body of of the given size is received
+        its content is written to the directory the ``TahoeLAFSSite`` is
+        configured with.
+        """
+        tempdir = FilePath(self.mktemp())
+        tempdir.makedirs()
+        request = self._create_request(tempdir)
+
+        # So.  Bad news.  The temporary file for the uploaded content is
+        # unnamed (and this isn't even necessarily a bad thing since it is how
+        # you get automatic on-process-exit cleanup behavior on POSIX).  It's
+        # not visible by inspecting the filesystem.  It has no name we can
+        # discover.  Then how do we verify it is written to the right place?
+        # The question itself is meaningless if we try to be too precise.  It
+        # *has* no filesystem location.  However, it is still stored *on* some
+        # filesystem.  We still want to make sure it is on the filesystem we
+        # specified because otherwise it might be on a filesystem that's too
+        # small or undesirable in some other way.
+        #
+        # I don't know of any way to ask a file descriptor which filesystem
+        # it's on, either, though.  It might be the case that the [f]statvfs()
+        # result could be compared somehow to infer the filesystem but
+        # ... it's not clear what the failure modes might be there, across
+        # different filesystems and runtime environments.
+        #
+        # Another approach is to make the temp directory unwriteable and
+        # observe the failure when an attempt is made to create a file there.
+        # This is hardly a lovely solution but at least it's kind of simple.
+        #
+        # It would be nice if it worked consistently cross-platform but on
+        # Windows os.chmod is more or less broken.
+        if platform.isWindows():
+            request.gotLength(request_body_size)
+            self.assertThat(
+                tempdir.children(),
+                HasLength(1),
+            )
+        else:
+            tempdir.chmod(0o550)
+            with self.assertRaises(OSError) as ctx:
+                request.gotLength(request_body_size)
+                raise Exception(
+                    "OSError not raised, instead tempdir.children() = {}".format(
+                        tempdir.children(),
+                    ),
+                )
+
+            self.assertThat(
+                ctx.exception.errno,
+                Equals(EACCES),
+            )
+
+    def test_unknown_request_size(self):
+        """
+        A request body with an unknown size is written to a file in the temporary
+        directory passed to ``TahoeLAFSSite``.
+        """
+        self._large_request_test(None)
+
+    @given(integers(min_value=1024 * 1024))
+    def test_large_request(self, request_body_size):
+        """
+        A request body of 1 MiB or more is written to a file in the temporary
+        directory passed to ``TahoeLAFSSite``.
+        """
+        self._large_request_test(request_body_size)
+
 
 def param(name, value):

src/allmydata/util/_python3.py ¶

@@ -57 +57 @@
     "allmydata.mutable.common",
     "allmydata.mutable.filenode",
+    "allmydata.mutable.layout",
     "allmydata.mutable.publish",
+    "allmydata.mutable.repairer",
+    "allmydata.mutable.retrieve",
+    "allmydata.mutable.servermap",
     "allmydata.node",
     "allmydata.storage_client",
@@ -156 +160 @@
     "allmydata.test.test_uri",
     "allmydata.test.test_util",
-    "allmydata.test.test_version",
 ]

src/allmydata/util/log.py ¶

@@ -12 +12 @@
 if PY2:
     from builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
+from six import ensure_str
 
 from pyutil import nummedobj
@@ -56 +57 @@
             if pmsgid is None:
                 pmsgid = self._grandparentmsgid
+        kwargs = {ensure_str(k): v for (k, v) in kwargs.items()}
         msgid = log.msg(msg, facility=facility, parent=pmsgid, *args, **kwargs)
         if self._parentmsgid is None:

src/allmydata/web/introweb.py ¶

@@ -7 +7 @@
 import allmydata
 import json
-from allmydata.version_checks import get_package_versions_string
 from allmydata.util import idlib
 from allmydata.web.common import (
@@ -90 +89 @@
         self.node_data_dict = {
             "my_nodeid": idlib.nodeid_b2a(self.introducer_node.nodeid),
-            "version": get_package_versions_string(),
+            "version": allmydata.__full_version__,
             "import_path": str(allmydata).replace("/", "/ "),  # XXX kludge for wrapping
             "rendered_at": render_time(time.time()),

src/allmydata/web/root.py ¶

@@ -22 +22 @@
 
 import allmydata # to display import path
-from allmydata.version_checks import get_package_versions_string
 from allmydata.util import log
 from allmydata.interfaces import IFileNode
@@ -567 +566 @@
     @renderer
     def version(self, req, tag):
-        return tag(get_package_versions_string())
+        return tag(allmydata.__full_version__)
 
     @renderer

src/allmydata/webish.py ¶

@@ -1 +1 @@
 from six import ensure_str
 
-import re, time
-
-from functools import (
-    partial,
-)
+import re, time, tempfile
+
 from cgi import (
     FieldStorage,
+)
+from io import (
+    BytesIO,
 )
 
@@ -151 +151 @@
 
 
-tahoe_lafs_site = partial(
-    Site,
-    requestFactory=TahoeLAFSRequest,
-    logFormatter=_logFormatter,
-)
+class TahoeLAFSSite(Site, object):
+    """
+    The HTTP protocol factory used by Tahoe-LAFS.
+
+    Among the behaviors provided:
+
+    * A configurable temporary directory where large request bodies can be
+      written so they don't stay in memory.
+
+    * A log formatter that writes some access logs but omits capability
+      strings to help keep them secret.
+    """
+    requestFactory = TahoeLAFSRequest
+
+    def __init__(self, tempdir, *args, **kwargs):
+        Site.__init__(self, *args, logFormatter=_logFormatter, **kwargs)
+        self._tempdir = tempdir
+
+    def getContentFile(self, length):
+        if length is None or length >= 1024 * 1024:
+            return tempfile.TemporaryFile(dir=self._tempdir)
+        return BytesIO()
 
 
@@ -161 +178 @@
     name = "webish"
 
-    def __init__(self, client, webport, nodeurl_path=None, staticdir=None,
+    def __init__(self, client, webport, tempdir, nodeurl_path=None, staticdir=None,
                  clock=None, now_fn=time.time):
         service.MultiService.__init__(self)
@@ -171 +188 @@
 
         self.root = root.Root(client, clock, now_fn)
-        self.buildServer(webport, nodeurl_path, staticdir)
+        self.buildServer(webport, tempdir, nodeurl_path, staticdir)
 
         # If set, clock is a twisted.internet.task.Clock that the tests
@@ -181 +198 @@
         self.root.putChild(b"storage-plugins", StoragePlugins(client))
 
-    def buildServer(self, webport, nodeurl_path, staticdir):
+    def buildServer(self, webport, tempdir, nodeurl_path, staticdir):
         self.webport = webport
-        self.site = tahoe_lafs_site(self.root)
+        self.site = TahoeLAFSSite(tempdir, self.root)
         self.staticdir = staticdir # so tests can check
         if staticdir:
@@ -261 +278 @@
         service.MultiService.__init__(self)
         self.root = introweb.IntroducerRoot(introducer)
-        self.buildServer(webport, nodeurl_path, staticdir)
+        self.buildServer(webport, tempfile.tempdir, nodeurl_path, staticdir)