Changeset c385e95 in trunk

Timestamp:

2020-04-23T18:24:00Z (5 years ago)

Author:

meejah <meejah@…>

Branches:

master

Children:

4dc3702

Parents:

8df1ed1b (diff), bd8bf0f (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.

Message:

Merge branch 'master' into ticket3252-port-web-directory.remaining.1

Files:

• .circleci/Dockerfile.pypy (modified) (1 diff)
• .circleci/config.yml (modified) (4 diffs)
• .github/workflows/ci.yml (modified) (1 diff)
• .gitignore (modified) (1 diff)
• Makefile (modified) (1 diff)
• docs/configuration.rst (modified) (2 diffs)
• docs/frontends/magic-folder.rst (deleted)
• docs/index.rst (modified) (2 diffs)
• docs/magic-folder-howto.rst (deleted)
• docs/magic-wormhole-invites.rst (modified) (1 diff)
• docs/proposed/index.rst (modified) (1 diff)
• docs/proposed/magic-folder/filesystem-integration.rst (deleted)
• docs/proposed/magic-folder/multi-party-conflict-detection.rst (deleted)
• docs/proposed/magic-folder/remote-to-local-sync.rst (deleted)
• docs/proposed/magic-folder/user-interface-design.rst (deleted)
• integration/conftest.py (modified) (3 diffs)
• integration/test_aaa_aardvark.py (modified) (1 diff)
• integration/test_magic_folder.py (deleted)
• integration/test_tor.py (modified) (1 diff)
• integration/util.py (modified) (1 diff)
• newsfragments/3284.removed (added)
• newsfragments/3292.minor (moved) (moved from src/allmydata/watchdog/__init__.py)
• newsfragments/3299.minor (added)
• newsfragments/3300.minor (added)
• newsfragments/3302.minor (added)
• newsfragments/3303.minor (added)
• nix/eliot.nix (modified) (1 diff)
• setup.py (modified) (2 diffs)
• src/allmydata/client.py (modified) (6 diffs)
• src/allmydata/frontends/magic_folder.py (deleted)
• src/allmydata/magicfolderdb.py (deleted)
• src/allmydata/magicpath.py (deleted)
• src/allmydata/scripts/magic_folder_cli.py (deleted)
• src/allmydata/scripts/runner.py (modified) (3 diffs)
• src/allmydata/test/__init__.py (modified) (2 diffs)
• src/allmydata/test/cli/test_magic_folder.py (deleted)
• src/allmydata/test/test_client.py (modified) (2 diffs)
• src/allmydata/test/test_inotify.py (deleted)
• src/allmydata/test/test_magic_folder.py (deleted)
• src/allmydata/test/test_magicpath.py (deleted)
• src/allmydata/test/test_python2_regressions.py (modified) (1 diff)
• src/allmydata/test/test_websocket_logs.py (modified) (1 diff)
• src/allmydata/test/web/test_web.py (modified) (6 diffs)
• src/allmydata/util/eliotutil.py (modified) (4 diffs)
• src/allmydata/util/fake_inotify.py (deleted)
• src/allmydata/watchdog/_watchdog_541.py (deleted)
• src/allmydata/watchdog/inotify.py (deleted)
• src/allmydata/web/magic_folder.py (deleted)
• src/allmydata/web/root.py (modified) (3 diffs)
• src/allmydata/web/static/css/new-tahoe.css (modified) (1 diff)
• src/allmydata/web/statistics.xhtml (modified) (3 diffs)
• src/allmydata/web/status.py (modified) (3 diffs)
• src/allmydata/web/welcome.xhtml (modified) (1 diff)
• src/allmydata/windows/inotify.py (deleted)

.circleci/Dockerfile.pypy ¶

@@ -1 @@
-FROM pypy:2.7-7.1.1-jessie
+FROM pypy:2.7-buster
 
 ENV WHEELHOUSE_PATH /tmp/wheelhouse

.circleci/config.yml ¶

@@ -28 +28 @@
       - "nixos-19.09"
 
-      # Test against PyPy 2.7/7.1.1
-      - "pypy2.7-7.1"
+      # Test against PyPy 2.7
+      - "pypy2.7-buster"
 
       # Other assorted tasks and configurations
@@ -70 +70 @@
       - "build-image-centos-8"
       - "build-image-slackware-14.2"
-      - "build-image-pypy-2.7-7.1.1-jessie"
+      - "build-image-pypy-2.7-buster"
 
 
@@ -199 +199 @@
 
 
-  pypy2.7-7.1:
-    <<: *DEBIAN
-    docker:
-      - image: "tahoelafsci/pypy:2.7-7.1.1-jessie"
+  pypy2.7-buster:
+    <<: *DEBIAN
+    docker:
+      - image: "tahoelafsci/pypy:2.7-buster"
         user: "nobody"
 
@@ -514 +514 @@
 
 
-  build-image-pypy-2.7-7.1.1-jessie:
+  build-image-pypy-2.7-buster:
     <<: *BUILD_IMAGE
 
     environment:
       DISTRO: "pypy"
-      TAG: "2.7-7.1.1-jessie"
+      TAG: "2.7-buster"

.github/workflows/ci.yml ¶

@@ -3 +3 @@
 on:
   push:
+    branches:
+      - "master"
   pull_request:
 

.gitignore ¶

@@ -44 +44 @@
 /docs/_build/
 /coverage.xml
-/smoke_magicfolder/
 /.hypothesis/
 

Makefile ¶

@@ -42 +42 @@
         #   echo not uploading tahoe-lafs-osx-pkg because this is not trunk but is branch \"${BB_BRANCH}\" ; \
         # fi
-
-.PHONY: smoketest
-smoketest:
-        -python ./src/allmydata/test/check_magicfolder_smoke.py kill
-        -rm -rf smoke_magicfolder/
-        python ./src/allmydata/test/check_magicfolder_smoke.py
 
 # code coverage-based testing is disabled temporarily, as we switch to tox.

docs/configuration.rst ¶

@@ -83 +83 @@
 * SFTP service
 * FTP service
-* Magic Folder service
 * helper service
 * storage service.
@@ -719 +718 @@
     for instructions on configuring these services, and the ``[sftpd]`` and
     ``[ftpd]`` sections of ``tahoe.cfg``.
-
-Magic Folder
-
-    A node running on Linux or Windows can be configured to automatically
-    upload files that are created or changed in a specified local directory.
-    See :doc:`frontends/magic-folder` for details.
 
 

docs/index.rst ¶

@@ -21 +21 @@
    frontends/webapi
    frontends/FTP-and-SFTP
-   frontends/magic-folder
    frontends/download-status
 
@@ -38 +37 @@
    cautions
    write_coordination
-   magic-folder-howto
    backupdb
 

docs/magic-wormhole-invites.rst ¶

@@ -20 +20 @@
 Inside Tahoe-LAFS we are using a channel created using `magic
 wormhole`_ to exchange configuration and the secret fURL of the
-Introducer with new clients. In the future, we would like to make the
-Magic Folder (:ref:`Magic Folder HOWTO <magic-folder-howto>`) invites and joins work this way
-as well.
+Introducer with new clients.
 
 This is a two-part process. Alice runs a grid and wishes to have her

docs/proposed/index.rst ¶

@@ -15 +15 @@
 
    leasedb
-   magic-folder/filesystem-integration
-   magic-folder/remote-to-local-sync
-   magic-folder/user-interface-design
-   magic-folder/multi-party-conflict-detection
    http-storage-node-protocol

integration/conftest.py ¶

@@ -333 +333 @@
 @log_call(action_type=u"integration:alice", include_args=[], include_result=False)
 def alice(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, request):
-    try:
-        mkdir(join(temp_dir, 'magic-alice'))
-    except OSError:
-        pass
-
     process = pytest_twisted.blockon(
         _create_node(
@@ -352 +347 @@
 @log_call(action_type=u"integration:bob", include_args=[], include_result=False)
 def bob(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, request):
-    try:
-        mkdir(join(temp_dir, 'magic-bob'))
-    except OSError:
-        pass
-
     process = pytest_twisted.blockon(
         _create_node(
@@ -366 +356 @@
     await_client_ready(process)
     return process
-
-
-@pytest.fixture(scope='session')
-@log_call(action_type=u"integration:alice:invite", include_args=["temp_dir"])
-def alice_invite(reactor, alice, temp_dir, request):
-    node_dir = join(temp_dir, 'alice')
-
-    with start_action(action_type=u"integration:alice:magic_folder:create"):
-        # FIXME XXX by the time we see "client running" in the logs, the
-        # storage servers aren't "really" ready to roll yet (uploads fairly
-        # consistently fail if we don't hack in this pause...)
-        proto = _CollectOutputProtocol()
-        _tahoe_runner_optional_coverage(
-            proto,
-            reactor,
-            request,
-            [
-                'magic-folder', 'create',
-                '--poll-interval', '2',
-                '--basedir', node_dir, 'magik:', 'alice',
-                join(temp_dir, 'magic-alice'),
-            ]
-        )
-        pytest_twisted.blockon(proto.done)
-
-    with start_action(action_type=u"integration:alice:magic_folder:invite") as a:
-        proto = _CollectOutputProtocol()
-        _tahoe_runner_optional_coverage(
-            proto,
-            reactor,
-            request,
-            [
-                'magic-folder', 'invite',
-                '--basedir', node_dir, 'magik:', 'bob',
-            ]
-        )
-        pytest_twisted.blockon(proto.done)
-        invite = proto.output.getvalue()
-        a.add_success_fields(invite=invite)
-
-    with start_action(action_type=u"integration:alice:magic_folder:restart"):
-        # before magic-folder works, we have to stop and restart (this is
-        # crappy for the tests -- can we fix it in magic-folder?)
-        try:
-            alice.transport.signalProcess('TERM')
-            pytest_twisted.blockon(alice.transport.exited)
-        except ProcessExitedAlready:
-            pass
-        with start_action(action_type=u"integration:alice:magic_folder:magic-text"):
-            magic_text = 'Completed initial Magic Folder scan successfully'
-            pytest_twisted.blockon(_run_node(reactor, node_dir, request, magic_text))
-            await_client_ready(alice)
-    return invite
-
-
-@pytest.fixture(scope='session')
-@log_call(
-    action_type=u"integration:magic_folder",
-    include_args=["alice_invite", "temp_dir"],
-)
-def magic_folder(reactor, alice_invite, alice, bob, temp_dir, request):
-    print("pairing magic-folder")
-    bob_dir = join(temp_dir, 'bob')
-    proto = _CollectOutputProtocol()
-    _tahoe_runner_optional_coverage(
-        proto,
-        reactor,
-        request,
-        [
-            'magic-folder', 'join',
-            '--poll-interval', '1',
-            '--basedir', bob_dir,
-            alice_invite,
-            join(temp_dir, 'magic-bob'),
-        ]
-    )
-    pytest_twisted.blockon(proto.done)
-
-    # before magic-folder works, we have to stop and restart (this is
-    # crappy for the tests -- can we fix it in magic-folder?)
-    try:
-        print("Sending TERM to Bob")
-        bob.transport.signalProcess('TERM')
-        pytest_twisted.blockon(bob.transport.exited)
-    except ProcessExitedAlready:
-        pass
-
-    magic_text = 'Completed initial Magic Folder scan successfully'
-    pytest_twisted.blockon(_run_node(reactor, bob_dir, request, magic_text))
-    await_client_ready(bob)
-    return (join(temp_dir, 'magic-alice'), join(temp_dir, 'magic-bob'))
 
 

integration/test_aaa_aardvark.py ¶

@@ -17 +17 @@
 def test_create_storage(storage_nodes):
     print("Created {} storage nodes".format(len(storage_nodes)))
-
-
-def test_create_alice_bob_magicfolder(magic_folder):
-    print("Alice and Bob have paired magic-folders")

integration/test_tor.py ¶

@@ -15 +15 @@
 import util
 
-# see "conftest.py" for the fixtures (e.g. "magic_folder")
+# see "conftest.py" for the fixtures (e.g. "tor_network")
 
 @pytest_twisted.inlineCallbacks

integration/util.py ¶

@@ -499 +499 @@
         )
     )
-
-
-def magic_folder_cli(request, reactor, node_dir, *argv):
-    return cli(request, reactor, node_dir, "magic-folder", *argv)

nix/eliot.nix ¶

@@ -13 +13 @@
     substituteInPlace setup.py \
       --replace "boltons >= 19.0.1" boltons
-    # depends on eliot.prettyprint._main which we don't have here.
-    rm eliot/tests/test_prettyprint.py
+  '';
 
-    # Fails intermittently.
-    substituteInPlace eliot/tests/test_validation.py \
-      --replace "def test_omitLoggerFromActionType" "def xtest_omitLoggerFromActionType" \
-      --replace "def test_logCallsDefaultLoggerWrite" "def xtest_logCallsDefaultLoggerWrite"
-  '';
+  # A seemingly random subset of the test suite fails intermittently.  After
+  # Tahoe-LAFS is ported to Python 3 we can update to a newer Eliot and, if
+  # the test suite continues to fail, maybe it will be more likely that we can
+  # have upstream fix it for us.
+  doCheck = false;
 
   checkInputs = [ testtools pytest hypothesis ];

setup.py ¶

@@ -63 +63 @@
     "cryptography >= 2.6",
 
-    # * On Linux we need at least Twisted 10.1.0 for inotify support
-    #   used by the drop-upload frontend.
-    # * We also need Twisted 10.1.0 for the FTP frontend in order for
+    # * We need Twisted 10.1.0 for the FTP frontend in order for
     #   Twisted's FTP server to support asynchronous close.
     # * The SFTP frontend depends on Twisted 11.0.0 to fix the SSH server
@@ -355 +353 @@
           # discussion.
           ':sys_platform=="win32"': ["pywin32 != 226"],
-          ':sys_platform!="win32" and sys_platform!="linux2"': ["watchdog"],  # For magic-folder on "darwin" (macOS) and the BSDs
           "test": [
               # Pin a specific pyflakes so we don't have different folks

src/allmydata/client.py ¶

@@ -85 +85 @@
             "stats_gatherer.furl",
             "storage.plugins",
-        ),
-        "drop_upload": (  # deprecated already?
-            "enabled",
         ),
         "ftpd": (
@@ -122 +119 @@
             "enabled",
         ),
-        "magic_folder": (
-            "download.umask",
-            "enabled",
-            "local.directory",
-            "poll_interval",
-        ),
     },
     is_valid_section=_is_valid_section,
@@ -682 +673 @@
         node.Node.__init__(self, config, main_tub, control_tub, i2p_provider, tor_provider)
 
-        self._magic_folders = dict()
         self.started_timestamp = time.time()
         self.logSource = "Client"
@@ -708 +698 @@
         self.init_ftp_server()
         self.init_sftp_server()
-        self.init_magic_folder()
 
         # If the node sees an exit_trigger file, it will poll every second to see
@@ -969 +958 @@
         random data in "api_auth_token" which must be echoed to API
         calls.
-
-        Currently only the URI '/magic' for magic-folder status; other
-        endpoints are invited to include this as well, as appropriate.
         """
         return self.config.get_private_config('api_auth_token')
@@ -1089 +1075 @@
             s.setServiceParent(self)
 
-    def init_magic_folder(self):
-        #print "init_magic_folder"
-        if self.config.get_config("drop_upload", "enabled", False, boolean=True):
-            raise node.OldConfigOptionError(
-                "The [drop_upload] section must be renamed to [magic_folder].\n"
-                "See docs/frontends/magic-folder.rst for more information."
-            )
-
-        if self.config.get_config("magic_folder", "enabled", False, boolean=True):
-            from allmydata.frontends import magic_folder
-
-            try:
-                magic_folders = magic_folder.load_magic_folders(self.config._basedir)
-            except Exception as e:
-                log.msg("Error loading magic-folder config: {}".format(e))
-                raise
-
-            # start processing the upload queue when we've connected to
-            # enough servers
-            threshold = min(self.encoding_params["k"],
-                            self.encoding_params["happy"] + 1)
-
-            for (name, mf_config) in magic_folders.items():
-                self.log("Starting magic_folder '{}'".format(name))
-                s = magic_folder.MagicFolder.from_config(self, name, mf_config)
-                self._magic_folders[name] = s
-                s.setServiceParent(self)
-
-                connected_d = self.storage_broker.when_connected_enough(threshold)
-                def connected_enough(ign, mf):
-                    mf.ready()  # returns a Deferred we ignore
-                    return None
-                connected_d.addCallback(connected_enough, s)
-
     def _check_exit_trigger(self, exit_trigger_file):
         if os.path.exists(exit_trigger_file):

src/allmydata/scripts/runner.py ¶

@@ -10 +10 @@
 from allmydata.scripts.common import get_default_nodedir
 from allmydata.scripts import debug, create_node, cli, \
-    stats_gatherer, admin, magic_folder_cli, tahoe_daemonize, tahoe_start, \
+    stats_gatherer, admin, tahoe_daemonize, tahoe_start, \
     tahoe_stop, tahoe_restart, tahoe_run, tahoe_invite
 from allmydata.util.encodingutil import quote_output, quote_local_unicode_path, get_io_encoding
@@ -62 +62 @@
                     +   debug.subCommands
                     +   cli.subCommands
-                    +   magic_folder_cli.subCommands
                     +   tahoe_invite.subCommands
                     )
@@ -155 +154 @@
         f0 = cli.dispatch[command]
         f = lambda so: threads.deferToThread(f0, so)
-    elif command in magic_folder_cli.dispatch:
-        # same
-        f0 = magic_folder_cli.dispatch[command]
-        f = lambda so: threads.deferToThread(f0, so)
     elif command in tahoe_invite.dispatch:
         f = tahoe_invite.dispatch[command]

src/allmydata/test/__init__.py ¶

@@ +1 @@
+# -*- coding: utf-8 -*-
+# Tahoe-LAFS -- secure, distributed storage grid
+#
+# Copyright © 2020 The Tahoe-LAFS Software Foundation
+#
+# This file is part of Tahoe-LAFS.
+#
+# See the docs/about.rst file for licensing information.
+
+"""
+Some setup that should apply across the entire test suite.
+
+Rather than defining interesting APIs for other code to use, this just causes
+some side-effects which make things better when the test suite runs.
+"""
+
+from traceback import extract_stack, format_list
+from foolscap.pb import Listener
+from twisted.python.log import err
+from twisted.application import service
+
 
 from foolscap.logging.incident import IncidentQualifier
@@ -53 +74 @@
 _configure_hypothesis()
 
+def logging_for_pb_listener():
+    """
+    Make Foolscap listen error reports include Listener creation stack
+    information.
+    """
+    original__init__ = Listener.__init__
+    def _listener__init__(self, *a, **kw):
+        original__init__(self, *a, **kw)
+        # Capture the stack here, where Listener is instantiated.  This is
+        # likely to explain what code is responsible for this Listener, useful
+        # information to have when the Listener eventually fails to listen.
+        self._creation_stack = extract_stack()
+
+    # Override the Foolscap implementation with one that has an errback
+    def _listener_startService(self):
+        service.Service.startService(self)
+        d = self._ep.listen(self)
+        def _listening(lp):
+            self._lp = lp
+        d.addCallbacks(
+            _listening,
+            # Make sure that this listen failure is reported promptly and with
+            # the creation stack.
+            err,
+            errbackArgs=(
+                "Listener created at {}".format(
+                    "".join(format_list(self._creation_stack)),
+                ),
+            ),
+        )
+    Listener.__init__ = _listener__init__
+    Listener.startService = _listener_startService
+logging_for_pb_listener()
 
 import sys

src/allmydata/test/test_client.py ¶

@@ -38 +38 @@
 
 import allmydata
-import allmydata.frontends.magic_folder
 import allmydata.util.log
 
-from allmydata.node import OldConfigError, OldConfigOptionError, UnescapedHashError, _Config, create_node_dir
+from allmydata.node import OldConfigError, UnescapedHashError, _Config, create_node_dir
 from allmydata.frontends.auth import NeedRootcapLookupScheme
 from allmydata.version_checks import (
@@ -659 +658 @@
         yield _check("helper.furl = pb://blah\n", "pb://blah")
 
-    @defer.inlineCallbacks
-    def test_create_magic_folder_service(self):
-        """
-        providing magic-folder options actually creates a MagicFolder service
-        """
-        boom = False
-        class Boom(Exception):
-            pass
-
-        class MockMagicFolder(allmydata.frontends.magic_folder.MagicFolder):
-            name = 'magic-folder'
-
-            def __init__(self, client, upload_dircap, collective_dircap, local_path_u, dbfile, umask, name,
-                         inotify=None, uploader_delay=1.0, clock=None, downloader_delay=3):
-                if boom:
-                    raise Boom()
-
-                service.MultiService.__init__(self)
-                self.client = client
-                self._umask = umask
-                self.upload_dircap = upload_dircap
-                self.collective_dircap = collective_dircap
-                self.local_dir = local_path_u
-                self.dbfile = dbfile
-                self.inotify = inotify
-
-            def startService(self):
-                self.running = True
-
-            def stopService(self):
-                self.running = False
-
-            def ready(self):
-                pass
-
-        self.patch(allmydata.frontends.magic_folder, 'MagicFolder', MockMagicFolder)
-
-        upload_dircap = "URI:DIR2:blah"
-        local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
-        local_dir_utf8 = local_dir_u.encode('utf-8')
-        config = (BASECONFIG +
-                  "[storage]\n" +
-                  "enabled = false\n" +
-                  "[magic_folder]\n" +
-                  "enabled = true\n")
-
-        basedir1 = "test_client.Basic.test_create_magic_folder_service1"
-        os.mkdir(basedir1)
-        os.mkdir(local_dir_u)
-
-        # which config-entry should be missing?
-        fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
-                       config + "local.directory = " + local_dir_utf8 + "\n")
-        with self.assertRaises(IOError):
-            yield client.create_client(basedir1)
-
-        # local.directory entry missing .. but that won't be an error
-        # now, it'll just assume there are not magic folders
-        # .. hrm...should we make that an error (if enabled=true but
-        # there's not yaml AND no local.directory?)
-        fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
-        fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow")
-
-        fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
-                       config.replace("[magic_folder]\n", "[drop_upload]\n"))
-
-        with self.assertRaises(OldConfigOptionError):
-            yield client.create_client(basedir1)
-
-        fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
-                       config + "local.directory = " + local_dir_utf8 + "\n")
-        c1 = yield client.create_client(basedir1)
-        magicfolder = c1.getServiceNamed('magic-folder')
-        self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder)
-        self.failUnlessReallyEqual(magicfolder.client, c1)
-        self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap)
-        self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u)
-        self.failUnless(magicfolder.inotify is None, magicfolder.inotify)
-        # It doesn't start until the client starts.
-        self.assertFalse(magicfolder.running)
-
-        # See above.
-        boom = True
-
-        basedir2 = "test_client.Basic.test_create_magic_folder_service2"
-        os.mkdir(basedir2)
-        os.mkdir(os.path.join(basedir2, "private"))
-        fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
-                       BASECONFIG +
-                       "[magic_folder]\n" +
-                       "enabled = true\n" +
-                       "local.directory = " + local_dir_utf8 + "\n")
-        fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow")
-        with self.assertRaises(Boom):
-            yield client.create_client(basedir2)
-
 
 def flush_but_dont_ignore(res):

src/allmydata/test/test_python2_regressions.py ¶

@@ -16 +16 @@
 BLACKLIST = {
     "allmydata.test.check_load",
-    "allmydata.watchdog._watchdog_541",
-    "allmydata.watchdog.inotify",
-    "allmydata.windows.inotify",
     "allmydata.windows.registry",
 }

src/allmydata/test/test_websocket_logs.py ¶

@@ -41 +41 @@
         proto.on("message", got_message)
 
-        @log_call(action_type=u"test:cli:magic-folder:cleanup")
+        @log_call(action_type=u"test:cli:some-exciting-action")
         def do_a_thing():
             pass

src/allmydata/test/web/test_web.py ¶

@@ -4 +4 @@
 import json
 import treq
-import mock
 
 from bs4 import BeautifulSoup
@@ -35 +34 @@
 from allmydata.dirnode import DirectoryNode
 from allmydata.nodemaker import NodeMaker
-from allmydata.frontends.magic_folder import QueuedItem
 from allmydata.web import status
 from allmydata.web.common import WebError, MultiFormatPage
@@ -67 +65 @@
 from allmydata.client import _Client, SecretHolder
 from .common import unknown_rwcap, unknown_rocap, unknown_immcap, FAVICON_MARKUP
-from ..status import FakeStatus
 
 # create a fake uploader/downloader, and a couple of fake dirnodes, then
@@ -128 +125 @@
 
 
-def create_test_queued_item(relpath_u, history=[]):
-    progress = mock.Mock()
-    progress.progress = 100.0
-    item = QueuedItem(relpath_u, progress, 1234)
-    for the_status, timestamp in history:
-        item.set_status(the_status, current_time=timestamp)
-    return item
-
-
-class FakeMagicFolder(object):
-    def __init__(self):
-        self.uploader = FakeStatus()
-        self.downloader = FakeStatus()
-
-    def get_public_status(self):
-        return (
-            True,
-            [
-                'a magic-folder status message'
-            ],
-        )
-
-
 def build_one_ds():
     ds = DownloadStatus("storage_index", 1234)
@@ -285 +259 @@
         # minimal subset
         service.MultiService.__init__(self)
-        self._magic_folders = dict()
         self.all_contents = {}
         self.nodeid = "fake_nodeid"
@@ -941 +914 @@
         d.addCallback(_check)
         return d
-
-    @defer.inlineCallbacks
-    def test_magicfolder_status_bad_token(self):
-        with self.assertRaises(Error):
-            yield self.POST(
-                '/magic_folder?t=json',
-                t='json',
-                name='default',
-                token='not the token you are looking for',
-            )
-
-    @defer.inlineCallbacks
-    def test_magicfolder_status_wrong_folder(self):
-        with self.assertRaises(Exception) as ctx:
-            yield self.POST(
-                '/magic_folder?t=json',
-                t='json',
-                name='a non-existent magic-folder',
-                token=self.s.get_auth_token(),
-            )
-        self.assertIn(
-            "Not Found",
-            str(ctx.exception)
-        )
-
-    @defer.inlineCallbacks
-    def test_magicfolder_status_success(self):
-        self.s._magic_folders['default'] = mf = FakeMagicFolder()
-        mf.uploader.status = [
-            create_test_queued_item(u"rel/uppath", [('done', 12345)])
-        ]
-        mf.downloader.status = [
-            create_test_queued_item(u"rel/downpath", [('done', 23456)])
-        ]
-        data = yield self.POST(
-            '/magic_folder?t=json',
-            t='json',
-            name='default',
-            token=self.s.get_auth_token(),
-        )
-        data = json.loads(data)
-        self.assertEqual(
-            data,
-            [
-                {
-                    "status": "done",
-                    "path": "rel/uppath",
-                    "kind": "upload",
-                    "percent_done": 100.0,
-                    "done_at": 12345,
-                    "size": 1234,
-                },
-                {
-                    "status": "done",
-                    "path": "rel/downpath",
-                    "kind": "download",
-                    "percent_done": 100.0,
-                    "done_at": 23456,
-                    "size": 1234,
-                },
-            ]
-        )
-
-    @defer.inlineCallbacks
-    def test_magicfolder_root_success(self):
-        self.s._magic_folders['default'] = mf = FakeMagicFolder()
-        mf.uploader.status = [
-            create_test_queued_item(u"rel/path", [('done', 12345)])
-        ]
-        data = yield self.GET(
-            '/',
-        )
-        del data
 
     def test_status(self):

src/allmydata/util/eliotutil.py ¶

@@ -17 +17 @@
     "validateInstanceOf",
     "validateSetMembership",
-    "MAYBE_NOTIFY",
-    "CALLBACK",
-    "INOTIFY_EVENTS",
-    "RELPATH",
-    "VERSION",
-    "LAST_UPLOADED_URI",
-    "LAST_DOWNLOADED_URI",
-    "LAST_DOWNLOADED_TIMESTAMP",
-    "PATHINFO",
 ]
 
@@ -52 +43 @@
     ILogger,
     Message,
-    Field,
-    ActionType,
     FileDestination,
     add_destinations,
@@ -87 +76 @@
 from twisted.application.service import Service
 
-
-from .fileutil import (
-    PathInfo,
-)
-from .fake_inotify import (
-    humanReadableMask,
-)
-
 def validateInstanceOf(t):
     """
@@ -112 +93 @@
             raise ValidationError("{} not in {}".format(v, s))
     return validator
-
-RELPATH = Field.for_types(
-    u"relpath",
-    [unicode],
-    u"The relative path of a file in a magic-folder.",
-)
-
-VERSION = Field.for_types(
-    u"version",
-    [int, long],
-    u"The version of the file.",
-)
-
-LAST_UPLOADED_URI = Field.for_types(
-    u"last_uploaded_uri",
-    [unicode, bytes, None],
-    u"The filecap to which this version of this file was uploaded.",
-)
-
-LAST_DOWNLOADED_URI = Field.for_types(
-    u"last_downloaded_uri",
-    [unicode, bytes, None],
-    u"The filecap from which the previous version of this file was downloaded.",
-)
-
-LAST_DOWNLOADED_TIMESTAMP = Field.for_types(
-    u"last_downloaded_timestamp",
-    [float, int, long],
-    u"(XXX probably not really, don't trust this) The timestamp of the last download of this file.",
-)
-
-PATHINFO = Field(
-    u"pathinfo",
-    lambda v: None if v is None else {
-        "isdir": v.isdir,
-        "isfile": v.isfile,
-        "islink": v.islink,
-        "exists": v.exists,
-        "size": v.size,
-        "mtime_ns": v.mtime_ns,
-        "ctime_ns": v.ctime_ns,
-    },
-    u"The metadata for this version of this file.",
-    validateInstanceOf((type(None), PathInfo)),
-)
-
-INOTIFY_EVENTS = Field(
-    u"inotify_events",
-    humanReadableMask,
-    u"Details about a filesystem event generating a notification event.",
-    validateInstanceOf((int, long)),
-)
-
-MAYBE_NOTIFY = ActionType(
-    u"filesystem:notification:maybe-notify",
-    [],
-    [],
-    u"A filesystem event is being considered for dispatch to an application handler.",
-)
-
-CALLBACK = ActionType(
-    u"filesystem:notification:callback",
-    [INOTIFY_EVENTS],
-    [],
-    u"A filesystem event is being dispatched to an application callback."
-)
 
 def eliot_logging_service(reactor, destinations):

src/allmydata/web/root.py ¶

@@ -22 +22 @@
 from allmydata.interfaces import IFileNode
 from allmydata.web import filenode, directory, unlinked, status
-from allmydata.web import storage, magic_folder
+from allmydata.web import storage
 from allmydata.web.common import (
     abbreviate_size,
@@ -209 +209 @@
         self.putChild("cap", URIHandler(client))
 
-        # handler for "/magic_folder" URIs
-        self.putChild("magic_folder", magic_folder.MagicFolderWebApi(client))
-
         # Handler for everything beneath "/private", an area of the resource
         # hierarchy which is only accessible with the private per-node API
@@ -307 +304 @@
         return description
 
-
-    def data_magic_folders(self, ctx, data):
-        return self.client._magic_folders.keys()
-
-    def render_magic_folder_row(self, ctx, data):
-        magic_folder = self.client._magic_folders[data]
-        (ok, messages) = magic_folder.get_public_status()
-        ctx.fillSlots("magic_folder_name", data)
-        if ok:
-            ctx.fillSlots("magic_folder_status", "yes")
-            ctx.fillSlots("magic_folder_status_alt", "working")
-        else:
-            ctx.fillSlots("magic_folder_status", "no")
-            ctx.fillSlots("magic_folder_status_alt", "not working")
-
-        status = T.ul(class_="magic-folder-status")
-        for msg in messages:
-            status[T.li[str(msg)]]
-        return ctx.tag[status]
-
-    def render_magic_folder(self, ctx, data):
-        if not self.client._magic_folders:
-            return T.p()
-        return ctx.tag
 
     def render_services(self, ctx, data):

src/allmydata/web/static/css/new-tahoe.css ¶

@@ -54 +54 @@
 }
 
-.magic-folder-status {
-    clear: left;
-    margin-left: 40px;  /* width of status-indicator + margins */
-}
-
 .furl {
     font-size: 0.8em;

src/allmydata/web/statistics.xhtml ¶

@@ -1 @@
-<html xmlns:n="http://nevow.com/ns/nevow/0.1">
+<html xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
   <head>
     <title>Tahoe-LAFS - Operational Statistics</title>
@@ -6 +6 @@
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
   </head>
-  <body n:data="get_stats">
+  <body>
 
 <h1>Operational Statistics</h1>
@@ -12 +12 @@
 <h2>General</h2>
 
-<ul>
-  <li>Load Average: <span n:render="load_average" /></li>
-  <li>Peak Load: <span n:render="peak_load" /></li>
-  <li>Files Uploaded (immutable): <span n:render="uploads" /></li>
-  <li>Files Downloaded (immutable): <span n:render="downloads" /></li>
-  <li>Files Published (mutable): <span n:render="publishes" /></li>
-  <li>Files Retrieved (mutable): <span n:render="retrieves" /></li>
-</ul>
-
-<h2>Magic Folder</h2>
-
-<ul>
-  <li>Local Directories Monitored: <span n:render="magic_uploader_monitored" /></li>
-  <li>Files Uploaded: <span n:render="magic_uploader_succeeded" /></li>
-  <li>Files Queued for Upload: <span n:render="magic_uploader_queued" /></li>
-  <li>Failed Uploads: <span n:render="magic_uploader_failed" /></li>
-  <li>Files Downloaded: <span n:render="magic_downloader_succeeded" /></li>
-  <li>Files Queued for Download: <span n:render="magic_downloader_queued" /></li>
-  <li>Failed Downloads: <span n:render="magic_downloader_failed" /></li>
-</ul>
+    <ul>
+      <li>Load Average: <t:transparent t:render="load_average" /></li>
+      <li>Peak Load: <t:transparent t:render="peak_load" /></li>
+      <li>Files Uploaded (immutable): <t:transparent t:render="uploads" /></li>
+      <li>Files Downloaded (immutable): <t:transparent t:render="downloads" /></li>
+      <li>Files Published (mutable): <t:transparent t:render="publishes" /></li>
+      <li>Files Retrieved (mutable): <t:transparent t:render="retrieves" /></li>
+    </ul>
 
 <h2>Raw Stats:</h2>
-<pre n:render="raw" />
+<pre t:render="raw" />
 
 <div>Return to the <a href="/">Welcome Page</a></div>

src/allmydata/web/status.py ¶

@@ -3 +3 @@
 import json
 from twisted.internet import defer
+from twisted.python.filepath import FilePath
 from twisted.web.resource import Resource
+from twisted.web.template import (
+    Element,
+    XMLFile,
+    renderer,
+    renderElement,
+)
 from nevow import rend, tags as T
 from allmydata.util import base32, idlib
@@ -15 +22 @@
     render_time,
     MultiFormatPage,
+    MultiFormatResource,
 )
 from allmydata.interfaces import IUploadStatus, IDownloadStatus, \
@@ -1165 +1173 @@
         return str(data["chk_upload_helper.encoded_bytes"])
 
-
-class Statistics(MultiFormatPage):
-    docFactory = getxmlfile("statistics.xhtml")
+# Render "/statistics" page.
+class Statistics(MultiFormatResource):
+    """Class that renders "/statistics" page.
+
+    :param _allmydata.stats.StatsProvider provider: node statistics
+           provider.
+    """
 
     def __init__(self, provider):
-        rend.Page.__init__(self, provider)
-        self.provider = provider
+        super(Statistics, self).__init__()
+        self._provider = provider
+
+    def render_HTML(self, req):
+        return renderElement(req, StatisticsElement(self._provider))
 
     def render_JSON(self, req):
-        stats = self.provider.get_stats()
+        stats = self._provider.get_stats()
         req.setHeader("content-type", "text/plain")
         return json.dumps(stats, indent=1) + "\n"
 
-    def data_get_stats(self, ctx, data):
-        return self.provider.get_stats()
-
-    def render_load_average(self, ctx, data):
-        return str(data["stats"].get("load_monitor.avg_load"))
-
-    def render_peak_load(self, ctx, data):
-        return str(data["stats"].get("load_monitor.max_load"))
-
-    def render_uploads(self, ctx, data):
-        files = data["counters"].get("uploader.files_uploaded", 0)
-        bytes = data["counters"].get("uploader.bytes_uploaded", 0)
-        return ("%s files / %s bytes (%s)" %
-                (files, bytes, abbreviate_size(bytes)))
-
-    def render_downloads(self, ctx, data):
-        files = data["counters"].get("downloader.files_downloaded", 0)
-        bytes = data["counters"].get("downloader.bytes_downloaded", 0)
-        return ("%s files / %s bytes (%s)" %
-                (files, bytes, abbreviate_size(bytes)))
-
-    def render_publishes(self, ctx, data):
-        files = data["counters"].get("mutable.files_published", 0)
-        bytes = data["counters"].get("mutable.bytes_published", 0)
-        return "%s files / %s bytes (%s)" % (files, bytes,
-                                             abbreviate_size(bytes))
-
-    def render_retrieves(self, ctx, data):
-        files = data["counters"].get("mutable.files_retrieved", 0)
-        bytes = data["counters"].get("mutable.bytes_retrieved", 0)
-        return "%s files / %s bytes (%s)" % (files, bytes,
-                                             abbreviate_size(bytes))
-
-    def render_magic_uploader_monitored(self, ctx, data):
-        dirs = data["counters"].get("magic_folder.uploader.dirs_monitored", 0)
-        return "%s directories" % (dirs,)
-
-    def render_magic_uploader_succeeded(self, ctx, data):
-        # TODO: bytes uploaded
-        files = data["counters"].get("magic_folder.uploader.objects_succeeded", 0)
-        return "%s files" % (files,)
-
-    def render_magic_uploader_queued(self, ctx, data):
-        files = data["counters"].get("magic_folder.uploader.objects_queued", 0)
-        return "%s files" % (files,)
-
-    def render_magic_uploader_failed(self, ctx, data):
-        files = data["counters"].get("magic_folder.uploader.objects_failed", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_succeeded(self, ctx, data):
-        # TODO: bytes uploaded
-        files = data["counters"].get("magic_folder.downloader.objects_succeeded", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_queued(self, ctx, data):
-        files = data["counters"].get("magic_folder.downloader.objects_queued", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_failed(self, ctx, data):
-        files = data["counters"].get("magic_folder.downloader.objects_failed", 0)
-        return "%s files" % (files,)
-
-    def render_raw(self, ctx, data):
-        raw = pprint.pformat(data)
-        return ctx.tag[raw]
+class StatisticsElement(Element):
+
+    loader = XMLFile(FilePath(__file__).sibling("statistics.xhtml"))
+
+    def __init__(self, provider):
+        super(StatisticsElement, self).__init__()
+        # provider.get_stats() returns a dict of the below form, for
+        # example (there's often more data than this):
+        #
+        #  {
+        #    'stats': {
+        #      'storage_server.disk_used': 809601609728,
+        #      'storage_server.accepting_immutable_shares': 1,
+        #      'storage_server.disk_free_for_root': 131486851072,
+        #      'storage_server.reserved_space': 1000000000,
+        #      'node.uptime': 0.16520118713378906,
+        #      'storage_server.disk_total': 941088460800,
+        #      'cpu_monitor.total': 0.004513999999999907,
+        #      'storage_server.disk_avail': 82610759168,
+        #      'storage_server.allocated': 0,
+        #      'storage_server.disk_free_for_nonroot': 83610759168 },
+        #    'counters': {
+        #      'uploader.files_uploaded': 0,
+        #      'uploader.bytes_uploaded': 0,
+        #       ... }
+        #  }
+        #
+        # Note that `counters` can be empty.
+        self._stats = provider.get_stats()
+
+    @renderer
+    def load_average(self, req, tag):
+        return tag(str(self._stats["stats"].get("load_monitor.avg_load")))
+
+    @renderer
+    def peak_load(self, req, tag):
+        return tag(str(self._stats["stats"].get("load_monitor.max_load")))
+
+    @renderer
+    def uploads(self, req, tag):
+        files = self._stats["counters"].get("uploader.files_uploaded", 0)
+        bytes = self._stats["counters"].get("uploader.bytes_uploaded", 0)
+        return tag(("%s files / %s bytes (%s)" %
+                    (files, bytes, abbreviate_size(bytes))))
+
+    @renderer
+    def downloads(self, req, tag):
+        files = self._stats["counters"].get("downloader.files_downloaded", 0)
+        bytes = self._stats["counters"].get("downloader.bytes_downloaded", 0)
+        return tag("%s files / %s bytes (%s)" %
+                   (files, bytes, abbreviate_size(bytes)))
+
+    @renderer
+    def publishes(self, req, tag):
+        files = self._stats["counters"].get("mutable.files_published", 0)
+        bytes = self._stats["counters"].get("mutable.bytes_published", 0)
+        return tag("%s files / %s bytes (%s)" % (files, bytes,
+                                                 abbreviate_size(bytes)))
+
+    @renderer
+    def retrieves(self, req, tag):
+        files = self._stats["counters"].get("mutable.files_retrieved", 0)
+        bytes = self._stats["counters"].get("mutable.bytes_retrieved", 0)
+        return tag("%s files / %s bytes (%s)" % (files, bytes,
+                                                 abbreviate_size(bytes)))
+
+    @renderer
+    def raw(self, req, tag):
+        raw = pprint.pformat(self._stats)
+        return tag(raw)

src/allmydata/web/welcome.xhtml ¶

@@ -160 +160 @@
           </div>
 
-          <div n:render="magic_folder" class="row-fluid">
-            <h2>Magic Folders</h2>
-            <div n:render="sequence" n:data="magic_folders">
-              <div n:pattern="item" n:render="magic_folder_row"><div class="status-indicator"><img><n:attr name="src">img/connected-<n:slot name="magic_folder_status" />.png</n:attr><n:attr name="alt"><n:slot name="magic_folder_status_alt" /></n:attr></img></div><h3><n:slot name="magic_folder_name" /></h3></div>
-            </div>
-          </div><!--/row-->
-
           <div class="row-fluid">
             <h2>