Changeset 624916e in trunk


Ignore:
Timestamp:
2020-12-11T00:47:47Z (4 years ago)
Author:
GitHub <noreply@…>
Branches:
master
Children:
51e5067
Parents:
8b890d25 (diff), d096cc54 (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.
git-author:
Jean-Paul Calderone <exarkun@…> (2020-12-11 00:47:47)
git-committer:
GitHub <noreply@…> (2020-12-11 00:47:47)
Message:

Merge branch 'master' into 3550.remove-start-stop-restart-daemonize

Files:
3 added
21 edited

Legend:

Unmodified
Added
Removed

  • TabularUnified docs/configuration.rst

    r8b890d25 r624916e  
    7676==========
    7777
    78 A node can be a client/server, an introducer, or a statistics gatherer.
     78A node can be a client/server or an introducer.
    7979
    8080Client/server nodes provide one or more of the following services:
    … …  
    593593    If provided, the node will attempt to connect to and use the given helper
    594594    for uploads. See :doc:`helper` for details.
    595 
    596 ``stats_gatherer.furl = (FURL string, optional)``
    597 
    598     If provided, the node will connect to the given stats gatherer and
    599     provide it with operational statistics.
    600595
    601596``shares.needed = (int, optional) aka "k", default 3``

  • TabularUnified docs/stats.rst

    r8b890d25 r624916e  
    243243        sometimes be negative due to wraparound of the kernel's counter.
    244244
    245 **stats.load_monitor.\***
    246 
    247     When enabled, the "load monitor" continually schedules a one-second
    248     callback, and measures how late the response is. This estimates system load
    249     (if the system is idle, the response should be on time). This is only
    250     enabled if a stats-gatherer is configured.
    251 
    252     avg_load
    253         average "load" value (seconds late) over the last minute
    254 
    255     max_load
    256         maximum "load" value over the last minute
    257 
    258245
    259246Using Munin To Graph Stats Values

  • TabularUnified newsfragments/3549.removed

    r8b890d25 r624916e  
    1 The stats gatherer has been removed.  The ``[client]stats_gatherer.furl`` configuration item in ``tahoe.cfg`` is no longer allowed.
     1The stats gatherer, broken since at least Tahoe-LAFS 1.13.0, has been removed.  The ``[client]stats_gatherer.furl`` configuration item in ``tahoe.cfg`` is no longer allowed.  The Tahoe-LAFS project recommends using a third-party metrics aggregation tool instead.

  • TabularUnified src/allmydata/immutable/checker.py

    r8b890d25 r624916e  
     1"""
     2Ported to Python 3.
     3"""
     4from __future__ import absolute_import
     5from __future__ import division
     6from __future__ import print_function
     7from __future__ import unicode_literals
     8
     9from future.utils import PY2
     10if PY2:
     11    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
     12
    113from zope.interface import implementer
    214from twisted.internet import defer

  • TabularUnified src/allmydata/immutable/repairer.py

    r8b890d25 r624916e  
     1"""
     2Ported to Python 3.
     3"""
     4from __future__ import absolute_import
     5from __future__ import division
     6from __future__ import print_function
     7from __future__ import unicode_literals
     8
     9from future.utils import PY2
     10if PY2:
     11    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
     12
    113from zope.interface import implementer
    214from twisted.internet import defer

  • TabularUnified src/allmydata/introducer/client.py

    r8b890d25 r624916e  
    1 from past.builtins import unicode, long
     1"""
     2Ported to Python 3.
     3"""
     4from __future__ import absolute_import
     5from __future__ import division
     6from __future__ import print_function
     7from __future__ import unicode_literals
     8
     9from future.utils import PY2
     10if PY2:
     11    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
     12from past.builtins import long
     13
    214from six import ensure_text
    315
    … …  
    2840                 sequencer, cache_filepath):
    2941        self._tub = tub
    30         if isinstance(introducer_furl, unicode):
     42        if isinstance(introducer_furl, str):
    3143            introducer_furl = introducer_furl.encode("utf-8")
    3244        self.introducer_furl = introducer_furl
    3345
    34         assert type(nickname) is unicode
     46        assert isinstance(nickname, str)
    3547        self._nickname = nickname
    3648        self._my_version = my_version
    … …  
    115127    def _save_announcements(self):
    116128        announcements = []
    117         for _, value in self._inbound_announcements.items():
     129        for value in self._inbound_announcements.values():
    118130            ann, key_s, time_stamp = value
    119131            # On Python 2, bytes strings are encoded into YAML Unicode strings.
    … …  
    126138            announcements.append(server_params)
    127139        announcement_cache_yaml = yamlutil.safe_dump(announcements)
    128         if isinstance(announcement_cache_yaml, unicode):
     140        if isinstance(announcement_cache_yaml, str):
    129141            announcement_cache_yaml = announcement_cache_yaml.encode("utf-8")
    130142        self._cache_filepath.setContent(announcement_cache_yaml)
    … …  
    171183        self._subscribed_service_names.add(service_name)
    172184        self._maybe_subscribe()
    173         for index,(ann,key_s,when) in self._inbound_announcements.items():
     185        for index,(ann,key_s,when) in list(self._inbound_announcements.items()):
    174186            precondition(isinstance(key_s, bytes), key_s)
    175187            servicename = index[0]
    … …  
    216228
    217229        # publish all announcements with the new seqnum and nonce
    218         for service_name,ann_d in self._outbound_announcements.items():
     230        for service_name,ann_d in list(self._outbound_announcements.items()):
    219231            ann_d["seqnum"] = current_seqnum
    220232            ann_d["nonce"] = current_nonce
    … …  
    228240            return
    229241        # this re-publishes everything. The Introducer ignores duplicates
    230         for ann_t in self._published_announcements.values():
     242        for ann_t in list(self._published_announcements.values()):
    231243            self._debug_counts["outbound_message"] += 1
    232244            self._debug_outstanding += 1
    … …  
    268280        # for ASCII values, simplejson might give us unicode *or* bytes
    269281        if "nickname" in ann and isinstance(ann["nickname"], bytes):
    270             ann["nickname"] = unicode(ann["nickname"])
     282            ann["nickname"] = str(ann["nickname"])
    271283        nick_s = ann.get("nickname",u"").encode("utf-8")
    272284        lp2 = self.log(format="announcement for nickname '%(nick)s', service=%(svc)s: %(ann)s",

  • TabularUnified src/allmydata/introducer/common.py

    r8b890d25 r624916e  
     1"""
     2Ported to Python 3.
     3"""
     4from __future__ import absolute_import
     5from __future__ import division
     6from __future__ import print_function
     7from __future__ import unicode_literals
     8
     9from future.utils import PY2
     10if PY2:
     11    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
     12
    113import re
    214from allmydata.crypto.util import remove_prefix

  • TabularUnified src/allmydata/introducer/server.py

    r8b890d25 r624916e  
     1"""
     2Ported to Python 3.
     3"""
     4
     5from __future__ import absolute_import
     6from __future__ import division
     7from __future__ import print_function
     8from __future__ import unicode_literals
     9
     10
     11from future.utils import PY2
     12if PY2:
     13    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
    114from past.builtins import long
    2 from six import ensure_str, ensure_text
     15from six import ensure_text
    316
    417import time, os.path, textwrap
    … …  
    158171        # expected keys are: version, nickname, app-versions, my-version,
    159172        # oldest-supported
    160         self._subscribers = {}
     173        self._subscribers = dictutil.UnicodeKeyDict({})
    161174
    162175        self._debug_counts = {"inbound_message": 0,
    … …  
    182195        """Return a list of AnnouncementDescriptor for all announcements"""
    183196        announcements = []
    184         for (index, (_, canary, ann, when)) in self._announcements.items():
     197        for (index, (_, canary, ann, when)) in list(self._announcements.items()):
    185198            ad = AnnouncementDescriptor(when, index, canary, ann)
    186199            announcements.append(ad)
    … …  
    190203        """Return a list of SubscriberDescriptor objects for all subscribers"""
    191204        s = []
    192         for service_name, subscriptions in self._subscribers.items():
    193             for rref,(subscriber_info,when) in subscriptions.items():
     205        for service_name, subscriptions in list(self._subscribers.items()):
     206            for rref,(subscriber_info,when) in list(subscriptions.items()):
    194207                # note that if the subscriber didn't do Tub.setLocation,
    195208                # tubid will be None. Also, subscribers do not tell us which
    … …  
    282295        self.log("introducer: subscription[%s] request at %s"
    283296                 % (service_name, subscriber), umid="U3uzLg")
    284         service_name = ensure_str(service_name)
     297        service_name = ensure_text(service_name)
    285298        subscriber_info = dictutil.UnicodeKeyDict({
    286299            ensure_text(k): v for (k, v) in subscriber_info.items()
    … …  
    308321        subscriber.notifyOnDisconnect(_remove)
    309322
     323        # Make sure types are correct:
     324        for k in self._announcements:
     325            assert isinstance(k[0], type(service_name))
     326
    310327        # now tell them about any announcements they're interested in
    311         assert {type(service_name)}.issuperset(
    312             set(type(k[0]) for k in self._announcements)), (
    313                 service_name, self._announcements.keys()
    314         )
    315328        announcements = set( [ ann_t
    316329                               for idx,(ann_t,canary,ann,when)

  • TabularUnified src/allmydata/scripts/tahoe_add_alias.py

    r8b890d25 r624916e  
    11from __future__ import print_function
     2from __future__ import unicode_literals
    23
    34import os.path
    … …  
    1112from allmydata.scripts.common import get_aliases
    1213from allmydata.util.fileutil import move_into_place
    13 from allmydata.util.encodingutil import unicode_to_output, quote_output
     14from allmydata.util.encodingutil import quote_output, quote_output_u
    1415
    1516
    … …  
    4950    old_aliases = get_aliases(nodedir)
    5051    if alias in old_aliases:
    51         print("Alias %s already exists!" % quote_output(alias), file=stderr)
     52        show_output(stderr, "Alias {alias} already exists!", alias=alias)
    5253        return 1
    5354    aliasfile = os.path.join(nodedir, "private", "aliases")
    … …  
    5556
    5657    add_line_to_aliasfile(aliasfile, alias, cap)
    57 
    58     print("Alias %s added" % quote_output(alias), file=stdout)
     58    show_output(stdout, "Alias {alias} added", alias=alias)
    5959    return 0
    6060
    … …  
    7676    old_aliases = get_aliases(nodedir)
    7777    if alias in old_aliases:
    78         print("Alias %s already exists!" % quote_output(alias), file=stderr)
     78        show_output(stderr, "Alias {alias} already exists!", alias=alias)
    7979        return 1
    8080
    … …  
    9494
    9595    add_line_to_aliasfile(aliasfile, alias, new_uri)
     96    show_output(stdout, "Alias {alias} created", alias=alias)
     97    return 0
    9698
    97     print("Alias %s created" % (quote_output(alias),), file=stdout)
    98     return 0
     99
     100def show_output(fp, template, **kwargs):
     101    """
     102    Print to just about anything.
     103
     104    :param fp: A file-like object to which to print.  This handles the case
     105        where ``fp`` declares a support encoding with the ``encoding``
     106        attribute (eg sys.stdout on Python 3).  It handles the case where
     107        ``fp`` declares no supported encoding via ``None`` for its
     108        ``encoding`` attribute (eg sys.stdout on Python 2 when stdout is not a
     109        tty).  It handles the case where ``fp`` declares an encoding that does
     110        not support all of the characters in the output by forcing the
     111        "namereplace" error handler.  It handles the case where there is no
     112        ``encoding`` attribute at all (eg StringIO.StringIO) by writing
     113        utf-8-encoded bytes.
     114    """
     115    assert isinstance(template, unicode)
     116
     117    # On Python 3 fp has an encoding attribute under all real usage.  On
     118    # Python 2, the encoding attribute is None if stdio is not a tty.  The
     119    # test suite often passes StringIO which has no such attribute.  Make
     120    # allowances for this until the test suite is fixed and Python 2 is no
     121    # more.
     122    try:
     123        encoding = fp.encoding or "utf-8"
     124    except AttributeError:
     125        has_encoding = False
     126        encoding = "utf-8"
     127    else:
     128        has_encoding = True
     129
     130    output = template.format(**{
     131        k: quote_output_u(v, encoding=encoding)
     132        for (k, v)
     133        in kwargs.items()
     134    })
     135    safe_output = output.encode(encoding, "namereplace")
     136    if has_encoding:
     137        safe_output = safe_output.decode(encoding)
     138    print(safe_output, file=fp)
    99139
    100140
    … …  
    112152
    113153
     154def _escape_format(t):
     155    """
     156    _escape_format(t).format() == t
     157
     158    :param unicode t: The text to escape.
     159    """
     160    return t.replace("{", "{{").replace("}", "}}")
     161
     162
    114163def list_aliases(options):
    115     nodedir = options['node-directory']
    116     stdout = options.stdout
    117     stderr = options.stderr
    118 
    119     data = _get_alias_details(nodedir)
    120 
    121     max_width = max([len(quote_output(name)) for name in data.keys()] + [0])
    122     fmt = "%" + str(max_width) + "s: %s"
    123     rc = 0
     164    """
     165    Show aliases that exist.
     166    """
     167    data = _get_alias_details(options['node-directory'])
    124168
    125169    if options['json']:
    126         try:
    127             # XXX why are we presuming utf-8 output?
    128             print(json.dumps(data, indent=4).decode('utf-8'), file=stdout)
    129         except (UnicodeEncodeError, UnicodeDecodeError):
    130             print(json.dumps(data, indent=4), file=stderr)
    131             rc = 1
     170        output = _escape_format(json.dumps(data, indent=4).decode("ascii"))
    132171    else:
    133         for name, details in data.items():
    134             dircap = details['readonly'] if options['readonly-uri'] else details['readwrite']
    135             try:
    136                 print(fmt % (unicode_to_output(name), unicode_to_output(dircap.decode('utf-8'))), file=stdout)
    137             except (UnicodeEncodeError, UnicodeDecodeError):
    138                 print(fmt % (quote_output(name), quote_output(dircap)), file=stderr)
    139                 rc = 1
     172        def dircap(details):
     173            return (
     174                details['readonly']
     175                if options['readonly-uri']
     176                else details['readwrite']
     177            ).decode("utf-8")
    140178
    141     if rc == 1:
    142         print("\nThis listing included aliases or caps that could not be converted to the terminal" \
    143                         "\noutput encoding. These are shown using backslash escapes and in quotes.", file=stderr)
    144     return rc
     179        def format_dircap(name, details):
     180            return fmt % (name, dircap(details))
     181
     182        max_width = max([len(quote_output(name)) for name in data.keys()] + [0])
     183        fmt = "%" + str(max_width) + "s: %s"
     184        output = "\n".join(list(
     185            format_dircap(name, details)
     186            for name, details
     187            in data.items()
     188        ))
     189
     190    if output:
     191        # Show whatever we computed.  Skip this if there is no output to avoid
     192        # a spurious blank line.
     193        show_output(options.stdout, output)
     194
     195    return 0

  • TabularUnified src/allmydata/storage_client.py

    r8b890d25 r624916e  
    562562
    563563        *nickname* is optional.
     564
     565        The furl will be a Unicode string on Python 3; on Python 2 it will be
     566        either a native (bytes) string or a Unicode string.
    564567        """
    565568        furl = furl.encode("utf-8")

  • TabularUnified src/allmydata/test/cli/common.py

    r8b890d25 r624916e  
    11from ...util.encodingutil import unicode_to_argv
    22from ...scripts import runner
    3 from ..common_util import ReallyEqualMixin, run_cli
     3from ..common_util import ReallyEqualMixin, run_cli, run_cli_unicode
    44
    55def parse_options(basedir, command, args):
    … …  
    1111
    1212class CLITestMixin(ReallyEqualMixin):
    13     def do_cli(self, verb, *args, **kwargs):
     13    """
     14    A mixin for use with ``GridTestMixin`` to execute CLI commands against
     15    nodes created by methods of that mixin.
     16    """
     17    def do_cli_unicode(self, verb, argv, client_num=0, **kwargs):
     18        """
     19        Run a Tahoe-LAFS CLI command.
     20
     21        :param verb: See ``run_cli_unicode``.
     22
     23        :param argv: See ``run_cli_unicode``.
     24
     25        :param int client_num: The number of the ``GridTestMixin``-created
     26            node against which to execute the command.
     27
     28        :param kwargs: Additional keyword arguments to pass to
     29            ``run_cli_unicode``.
     30        """
    1431        # client_num is used to execute client CLI commands on a specific
    1532        # client.
    16         client_num = kwargs.get("client_num", 0)
     33        client_dir = self.get_clientdir(i=client_num)
     34        nodeargs = [ u"--node-directory", client_dir ]
     35        return run_cli_unicode(verb, argv, nodeargs=nodeargs, **kwargs)
     36
     37
     38    def do_cli(self, verb, *args, **kwargs):
     39        """
     40        Like ``do_cli_unicode`` but work with ``bytes`` everywhere instead of
     41        ``unicode``.
     42
     43        Where possible, prefer ``do_cli_unicode``.
     44        """
     45        # client_num is used to execute client CLI commands on a specific
     46        # client.
     47        client_num = kwargs.pop("client_num", 0)
    1748        client_dir = unicode_to_argv(self.get_clientdir(i=client_num))
    18         nodeargs = [ "--node-directory", client_dir ]
    19         return run_cli(verb, nodeargs=nodeargs, *args, **kwargs)
     49        nodeargs = [ b"--node-directory", client_dir ]
     50        return run_cli(verb, *args, nodeargs=nodeargs, **kwargs)

  • TabularUnified src/allmydata/test/cli/test_alias.py

    r8b890d25 r624916e  
    11import json
    2 from mock import patch
    32
    43from twisted.trial import unittest
    54from twisted.internet.defer import inlineCallbacks
    65
    7 from allmydata.util.encodingutil import unicode_to_argv
    86from allmydata.scripts.common import get_aliases
    97from allmydata.test.no_network import GridTestMixin
    108from .common import CLITestMixin
    11 from ..common_util import skip_if_cannot_represent_argv
     9from allmydata.util import encodingutil
    1210
    1311# see also test_create_alias
    … …  
    1614
    1715    @inlineCallbacks
    18     def test_list(self):
    19         self.basedir = "cli/ListAlias/test_list"
     16    def _check_create_alias(self, alias, encoding):
     17        """
     18        Verify that ``tahoe create-alias`` can be used to create an alias named
     19        ``alias`` when argv is encoded using ``encoding``.
     20
     21        :param unicode alias: The alias to try to create.
     22
     23        :param NoneType|str encoding: The name of an encoding to force the
     24            ``create-alias`` implementation to use.  This simulates the
     25            effects of setting LANG and doing other locale-foolishness without
     26            actually having to mess with this process's global locale state.
     27            If this is ``None`` then the encoding used will be ascii but the
     28            stdio objects given to the code under test will not declare any
     29            encoding (this is like Python 2 when stdio is not a tty).
     30
     31        :return Deferred: A Deferred that fires with success if the alias can
     32            be created and that creation is reported on stdout appropriately
     33            encoded or with failure if something goes wrong.
     34        """
     35        self.basedir = self.mktemp()
    2036        self.set_up_grid(oneshare=True)
    2137
    22         rc, stdout, stderr = yield self.do_cli(
    23             "create-alias",
    24             unicode_to_argv(u"tahoe"),
     38        # We can pass an encoding into the test utilities to invoke the code
     39        # under test but we can't pass such a parameter directly to the code
     40        # under test.  Instead, that code looks at io_encoding.  So,
     41        # monkey-patch that value to our desired value here.  This is the code
     42        # that most directly takes the place of messing with LANG or the
     43        # locale module.
     44        self.patch(encodingutil, "io_encoding", encoding or "ascii")
     45
     46        rc, stdout, stderr = yield self.do_cli_unicode(
     47            u"create-alias",
     48            [alias],
     49            encoding=encoding,
    2550        )
    2651
    27         self.failUnless(unicode_to_argv(u"Alias 'tahoe' created") in stdout)
    28         self.failIf(stderr)
     52        # Make sure the result of the create-alias command is as we want it to
     53        # be.
     54        self.assertEqual(u"Alias '{}' created\n".format(alias), stdout)
     55        self.assertEqual("", stderr)
     56        self.assertEqual(0, rc)
     57
     58        # Make sure it had the intended side-effect, too - an alias created in
     59        # the node filesystem state.
    2960        aliases = get_aliases(self.get_clientdir())
    30         self.failUnless(u"tahoe" in aliases)
    31         self.failUnless(aliases[u"tahoe"].startswith("URI:DIR2:"))
     61        self.assertIn(alias, aliases)
     62        self.assertTrue(aliases[alias].startswith(u"URI:DIR2:"))
    3263
    33         rc, stdout, stderr = yield self.do_cli("list-aliases", "--json")
     64        # And inspect the state via the user interface list-aliases command
     65        # too.
     66        rc, stdout, stderr = yield self.do_cli_unicode(
     67            u"list-aliases",
     68            [u"--json"],
     69            encoding=encoding,
     70        )
    3471
    3572        self.assertEqual(0, rc)
    3673        data = json.loads(stdout)
    37         self.assertIn(u"tahoe", data)
    38         data = data[u"tahoe"]
    39         self.assertIn("readwrite", data)
    40         self.assertIn("readonly", data)
     74        self.assertIn(alias, data)
     75        data = data[alias]
     76        self.assertIn(u"readwrite", data)
     77        self.assertIn(u"readonly", data)
    4178
    42     @inlineCallbacks
    43     def test_list_unicode_mismatch_json(self):
     79
     80    def test_list_none(self):
    4481        """
    45         pretty hack-y test, but we want to cover the 'except' on Unicode
    46         errors paths and I can't come up with a nicer way to trigger
    47         this
     82        An alias composed of all ASCII-encodeable code points can be created when
     83        stdio aren't clearly marked with an encoding.
    4884        """
    49         self.basedir = "cli/ListAlias/test_list_unicode_mismatch_json"
    50         skip_if_cannot_represent_argv(u"tahoe\u263A")
    51         self.set_up_grid(oneshare=True)
    52 
    53         rc, stdout, stderr = yield self.do_cli(
    54             "create-alias",
    55             unicode_to_argv(u"tahoe\u263A"),
     85        return self._check_create_alias(
     86            u"tahoe",
     87            encoding=None,
    5688        )
    5789
    58         self.failUnless(unicode_to_argv(u"Alias 'tahoe\u263A' created") in stdout)
    59         self.failIf(stderr)
    6090
    61         booms = []
    62 
    63         def boom(out, indent=4):
    64             if not len(booms):
    65                 booms.append(out)
    66                 raise UnicodeEncodeError("foo", u"foo", 3, 5, "foo")
    67             return str(out)
    68 
    69         with patch("allmydata.scripts.tahoe_add_alias.json.dumps", boom):
    70             aliases = get_aliases(self.get_clientdir())
    71             self.failUnless(u"tahoe\u263A" in aliases)
    72             self.failUnless(aliases[u"tahoe\u263A"].startswith("URI:DIR2:"))
    73 
    74             rc, stdout, stderr = yield self.do_cli("list-aliases", "--json")
    75 
    76             self.assertEqual(1, rc)
    77             self.assertIn("could not be converted", stderr)
    78 
    79     @inlineCallbacks
    80     def test_list_unicode_mismatch(self):
    81         self.basedir = "cli/ListAlias/test_list_unicode_mismatch"
    82         skip_if_cannot_represent_argv(u"tahoe\u263A")
    83         self.set_up_grid(oneshare=True)
    84 
    85         rc, stdout, stderr = yield self.do_cli(
    86             "create-alias",
    87             unicode_to_argv(u"tahoe\u263A"),
     91    def test_list_ascii(self):
     92        """
     93        An alias composed of all ASCII-encodeable code points can be created when
     94        the active encoding is ASCII.
     95        """
     96        return self._check_create_alias(
     97            u"tahoe",
     98            encoding="ascii",
    8899        )
    89100
    90         def boom(out):
    91             print("boom {}".format(out))
    92             return out
    93             raise UnicodeEncodeError("foo", u"foo", 3, 5, "foo")
    94101
    95         with patch("allmydata.scripts.tahoe_add_alias.unicode_to_output", boom):
    96             self.failUnless(unicode_to_argv(u"Alias 'tahoe\u263A' created") in stdout)
    97             self.failIf(stderr)
    98             aliases = get_aliases(self.get_clientdir())
    99             self.failUnless(u"tahoe\u263A" in aliases)
    100             self.failUnless(aliases[u"tahoe\u263A"].startswith("URI:DIR2:"))
     102    def test_list_latin_1(self):
     103        """
     104        An alias composed of all Latin-1-encodeable code points can be created
     105        when the active encoding is Latin-1.
    101106
    102             rc, stdout, stderr = yield self.do_cli("list-aliases")
     107        This is very similar to ``test_list_utf_8`` but the assumption of
     108        UTF-8 is nearly ubiquitous and explicitly exercising the codepaths
     109        with a UTF-8-incompatible encoding helps flush out unintentional UTF-8
     110        assumptions.
     111        """
     112        return self._check_create_alias(
     113            u"taho\N{LATIN SMALL LETTER E WITH ACUTE}",
     114            encoding="latin-1",
     115        )
    103116
    104             self.assertEqual(1, rc)
    105             self.assertIn("could not be converted", stderr)
     117
     118    def test_list_utf_8(self):
     119        """
     120        An alias composed of all UTF-8-encodeable code points can be created when
     121        the active encoding is UTF-8.
     122        """
     123        return self._check_create_alias(
     124            u"tahoe\N{SNOWMAN}",
     125            encoding="utf-8",
     126        )

  • TabularUnified src/allmydata/test/cli/test_cp.py

    r8b890d25 r624916e  
    662662        # a local directory without a specified file name.
    663663        # https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2027
    664         self.basedir = "cli/Cp/cp_verbose"
     664        self.basedir = "cli/Cp/ticket_2027"
    665665        self.set_up_grid(oneshare=True)
    666666

  • TabularUnified src/allmydata/test/common.py

    r8b890d25 r624916e  
    1111    "skipIf",
    1212]
     13
     14from past.builtins import chr as byteschr
    1315
    1416import os, random, struct
    … …  
    10581060        offset = 0x0c+0x44+sharedatasize-1
    10591061
    1060     newdata = data[:offset] + chr(ord(data[offset])^0xFF) + data[offset+1:]
     1062    newdata = data[:offset] + byteschr(ord(data[offset:offset+1])^0xFF) + data[offset+1:]
    10611063    if debug:
    10621064        log.msg("testing: flipping all bits of byte at offset %d: %r, newdata: %r" % (offset, data[offset], newdata[offset]))
    … …  
    10861088    if debug:
    10871089        log.msg("original data: %r" % (data,))
    1088     return data[:0x0c+0x221] + chr(ord(data[0x0c+0x221])^0x02) + data[0x0c+0x2210+1:]
     1090    return data[:0x0c+0x221] + byteschr(ord(data[0x0c+0x221:0x0c+0x221+1])^0x02) + data[0x0c+0x2210+1:]
    10891091
    10901092def _corrupt_block_hashes(data, debug=False):

  • TabularUnified src/allmydata/test/common_util.py

    r8b890d25 r624916e  
    66from random import randrange
    77from six.moves import StringIO
     8from io import (
     9    TextIOWrapper,
     10    BytesIO,
     11)
    812
    913from twisted.internet import reactor, defer
    … …  
    3640        raise unittest.SkipTest("A non-ASCII argv could not be encoded on this platform.")
    3741
    38 def run_cli(verb, *args, **kwargs):
    39     precondition(not [True for arg in args if not isinstance(arg, str)],
    40                  "arguments to do_cli must be strs -- convert using unicode_to_argv", args=args)
    41     nodeargs = kwargs.get("nodeargs", [])
     42
     43def _getvalue(io):
     44    """
     45    Read out the complete contents of a file-like object.
     46    """
     47    io.seek(0)
     48    return io.read()
     49
     50
     51def run_cli_bytes(verb, *args, **kwargs):
     52    """
     53    Run a Tahoe-LAFS CLI command specified as bytes.
     54
     55    Most code should prefer ``run_cli_unicode`` which deals with all the
     56    necessary encoding considerations.  This helper still exists so that novel
     57    misconfigurations can be explicitly tested (for example, receiving UTF-8
     58    bytes when the system encoding claims to be ASCII).
     59
     60    :param bytes verb: The command to run.  For example, ``b"create-node"``.
     61
     62    :param [bytes] args: The arguments to pass to the command.  For example,
     63        ``(b"--hostname=localhost",)``.
     64
     65    :param [bytes] nodeargs: Extra arguments to pass to the Tahoe executable
     66        before ``verb``.
     67
     68    :param bytes stdin: Text to pass to the command via stdin.
     69
     70    :param NoneType|str encoding: The name of an encoding which stdout and
     71        stderr will be configured to use.  ``None`` means stdout and stderr
     72        will accept bytes and unicode and use the default system encoding for
     73        translating between them.
     74    """
     75    nodeargs = kwargs.pop("nodeargs", [])
     76    encoding = kwargs.pop("encoding", None)
     77    precondition(
     78        all(isinstance(arg, bytes) for arg in [verb] + nodeargs + list(args)),
     79        "arguments to run_cli must be bytes -- convert using unicode_to_argv",
     80        verb=verb,
     81        args=args,
     82        nodeargs=nodeargs,
     83    )
    4284    argv = nodeargs + [verb] + list(args)
    4385    stdin = kwargs.get("stdin", "")
    44     stdout = StringIO()
    45     stderr = StringIO()
     86    if encoding is None:
     87        # The original behavior, the Python 2 behavior, is to accept either
     88        # bytes or unicode and try to automatically encode or decode as
     89        # necessary.  This works okay for ASCII and if LANG is set
     90        # appropriately.  These aren't great constraints so we should move
     91        # away from this behavior.
     92        stdout = StringIO()
     93        stderr = StringIO()
     94    else:
     95        # The new behavior, the Python 3 behavior, is to accept unicode and
     96        # encode it using a specific encoding.  For older versions of Python
     97        # 3, the encoding is determined from LANG (bad) but for newer Python
     98        # 3, the encoding is always utf-8 (good).  Tests can pass in different
     99        # encodings to exercise different behaviors.
     100        stdout = TextIOWrapper(BytesIO(), encoding)
     101        stderr = TextIOWrapper(BytesIO(), encoding)
    46102    d = defer.succeed(argv)
    47103    d.addCallback(runner.parse_or_exit_with_explanation, stdout=stdout)
    … …  
    50106                  stdout=stdout, stderr=stderr)
    51107    def _done(rc):
    52         return 0, stdout.getvalue(), stderr.getvalue()
     108        return 0, _getvalue(stdout), _getvalue(stderr)
    53109    def _err(f):
    54110        f.trap(SystemExit)
    55         return f.value.code, stdout.getvalue(), stderr.getvalue()
     111        return f.value.code, _getvalue(stdout), _getvalue(stderr)
    56112    d.addCallbacks(_done, _err)
    57113    return d
     114
     115
     116def run_cli_unicode(verb, argv, nodeargs=None, stdin=None, encoding=None):
     117    """
     118    Run a Tahoe-LAFS CLI command.
     119
     120    :param unicode verb: The command to run.  For example, ``u"create-node"``.
     121
     122    :param [unicode] argv: The arguments to pass to the command.  For example,
     123        ``[u"--hostname=localhost"]``.
     124
     125    :param [unicode] nodeargs: Extra arguments to pass to the Tahoe executable
     126        before ``verb``.
     127
     128    :param unicode stdin: Text to pass to the command via stdin.
     129
     130    :param NoneType|str encoding: The name of an encoding to use for all
     131        bytes/unicode conversions necessary *and* the encoding to cause stdio
     132        to declare with its ``encoding`` attribute.  ``None`` means ASCII will
     133        be used and no declaration will be made at all.
     134    """
     135    if nodeargs is None:
     136        nodeargs = []
     137    precondition(
     138        all(isinstance(arg, unicode) for arg in [verb] + nodeargs + argv),
     139        "arguments to run_cli_unicode must be unicode",
     140        verb=verb,
     141        nodeargs=nodeargs,
     142        argv=argv,
     143    )
     144    codec = encoding or "ascii"
     145    encode = lambda t: None if t is None else t.encode(codec)
     146    d = run_cli_bytes(
     147        encode(verb),
     148        nodeargs=list(encode(arg) for arg in nodeargs),
     149        stdin=encode(stdin),
     150        encoding=encoding,
     151        *list(encode(arg) for arg in argv)
     152    )
     153    def maybe_decode(result):
     154        code, stdout, stderr = result
     155        if isinstance(stdout, bytes):
     156            stdout = stdout.decode(codec)
     157        if isinstance(stderr, bytes):
     158            stderr = stderr.decode(codec)
     159        return code, stdout, stderr
     160    d.addCallback(maybe_decode)
     161    return d
     162
     163
     164run_cli = run_cli_bytes
     165
    58166
    59167def parse_cli(*argv):

  • TabularUnified src/allmydata/test/test_repairer.py

    r8b890d25 r624916e  
    11# -*- coding: utf-8 -*-
     2"""
     3Ported to Python 3.
     4"""
    25from __future__ import print_function
     6from __future__ import absolute_import
     7from __future__ import division
     8from __future__ import unicode_literals
     9
     10from future.utils import PY2
     11if PY2:
     12    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min  # noqa: F401
    313
    414from allmydata.test import common
    … …  
    6373        c1 = self.g.clients[1]
    6474        c0.encoding_params['max_segment_size'] = 12
    65         d = c0.upload(upload.Data(common.TEST_DATA, convergence=""))
     75        d = c0.upload(upload.Data(common.TEST_DATA, convergence=b""))
    6676        def _stash_uri(ur):
    6777            self.uri = ur.get_uri()
    … …  
    465475
    466476        d.addCallback(lambda ignored:
    467                       self.delete_shares_numbered(self.uri, range(3, 10+1)))
     477                      self.delete_shares_numbered(self.uri, list(range(3, 10+1))))
    468478        d.addCallback(lambda ignored: download_to_data(self.c1_filenode))
    469479        d.addCallback(lambda newdata:
    … …  
    477487        d = self.upload_and_stash()
    478488        d.addCallback(lambda ignored:
    479                       self.delete_shares_numbered(self.uri, range(7)))
     489                      self.delete_shares_numbered(self.uri, list(range(7))))
    480490        d.addCallback(lambda ignored: self._stash_counts())
    481491        d.addCallback(lambda ignored:
    … …  
    510520
    511521        d.addCallback(lambda ignored:
    512                       self.delete_shares_numbered(self.uri, range(3, 10+1)))
     522                      self.delete_shares_numbered(self.uri, list(range(3, 10+1))))
    513523        d.addCallback(lambda ignored: download_to_data(self.c1_filenode))
    514524        d.addCallback(lambda newdata:
    … …  
    528538        # happiness setting.
    529539        def _delete_some_servers(ignored):
    530             for i in xrange(7):
     540            for i in range(7):
    531541                self.g.remove_server(self.g.servers_by_number[i].my_nodeid)
    532542
    … …  
    641651                # unless it has already repaired the previously-corrupted share.
    642652                def _then_delete_7_and_try_a_download(unused=None):
    643                     shnums = range(10)
     653                    shnums = list(range(10))
    644654                    shnums.remove(shnum)
    645655                    random.shuffle(shnums)
    … …  
    680690        self.set_up_grid()
    681691        c0 = self.g.clients[0]
    682         DATA = "a"*135
     692        DATA = b"a"*135
    683693        c0.encoding_params['k'] = 22
    684694        c0.encoding_params['n'] = 66
    685         d = c0.upload(upload.Data(DATA, convergence=""))
     695        d = c0.upload(upload.Data(DATA, convergence=b""))
    686696        def _then(ur):
    687697            self.uri = ur.get_uri()

  • TabularUnified src/allmydata/test/test_system.py

    r8b890d25 r624916e  
    25642564        def _run_in_subprocess(ignored, verb, *args, **kwargs):
    25652565            stdin = kwargs.get("stdin")
     2566            # XXX https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3548
    25662567            env = kwargs.get("env", os.environ)
    25672568            # Python warnings from the child process don't matter.

  • TabularUnified src/allmydata/util/_python3.py

    r8b890d25 r624916e  
    3636    "allmydata.crypto.util",
    3737    "allmydata.hashtree",
     38    "allmydata.immutable.checker",
    3839    "allmydata.immutable.downloader",
    3940    "allmydata.immutable.downloader.common",
    … …  
    5051    "allmydata.immutable.literal",
    5152    "allmydata.immutable.offloaded",
     53    "allmydata.immutable.repairer",
    5254    "allmydata.immutable.upload",
    5355    "allmydata.interfaces",
     56    "allmydata.introducer.client",
     57    "allmydata.introducer.common",
    5458    "allmydata.introducer.interfaces",
     59    "allmydata.introducer.server",
    5560    "allmydata.monitor",
    5661    "allmydata.mutable.checker",
    … …  
    152157    "allmydata.test.test_pipeline",
    153158    "allmydata.test.test_python3",
     159    "allmydata.test.test_repairer",
    154160    "allmydata.test.test_spans",
    155161    "allmydata.test.test_statistics",

  • TabularUnified src/allmydata/util/encodingutil.py

    r8b890d25 r624916e  
    252252
    253253ESCAPABLE_8BIT    = re.compile( br'[^ !#\x25-\x5B\x5D-\x5F\x61-\x7E]', re.DOTALL)
     254
     255def quote_output_u(*args, **kwargs):
     256    """
     257    Like ``quote_output`` but always return ``unicode``.
     258    """
     259    result = quote_output(*args, **kwargs)
     260    if isinstance(result, unicode):
     261        return result
     262    return result.decode(kwargs.get("encoding", None) or io_encoding)
     263
    254264
    255265def quote_output(s, quotemarks=True, quote_newlines=None, encoding=None):

  • TabularUnified src/allmydata/web/statistics.xhtml

    r8b890d25 r624916e  
    1313
    1414    <ul>
    15       <li>Load Average: <t:transparent t:render="load_average" /></li>
    16       <li>Peak Load: <t:transparent t:render="peak_load" /></li>
    1715      <li>Files Uploaded (immutable): <t:transparent t:render="uploads" /></li>
    1816      <li>Files Downloaded (immutable): <t:transparent t:render="downloads" /></li>

  • TabularUnified src/allmydata/web/status.py

    r8b890d25 r624916e  
    15671567
    15681568    @renderer
    1569     def load_average(self, req, tag):
    1570         return tag(str(self._stats["stats"].get("load_monitor.avg_load")))
    1571 
    1572     @renderer
    1573     def peak_load(self, req, tag):
    1574         return tag(str(self._stats["stats"].get("load_monitor.max_load")))
    1575 
    1576     @renderer
    15771569    def uploads(self, req, tag):
    15781570        files = self._stats["counters"].get("uploader.files_uploaded", 0)
Note: See TracChangeset for help on using the changeset viewer.