Mon Apr 27 03:58:49 MDT 2009 francois@ctrlaltdel.ch * cli: plumbing for unicode support New patches: [cli: plumbing for unicode support francois@ctrlaltdel.ch**20090427095849 Ignore-this: 9cc8871d8fe54cde0dba52912754e4f4 ] { hunk ./src/allmydata/scripts/cli.py 4 import os.path, re, sys, fnmatch from twisted.python import usage from allmydata.scripts.common import BaseOptions, get_aliases +from allmydata.util.stringutils import argv_to_unicode NODEURL_RE=re.compile("http://([^:]*)(:([1-9][0-9]*))?") hunk ./src/allmydata/scripts/common.py 4 import os, sys, urllib from twisted.python import usage - +from allmydata.util.stringutils import unicode_to_url class BaseOptions: # unit tests can override these to point at StringIO instances hunk ./src/allmydata/test/test_cli.py 9 import urllib import re import simplejson +import sys from allmydata.util import fileutil, hashutil, base32 from allmydata import uri addfile ./src/allmydata/util/stringutils.py hunk ./src/allmydata/util/stringutils.py 1 +""" +Functions used to convert inputs from whatever encoding used in the system to +unicode and back. + +TODO: + * Accept two cli arguments --argv-encoding and --filesystem-encoding +""" + +import sys +from allmydata.util.assertutil import precondition +from twisted.python import usage + +def argv_to_unicode(s): + """ + Decode given argv element to unicode. + """ + # sys.argv encoding detection in Python is not trivial so utf-8 is + # currently used by default and an informative error message is given if + # the argument cannot be correctly decoded. + + precondition(isinstance(s, str), s) + try: + return unicode(s, 'utf-8') + except UnicodeEncodeError: + raise usageError("Argument '%s' cannot be decoded as UTF-8." % s) + +def fs_to_unicode(s): + """ + Decode a filename (or a directory name) to unicode using the same encoding + as the filesystem. + """ + # Filename encoding detection is a little bit better thanks to + # getfilesystemencoding() in the sys module. However, filenames can be + # encoded using another encoding than the one used on the filesystem. + + precondition(isinstance(s, str), s) + encoding = sys.getfilesystemencoding() + try: + return unicode(s, encoding) + except UnicodeDecodeError: + raise usage.UsageError("Filename '%s' cannot be decoded using the current encoding of your filesystem (%s). Please rename this file." % (s, encoding)) + +def unicode_to_fs(s): + """ + Encode an unicode object used in file or directoy name. + """ + + precondition(isinstance(s, unicode), s) + encoding = sys.getfilesystemencoding() + try: + return s.encode(encoding) + except UnicodeEncodeError: + raise usage.UsageError("Filename '%s' cannot be encoded using the current encoding of your filesystem (%s). Please configure your locale correctly or rename this file." % (s, encoding)) + +def unicode_to_url(s): + """ + Encode an unicode object used in an URL. + """ + # According to RFC 2718, non-ascii characters in url's must be UTF-8 encoded. + + precondition(isinstance(s, unicode), s) + return s.encode('utf-8') + +def unicode_to_stdout(s): + """ + Encode an unicode object for representation on stdout. + """ + + precondition(isinstance(s, unicode), s) + return s.encode(sys.stdout.encoding, 'replace') } Context: [test_cli.Backup: increase timeout massively, it takes 1200s on zandr's ARM linkstation warner@lothar.com**20090609052801] [tests: double the timeouts on some tests which time-out on Francois's box zooko@zooko.com**20090609021753 Ignore-this: b2727b04402f24a9b9123d2f84068106 ] [tests: bump up timeouts so that the tests can finish before timeout on Francois's little arm box zooko@zooko.com**20090608225557 Ignore-this: fb83698338b2f12546cd3e1dcb896d34 ] [tests: increase timeouts on some other tests that timed-out on Francois's arm box zooko@zooko.com**20090605143437 Ignore-this: 2903cc20d914fc074c8d7a6c47740ba6 ] [tests: bump up the timeout on a bunch of tests that took longer than the default timeout (120s) on François Lenny-armv5tel zooko@zooko.com**20090605031444 Ignore-this: 84d67849b1f8edc88bf7001e31b5f7f3 ] [backup: remove the --no-backupdb command, the handling of "can't import sqlite", and the related tests, and change an error message to more correctly indicate failure to load the database from disk rather than failure to import sqlite module zooko@zooko.com**20090604173131 Ignore-this: 8200a9fdfc49243c280ecd1d0c44fa19 Fixes #728. ] [more refactoring: move get_all_serverids() and get_nickname_for_serverid() from Client to storage_broker warner@lothar.com**20090602030750] [more storage_broker refactoring: downloader gets a broker instead of a client, warner@lothar.com**20090602022511 use Client.get_storage_broker() accessor instead of direct attribute access. ] [test_runner.py: remove test_client_no_noise: the issue in question is warner@lothar.com**20090601225007 ticketed in http://divmod.org/trac/ticket/2830 and doesn't need a Tahoe-side change, plus this test fails on win32 for unrelated reasons (and test_client is the place to think about the win32 issue). ] [remove plaintext-hashing code from the helper interface, to close #722 warner@lothar.com**20090601224916 and deny the Helper the ability to mount a partial-information-guessing attack. This will probably break compatibility between new clients and very old (pre-1.0) helpers. ] [start to factor server-connection-management into a distinct 'StorageServerFarmBroker' object, separate from the client and the introducer. This is the starting point for #467: static server selection warner@lothar.com**20090601210604] [mutable: catch and display first error, so code bugs which break all servers get displayed better warner@lothar.com**20090601210407] [misc/run-with-pythonpath.py: exec() the child (on unix), to remove the intermediate process warner@lothar.com**20090601210137] [setup: require pysqlite >= v2.0.5. if we are running on Python < 2.5 zooko@zooko.com**20090604154548 Ignore-this: cf04f46079821df209d01dad2e24b40b ] [setup: add pysqlite and sqlite to get_package_versions() zooko@zooko.com**20090604153728 Ignore-this: a1dea7fabeab2b08fb0d8d462facdb4d ] [docs: small edit to about.html zooko@zooko.com**20090528233422 Ignore-this: 1cfbb1f8426ed6d63b2d3952e4464ddc ] [docs: add links to Tahoe-LAFS for Paranoids and Tahoe-LAFS for Corporates in about.html zooko@zooko.com**20090528232717 Ignore-this: 7b70baa700d6b6f6e9ceec4132efe5 ] [docs: edit about.html and include network-and-reliance-topology.png (loaded from http://allmydata.org ) zooko@zooko.com**20090527150916 Ignore-this: 44adc61cde8ced8be2f0a7dfc7d95dad ] [docs: a few more edits to network-and-reliance-topology.svg zooko@zooko.com**20090527150458 Ignore-this: 2eac8c33fe71be25ff809b399c6193c1 ] [docs: update NEWS, relnotes.txt, CREDITS to mention WUI Style zooko@zooko.com**20090526233654 Ignore-this: 72d16ec833bc4a22af23d29ea1d5ff8b ] [docs: update network-and-reliance-topology.svg for beauty and clarity zooko@zooko.com**20090527031123 Ignore-this: 5510914849771900ac29b4312470d84 ] [Modify markup of Tahoe web pages to be more amenable to styling; some minor changes of wording. Kevin Reid **20090526232545 Ignore-this: 8845937f0df6c7ddc07abe3211428a6f ] [Tweak wording in directory page: not-read-only is "modifiable", mention creating a directory _in this directory_. Kevin Reid **20090526232414 Ignore-this: f006ec52ba2051802e025a60bcface56 ] [Comment on duplication of code/markup found during styling project. Kevin Reid **20090503203442 Ignore-this: a4b7f9f0ab57d2c03be9ba761be8d854 ] [Add CSS styles to spiff up the Tahoe WUI's appearance, particularly the welcome page and directories. Kevin Reid **20090503203142 Ignore-this: 5c50af241c1a958b5180ef2b6a49f626 ] [Link all Tahoe web pages to the /tahoe_css stylesheet which already exists. Kevin Reid **20090503202533 Ignore-this: 2ea8d14d3168b9502cf39d5ea3f2f2a8 ] [Fix broken link from Provisioning to Reliability page. Kevin Reid **20090501191050 Ignore-this: 56dc1a5e659b70cc02dc4df7b5d518cd ] [docs: network-and-reliance-topology.svg: nicer server icons, mv out of the "specifications" subdir zooko@zooko.com**20090526165842 Ignore-this: 8f47ab3a0ab782c1f0d46e10bcaebe5b ] [accounting-overview.txt: more edits warner@lothar.com**20090523190359] [accounting-overview.txt: small edits warner@lothar.com**20090523184011] [_auto_deps.py: require foolscap-0.4.1, which adds an important fix for py2.4 warner@lothar.com**20090523011103] [immutable/encode.py: tolerate immediate _remove_shareholder by copying the warner@lothar.com**20090522184424 landlord list before iterating over it. This can probably only happen in unit tests, but cleaning it up makes certain test failures easier to analyze. ] [switch to using RemoteException instead of 'wrapped' RemoteReferences. Should fix #653, the rref-EQ problem warner@lothar.com**20090522004632] [switch all foolscap imports to use foolscap.api or foolscap.logging warner@lothar.com**20090522003823] [_auto_deps.py: bump our foolscap dependency to 0.4.0, since I'm about to start using its new features warner@lothar.com**20090522002100] [test_runner.py: fix minor typo warner@lothar.com**20090520033620] [docs: update network-and-reliance-topology.svg zooko@zooko.com**20090526163105 Ignore-this: 2b864b4ed8743d4a15dfbb7eff3fa561 ] [setup: fix bug (wrong import) in error message, as noticed by pyflakes zooko@zooko.com**20090519195642 Ignore-this: f1b9f8c00b46c1b5f2f20e5fc424f341 ] [setup: fix trivial bug in recent patch to test base64.py at startup zooko@zooko.com**20090519195129 Ignore-this: f6be038f74b53ca69e7109fe34adfbc ] [setup: make Tahoe exit at startup with a useful error message if the base64.py module is buggy (fixes part of #710) zooko@zooko.com**20090519194555 Ignore-this: aa4d398235ddca8d417d61c9688e154 ] [test_introducer.py: add a test for the python2.4.0/2.4.1 bug in base64.b32decode warner@lothar.com**20090519034101] [immutable WriteBucketProxy: use pipeline to speed up uploads by overlapping roundtrips, for #392 warner@lothar.com**20090518234422] [util/pipeline.py: new utility class to manage size-limited work pipelines, for #392 warner@lothar.com**20090518234326] [docs: add a diagram that I'm about to show to the Boulder Linux Users Group: network-and-reliance-topology.svg zooko@zooko.com**20090514232059 Ignore-this: 2420c0a7c254c9f0f2349d9130490d33 ] [tests: mark test_runner as coded in utf-8 instead of ascii zooko@zooko.com**20090507223151 Ignore-this: ccf1ba9e5a9b53602701a36f9fdb545e ] [tests: raise timeout on test_runner.RunNode.test_introducer from 120s to 240s, since it hit the 120s time-out on François Lenny-armv5tel zooko@zooko.com**20090507215012 Ignore-this: ba18fe6832ba255d4971e8f623ed7da5 ] [setup: fix comment in setup.py zooko@zooko.com**20090507215003 Ignore-this: c46ef664630d52733138ef7fbc551c1c ] [docs: how_to_make_a_tahoe_release.txt: a couple of small edits zooko@zooko.com**20090507214932 Ignore-this: ae92aa835ad369f4b9e6e49d681957a3 ] [.darcs-boringfile: also ignore .gitignore warner@allmydata.com**20090415210550 Ignore-this: d29db314a1e506f6240859559436b4c3 ] [.darcs-boringfile: ignore .git, I'm starting to play around with it warner@allmydata.com**20090415205929 Ignore-this: 89234453516483c9586cd6e1351e88b5 ] [fix quicktest: stop using setuptools, add misc/run-with-pythonpath.py, to make it run faster warner@lothar.com**20090414201400] [TAG allmydata-tahoe-1.4.1 zooko@zooko.com**20090414025636 Ignore-this: de78fc32364c83e9f4e26b5abcfdea4a ] Patch bundle hash: 4c155d5d10ce71e7847030e7ddfdf575cc2129a2