From trac at allmydata.org Wed Jul 1 00:27:46 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 07:27:46 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.2c3e7ffd8fa97ec059773ceadf77e3d8@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I've missed the actual instruction from my previous post: {{{ vex x86->IR: unhandled instruction bytes: 0x66 0xF 0x6F 0x1 }}} Which maps to: {{{ MOVDQA 66 0F 6F /r xmm1, xmm2/m128 Move aligned double quadword from xmm2/m128 to xmm1 }}} Looks like valgrind doesnt like SSE2 stuff? -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 00:56:17 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 07:56:17 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.c68eb9b80c2e9eec637a2a7ff23c0f8c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Aha! Now i built it without SSE2 support (adding CRYPTOPP_DISABLE_SSE2) and the test passes (and valgrind too): {{{ ==26122== Memcheck, a memory error detector. ==26122== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al. ==26122== Using LibVEX rev 1884, a library for dynamic binary translation. ==26122== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP. ==26122== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation framework. ==26122== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al. ==26122== For more details, rerun with: -v ==26122== running darcsver running test running egg_info writing requirements to pycryptopp.egg-info/requires.txt writing pycryptopp.egg-info/PKG-INFO writing top-level names to pycryptopp.egg-info/top_level.txt writing dependency_links to pycryptopp.egg-info/dependency_links.txt writing manifest file 'pycryptopp.egg-info/SOURCES.txt' running build_ext copying build/lib.linux-i686-2.5/pycryptopp/_pycryptopp.so -> pycryptopp test_chunksize (pycryptopp.test.test_sha256.SHA256) ... ok ---------------------------------------------------------------------- Ran 1 test in 10.929s OK ==26122== ==26122== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 3736 from 9) ==26122== malloc/free: in use at exit: 4,983,817 bytes in 906 blocks. ==26122== malloc/free: 325,282 allocs, 324,376 frees, 71,999,912 bytes allocated. ==26122== For counts of detected errors, rerun with: -v ==26122== searching for pointers to 906 not-freed blocks. ==26122== checked 5,115,828 bytes. ==26122== ==26122== LEAK SUMMARY: ==26122== definitely lost: 0 bytes in 0 blocks. ==26122== possibly lost: 66,612 bytes in 181 blocks. ==26122== still reachable: 4,917,205 bytes in 725 blocks. ==26122== suppressed: 0 bytes in 0 blocks. ==26122== Rerun with --leak-check=full to see details of leaked memory. }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 01:43:12 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 08:43:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.83adce40d919b1bd21d2bcd7d39dbcfb@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): And even more fun: Using an SSE2-less build of pycryptopp makes tahoe pass all the tests without crashes/hangs (http://allmydata.org/trac/tahoe/ticket/737): {{{ Ran 599 tests in 395.987s PASSED (skips=3, expectedFailures=2, unexpectedSuccesses=7, successes=587) }}} So i guess there is a bug in either Crypto++ SSE code or my CPU SSE implementation that somehow also makes python's time() return NaNs or segfault. FWIW this system CPU is: {{{ processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 10 model name : AMD Athlon(tm) XP 2600+ stepping : 0 cpu MHz : 1916.490 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow up }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 02:10:58 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 09:10:58 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.d597553ffabd32659fd2cf5e85e284b5@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Now it hit me - AMD AthlonXP doesn't even support SSE2, only the base SSE - why is MOVDQA even being used? and why doesn't it outright crash with an invalid opcode when not in valgrind? Also for completeness tested with --disable-embedded-cryptopp - I'm getting the same erros from valgring/failed test/hanging tahoe test. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 06:26:41 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 13:26:41 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.ae22458ad79e8c269ed7223b90ee0d77@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): When a standard Crypto++ source tree is built (i.e., not the special embedded build in pycryptopp) then it produces a file named {{{cryptest.exe}}} (even on Unix). If you run {{{./cryptest.exe v}}} then it will execute the Crypto++ self-tests. (If you run {{{./cryptest.exe b}}} then it will execute the benchmarks.) I don't know yet whether the self-tests exercise the same paths of hashing, as Brian described in the original issue report: "two updates, in which the total number of bytes is a multiple of 128, and in which the first write is an odd number less than 64, will cause this bug." -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 06:57:46 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 13:57:46 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): Wei Dai added optimized assembly for SHA-256 for amd64 in the Crypto++ v5.6.0 release. Perhaps a bug was added in that change. Isn't MOVDQA an SSE instruction (not requiring SSE2)? -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 08:49:47 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 15:49:47 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.f82eb11c4ef240ed6af4440c79217d62@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): According to http://en.wikipedia.org/wiki/X86_instruction_listings - MOVDQA belongs to SSE2. However, according to http://mubench.sourceforge.net/results.html - "this cpu does not support SSE2, but appears to run some SSE2 opcodes that have corresponding MMX opcodes without error". -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 16:53:29 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 23:53:29 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.657e5e78c888f4a820ba065eba370833@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I've downloaded and built crypto++ from source, all the tests succeed in cryptest.exe -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 19:39:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 02:39:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.14a3ab86fd863103692555ef7455a11a@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): So, summarizing discussions with zooko and warner... * We first need to handle testing for backwards compatibility. This means that we'll want to capture a base32-encoded representation of the output of _pack_contents as it sits now when run against some small, known directory structure. We'll then use that to test _unpack_contents (to make sure that the unpacked directory structure is what we're expecting), and _pack contents (to make sure that packing the same known directory structure results in the same packed representation). * Then we can do the optimizations that this ticket suggests. (that didn't really need to be a list, huh.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 19:51:12 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 02:51:12 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.f24683f987bf118a2b8b59219fb8d3a1@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): I've started work on the first point by writing a little program to try to get the base32 representation of the output of _pack_contents. Unfortunately, I run into path issues when I try to run it. At the top of my script, I have {{{ import sys base = "/path-to-tahoe/tahoe/support/lib/python2.6/site-packages" sys.path.append(base) }}} to tell the Python interpreter about tahoe's support directory. The goal was to have a somewhat less robust version of the detection that the tahoe executable does when it first starts (since I know where the directory is on my system, and don't imagine that the script is going to be run elsewhere, since it's kind of a one-time deal to make tests work). When I run the script, and try to import something from the tests directory, I get {{{ Traceback (most recent call last): File "get_dirnode_strings.py", line 7, in from test.test_dirnode import FakeClient File "/path-to-tahoe/tahoe/src/allmydata/test/__init__.py", line 2, in from foolscap.logging.incident import IncidentQualifier ImportError: No module named foolscap.logging.incident }}} which would seem to suggest that my path hacking doesn't do what I want it to. Is there something easy that I'm missing that would make this work? (also, my approach for getting this information is based on what of tahoe's code I've read, which isn't much. if there's a more obvious way to get the strings that we need, or if my program is flawed in some other way, please let me know) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 20:15:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 03:15:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.08e98101386c9d89e22d34678edded96@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): The way to do that is to set the {{{PYTHONPATH}}} environment variable before invoking the {{{python}}} interpreter. There are some hacks (mostly due to setuptools, I think) that need to run at interpreter startup time and learn about the available Python packages, so adding to your {{{sys.path}}} after interpreter startup won't work. It was a good idea though. If you can't or don't want to set your {{{PYTHONPATH}}}, then you'll need to find the {{{foolscap}}} package, probably in {{{/path-to- tahoe/tahoe/support/lib/python2.6/site- packages/foolscap-0.4.2-py2.6.egg}}}, and add that whole path (including the {{{thing.egg}}} part) to your {{{sys.path}}}, and then do likewise with the other packages. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 21:28:27 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 04:28:27 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.967b100497482f651da290da3f3d08b5@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): That fixed the issue -- thanks. Just so I'm clear on the next part: Currently, {{{_unpack_contents}}} returns a dictionary mapping the name of each child node to a pair {{{(child node instance, metadata)}}}, where metadata is a dictionary of metadata. {{{_pack_contents}}} gets the {{{rocap}}} and {{{rwcap}}} (if present) from the child node instance. What we want to do is * Modify {{{_unpack_contents}}} to return a dict mapping the name of each child node to a triple {{{(child node instance, metadata, serialized)}}}, where child node instance and metadata are as they are now, and serialized is the serialized from of the other data. * Modify {{{_pack_contents}}} to deal with this new representation; specifically, to use {{{serialized}}} if present instead of serializing the other data, which we think is expensive. * Modify {{{Adder}}}, {{{MetadataSetter}}} and {{{Deleter}}} to work with these changes; specifically, they should, when modifying a node, set the corresponding {{{serialized}}} value to None, which will tell {{{_pack_contents}}} that it needs to re-serialize that node. * Write tests to make sure that all of this works (probably before doing the first 3 items). In particular, * If we pass contents packed with an unmodified {{{_pack_contents}}} to the new {{{_unpack_contents}}}, we should get the same underlying directory structure back (i.e.: all expected child node names are present, and metadata values are as they should be. Is there anything else we'd want to check?) * The output of the new {{{_pack_contents}}} should be the same as the output from the old one, given the same directory structure. * We should be able to interact with {{{Adder}}}, {{{Deleter}}}, and {{{MetadataSetter}}} in the same way that we do now. Are there already tests to adequately verify this, or should I plan on writing those, too? (apologies if this is pedantic, but I'm still getting the hang of tahoe, and don't want to go off on an irrelevant tangent because I've made a bad assumption somewhere) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 21:39:17 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 04:39:17 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.0e129bcc467803e21b66338dd266559d@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): That sounds good to me! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Thu Jul 2 07:20:06 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 2 Jul 2009 08:20:06 -0600 Subject: [tahoe-dev] v1.5 status Message-ID: Folks: To track the imminent v1.5 release, see The Roadmap: http://allmydata.org/trac/tahoe/roadmap and particularly the list of open tickets in the 1.5 Milestone: http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.5.0 If you can help with some of these, please do! :-) I have a 3-day weekend coming up (tomorrow, Friday, off for USA Independence Day), so I hope to make some progress or even completely release v1.5 this weekend. Regards, Zooko From trac at allmydata.org Thu Jul 2 11:27:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 18:27:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.da82c521f420a10c6e91b67acab7777f@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): I usually write code like this by adding a new unit test. You can run a specific unit test with a minimum of fuss (and startup time and extraneous noise) by doing e.g. {{{make quicktest TEST=allmydata.test.test_dirnode.DeepStats.test_stats}}} . That will get all the PYTHONPATH stuff set up for you. Anything you {{{print}}} from the unit test will get displayed, so I use this for one- off tools all the time. Also, you can use {{{misc/run-with-pythonpath.py}}}, which will set up the right environment and then run the command of your choice. For example, if you wrote a {{{foo.py}}} to do the stuff you just described, then you could invoke {{{python misc/run-with-pythonpath.py python foo.py}}}. I'm disappointed that the sys.path technique you tried didn't work.. it used to. Your approach to {{{_pack_contents}}} sounds great! Note that the {{{serialized}}} form should contain the whole {{{name+rocap+encrwcap+metadata}}} string (i.e. be sure to include the name). That way, if {{{_pack_contents}}} sees that it has a pre-serialized string available, it can just append that to the list of entries that it's building, and doesn't need to re-serialize the childname either. At the end of that loop, it should do a single {{{"".join(entries)}}} to perform the final assembly (rather than doing incremental {{{+=}}} operations, which would be a lot slower). > If we pass contents packed with an unmodified _pack_contents to the new _unpack_contents, we should get the same underlying directory structure back (i.e.: all expected child node names are present, and metadata values are as they should be. Is there anything else we'd want to check?) We should make sure that the rocap/rwcap is the same too. Doing child.get_uri() and comparing it against a constant is plenty. > We should be able to interact with Adder, Deleter, and MetadataSetter in the same way that we do now. Are there already tests to adequately verify this, or should I plan on writing those, too? test_dirnode.py should already have test coverage for the modifiers. It exercises all the {{{NewDirectoryNode}}} methods, like {{{set_uri}}} and {{{list}}}. If they pass, then you've updated the modifier classes successfully. Also, if it works on your platform (it's somewhat touchy right now), use our test-coverage tools: {{{make quicktest-figleaf figleaf-output}}}, and check to see that all the code you've touched is actually being run. The buildbot has a link to the current coverage data (generated under py2.4; it seems that py2.5 gives slightly different answers). I think it should be pretty easy to make these changes and achieve the same or better coverage than before. Also, I don't know if it'd be worth it, but you could get fancy and instead of changing the data structure from a list of two-tuples to a list of three-tuples, you could: * create a subclass of {{{list}}} * have it store an auxilliary value (the unmodified serialized string) for each key * override {{{__setitem__}}} to remove that auxilliary value * add a {{{get_both_items}}} method, which takes a key and returns a two-tuple of (aux-serialized, value) (which is really (aux-serialized, (child,metadata))) * add a {{{set_both_items}}} method which takes (key, aux-serialized, (child,metadata)) * then the Modifier classes like {{{Adder}}} and {{{Deleter}}} don't have to change: they'll set the entries that have changed and leave the rest alone * now {{{_pack_contents}}} uses {{{get_both_items}}} and prefers the pre- serialized form if it's still there Since {{{NewDirectoryNode.list}}} returns whatever {{{_unpack_contents}}} returns, you might manage to write less code if you use this technique. Otherwise you'll need to find all the callers of {{{_read}}} and update them to tolerate the new three-tuple (and add a wrapper to {{{list}}} to convert it into two-tuple form for external callers, so that it continues to honor the same interface defined in {{{allmydata.interfaces.IDirectoryNode}}}). Oh, and of course, it would be a good idea to actually measure the time that serialization/deserialization takes before doing any of this work. Create 10/100/1k/10k/100k entries at random, call {{{_pack_contents}}} under {{{timeit.py}}} or some other sort of how-long-does-it-take loop, and figure out an {{{A+Bx}}} curve (I'd expect the serialization time to be some constant A plus some per-entry time B, let's figure out what A and B are). Let's make sure that serialization is the culprit, it might be that call to {{{_create_node}}} in unpack that we should focus on. And let's set a reasonable goal.. maybe we should be able to unpack+modify+pack 10k entries in less than a second, or something. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 11:34:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 18:34:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.d82e373ae33bdcb541d7287ea61c05e6@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): The measuring part is my job -- #327 (performance measurement of directories). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 13:45:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 20:45:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.f9180504d3048799a3abc0223ffca6fb@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: [4-tea-2] Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by zooko): Oh, when I wrote comment:7, I was thinking that Python >= 2.6 comes with sqlite, but it is actually Python >= 2.5. [4-tea-2], would you please run the following and post the results? {{{ PYTHONPATH=./support/lib/python2.5/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' }}} ? For example, here are the results on my machine: {{{ $ PYTHONPATH=./support/lib/python2.4/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' [allmydata-tahoe 1.4.1-r3914 (/home/zooko/playground/allmydata/tahoe/trunk/src), pysqlite 2.5.5 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/pysqlite-2.5.5-py2.4-solaris-2.11-i86pc.egg), Nevow 0.9.32 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/Nevow-0.9.32-py2.4.egg), foolscap 0.4.1 (/usr/lib/python2.4/site- packages/foolscap-0.4.1-py2.4.egg), Twisted 8.2.0 (/usr/lib/python2.4 /site-packages/Twisted-8.2.0-py2.4-solaris-2.11-i86pc.egg), zope.interface 3.5.1 (/usr/lib/python2.4/site- packages/zope.interface-3.5.1-py2.4-solaris-2.11-i86pc.egg), simplejson 2.0.7 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4 /site-packages/simplejson-2.0.7-py2.4-solaris-2.11-i86pc.egg), zfec 1.4.4 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/zfec-1.4.4-py2.4-solaris-2.11-i86pc.egg), pycryptopp 0.5.4 (/usr/lib/python2.4/site- packages/pycryptopp-0.5.4-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), pyOpenSSL 0.8 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/pyOpenSSL-0.8-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), pyutil 1.3.26 (/usr/lib/python2.4 /site-packages/pyutil-1.3.26-py2.4.egg), argparse 0.8.0 (/usr/lib/python2.4/site-packages/argparse-0.8.0-py2.4.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), zbase32 1.1.0 (/usr/lib/python2.4 /site-packages/zbase32-1.1.0-py2.4.egg)] }}} Then, please look at the {{{_auto_deps.py}}} file which is in the {{{allmydata-tahoe}}} package. For example, on my machine it is: {{{ $ grep -A1 -B1 -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata/_auto_deps.py # Sqlite comes built into Python >= 2.5, and is provided by the "pysqlite" # distribution for Python 2.4. -- if sys.version_info < (2, 5): # pysqlite v2.0.5 was shipped in Ubuntu 6.06 LTS "dapper" and Nexenta NCP 1. install_requires.append("pysqlite >= 2.0.5") }}} The {{{_auto_deps.py}}} ought to be the same as the current trunk version: [source:_auto_deps.py]. Finally, please look at the {{{requires.txt}}} file in the {{{.egg- info}}}, like this: {{{ $ grep -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata_tahoe.egg- info/requires.txt pysqlite >= 2.0.5 }}} This shows that on my machine {{{pysqlite}}} appears in the {{{requires.txt}}}, because the .egg-info was built using Python 2.4. If you built with Python 2.5 then {{{pysqlite}}} should not appear in your {{{.egg-info}}}. You could try rm'rf the whole {{{allmydata_tahoe.egg- info}}} directory and rebuilding. Thanks! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 14:38:33 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 21:38:33 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.cbeaf0fd2b138d4931ca04faf753b360@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Changes (by zooko): * owner: [4-tea-2] => warner Comment: Ooh, I get it. [4-tea-2] didn't build the {{{.egg-info}}} himself, it came with the .deb that we built. So, we have to decide whether we want our .deb to depend on {{{python-pysqlite}}}, or whether we want it to not completely support Python 2.4 (or we could even build different .deb's for each Python version). I'm okay with either of those two options. (Not so much with the Python-version-specific debs option.) I guess that the build machine where the Debian packages are being created is using Python 2.4, so {{{pysqlite}}} is being added to the {{{.egg- info}}}'s {{{requires.txt}}}. Assigning to Brian to tell me what our policy is about .debs and pysqlite. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 14:53:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 21:53:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.2cd9af12ac0e38a7a8349689d1958fd2@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): Sigh. So, building separate .debs for py2.4/py2.5/py2.6 is painful (nobody does it anymore, now that debian-python-policy generates .pycs at install time). And I don't believe the .deb control-file "Depends:" format isn't rich enough to express (py>=2.5 or pysqlite). On the other hand, the "standard" version of python is well-known per- platform (it's 2.4 in the older ones, 2.5 in the newer ones, and I guess Ubuntu Karmic will probably have 2.6). So we could say that when we build a .deb on a platform with /usr/bin/python=2.4, we declare the dependency upon pysqlite, and when we build it on a py2.5-using platform, we don't. To implement this, we should update the misc/*/debian/control files to declare the pysqlite dependency on the older platforms, and leave the newer ones alone (not declaring it). This also means that we should probably stop using the feisty rules for all platforms.. we at least need to identify the release for which python switched to 2.5 and use a newer set of rules for the 2.5-using set. I'll try to look at this today, but I'm not sure I'll get the time. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 18:25:39 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 01:25:39 -0000 Subject: [tahoe-dev] [tahoe-lafs] #683: handle arbitrary URIs in directories In-Reply-To: <038.2582f5dad5cb978edfed1f357bfc6e52@allmydata.org> References: <038.2582f5dad5cb978edfed1f357bfc6e52@allmydata.org> Message-ID: <047.1ccc7c8ba20c048bd9fa86981af1db8e@allmydata.org> #683: handle arbitrary URIs in directories ---------------------------+------------------------------------------------ Reporter: kpreid | Type: defect Status: new | Priority: major Milestone: undecided | Component: code-dirnodes Version: 1.3.0 | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): [3960] changes the way dirnodes are processed to tolerate unrecognized URIs. This should make tahoe-1.5 able to survive new formats that come from the future (i.e. if a 1.5 client tries to read or modify a directory which has new-format entries which were placed there by some 1.6-or-beyond version). It's at least a start. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 19:23:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 02:23:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.1ed7db4171fc132c4f72f7b0021bc38b@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): Ok, I think I understand this better now. * debian/etch ships with python2.4 as the standard /usr/bin/python (http://packages.debian.org/etch/python). * our most recent tahoe .deb for etch (http://allmydata.org/debian/dists/etch/tahoe/binary-i386/allmydata- tahoe_1.4.1-r3916_all.deb) was built on an Etch system with python2.4 * therefore its {{{_auto_deps.py}}} correctly declares a requirement on pysqlite, enforced at runtime, via the call to {{{require_auto_deps()}}} in {{{src/allmydata/__init__.py}}} * the .deb's .egg-info/requires.txt correctly includes a line which requires {{{pysqlite >= 2.0.5}}}, which will probably be enforced by anything that goes poking around .egg-info . In particular, the code that was added to src/allmydata/scripts/runner.py (the main CLI entry point) to call {{{pkg_requires.require('allmydata-tahoe')}}} will read the .egg-info data and enforce its requirements. * however, the packaging (i.e. debian/control) does not declare this requirement. This is a bug. Installing just this .deb (and no extra non-Depends-required packages) and then running the supplied /usr/bin/tahoe with python2.4 should fail with the same sort of exception displayed, because {{{_auto_deps.py}}} insists on having pysqlite, but APT did not know that python-pysqlite2 should have been installed. * But, 4-tea-2 appears to be using python2.5 to start the Tahoe node on this etch system, because the traceback listed above shows {{{/usr/lib/python2.5/site-packages}}}. Etch includes a python2.5 package (http://packages.debian.org/etch/python2.5), but it does not touch /usr/bin/python, so either 4-tea-2 has set up a symlink or installed python2.5 outside the awareness of APT (perhaps directly to /usr/local/bin). This is what was confusing me. * the .deb provides an allmydata_tahoe-1.4.1_r3916-py2.4.egg-info/ directory, and does not provide any .egg-info data for python2.5 . I don't know how setuptools reacts to this: would it refuse to believe that tahoe was actually installed when run under py2.5? If so, then I'm still confused as to where 4-tea-2's tahoe was finding the .egg-info data. * when run under python2.5, the code in {{{_auto_deps.py}}} should *not* require pysqlite * however the .egg-info line (generated on our Etch buildslave) still requires it, and thus the scripts/runner.py call will demand it. Why is that {{{pkg_resources.require('allmydata-tahoe')}}} line there? If it weren't, then the correct runtime-computed dependencies in _auto_deps.py would be enforced, and the incorrect package-build-time dependencies in .egg-info/requires.txt would not. Now, it happens that the .deb packaging for this is incorrect, and the Etch .deb should declare a dependency on python-sqlite2 because that's needed by the default version of python on an Etch system, and it happens that fixing the packaging would also happen to make this problem go away. But it's still wrong: this is a dependency that changes depending upon the version of Python used to run bin/tahoe, and recording the value that was computed at package-build time seems inappropriate. It should be possible to run tahoe under py2.5 on this system without having pysqlite installed. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 19:29:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 02:29:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.049fe36022d8d9bfa0710dcb477daff3@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by zooko): I guess a quicker solution is just to have the {{{allmydata-tahoe}}} .deb always depend on {{{python-sqlite}}}. While we're at it, I can swap the order of attempted-import in [source:src/allmydata/__init__.py] and [source:src/allmydata/scripts/backupdb.py], so that whatever {{{python- sqlite}}} package that is installed is the one that actually gets used (in preference to the {{{sqlite}}} module that comes in the Python standard library). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 20:29:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 03:29:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.2fbe75c17f968a8f126390e1d12f00d0@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by zooko): Brian has implemented much of this, but it isn't commented-in, documented, etc. in the current trunk. The next step (after the Tahoe v1.5.0 release) is to experiment with use cases and UI. Anybody out there want to actually use this for something? Shawn Willden has mentioned that he wants to make sure his mom's PC gets {{{K}}} shares. Jake Appelbaum has mentioned that he wants to make sure non-Tor-hidden-server nodes get {{{N-K}}} shares. Some corporations have mentioned that they would like to send approximately the same number of shares to each of {{{Q}}} different geographically distributed data centers. I think we need some brave user like Shawn or Jake to try this feature out and complain about how it doesn't do what he wants. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 21:26:03 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 04:26:03 -0000 Subject: [tahoe-dev] [tahoe-lafs] #708: graceful handling of capabilities in a format from the future that you can't understand In-Reply-To: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> References: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> Message-ID: <046.4ebceb7e4d279aa03cb1adae2dcce551@allmydata.org> #708: graceful handling of capabilities in a format from the future that you can't understand -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Resolution: fixed | Keywords: forward-compatibility Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Over on [comment:5:ticket:683 ticket #683, comment 5], Brian wrote: > [3960] changes the way dirnodes are processed to tolerate unrecognized URIs. This should make tahoe-1.5 able to survive new formats that come from the future (i.e. if a 1.5 client tries to read or modify a directory which has new-format entries which were placed there by some 1.6-or-beyond version). It's at least a start. As I understand it, the fact that we can't add unknown caps into a directory means that we can't copy directories which contain caps from the future. (If we do copy such a directory then the entries in it which had new-style caps will be omitted from the newly created copy of the directory). In theory it should be possible to do that safely just by copying the write-cap field from the entry in the source dir into the write-cap field of the newly created entry in the target dir, and likewise copying the read-cap. Brian: would there be any danger in that approach? I don't know how important it would be for clients from the past to be able to copy your new-style caps. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 21:57:21 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 04:57:21 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.cfd527d59088cddae35e9cfbd3c11aa6@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by swillden): Even before the ability to ensure that specific servers get {{{k}}} shares, I want to be able to specify that the local node get ''no'' shares. For backup purposes, you're probably going to use a local (often on the machine being backed up!) Tahoe node as the interface to the grid, and it makes little sense to store a share there, at least if the primary purpose of your backups is disaster recovery. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 22:07:23 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 05:07:23 -0000 Subject: [tahoe-dev] [tahoe-lafs] #422: have buildslaves automatically build debian packages of foolscap, zfec, pycryptopp, etc. In-Reply-To: <037.be7b09b76050f4a640d70974371002b3@allmydata.org> References: <037.be7b09b76050f4a640d70974371002b3@allmydata.org> Message-ID: <046.d7cf98978ad8b1234ea381769b5fc3f7@allmydata.org> #422: have buildslaves automatically build debian packages of foolscap, zfec, pycryptopp, etc. -------------------------+-------------------------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.0.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by zooko): [http://allmydata.org/buildbot-pycryptopp/waterfall pycryptopp buildbot] and [http://allmydata.org/buildbot-zfec/waterfall zfec buildbot] are now automatically building .deb's. Those .deb's are not currently being uploaded to anywhere and integrated into an apt repository, though. Also zfec .deb's depend on other .deb's (pyutil and argparse) which are not yet available. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 22:13:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 05:13:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.5f7fcaf3af9ecba367de10baa4913267@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Changes (by zooko): * owner: => zooko * status: new => assigned * milestone: undecided => 1.5.0 Comment: I've implemented most of this, but not yet the part about: * Add a hyperlink with anchor text "What is this?" next to the "mutable" check-box which explains what mutable files are and how you never want to use them. Hm, maybe this also means hide the "mutable" check-box behind some sort of expert-user-mode? :-/ I haven't committed it to trunk yet, but probably will before the v1.5 release. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Thu Jul 2 23:41:39 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Fri, 3 Jul 2009 00:41:39 -0600 Subject: [tahoe-dev] GridBackup functional (barely) In-Reply-To: <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> Message-ID: <200907030041.39696.shawn-tahoe@willden.org> After far too long, GridBackup works, for a suitable definition of "works". Specifically, it scans the file system, produces and stores the backup logs (full and incremental) containing file metadata, and generates a queue of upload jobs to back up the actual file content. It also backs up the file content, and produces the read cap index entries that links the backup logs to the content. When doing backups, it packs small files together to help speed up the upload process. Very small files are actually stored in the read cap index directly, and tiny files (including empty files) are stored in the backup log. It also prioritizes the uploads, favoring recently-modified files, small files and user files. The favoritism for recently-modified files in particular tends to mean that the stuff you're actually working on gets uploaded very early. Both the backup scanning and upload processes are completely interruptable and restartable. The initial full scan is slow (gotta hash your whole drive, basically), but subsequent incremental scans are pretty fast. Uploading is slow, but that's pretty much unavoidable (though I do have some plans to improve it a little via parallelism). I am actually going to begin using GridBackup for my daily backups tomorrow, after I get an allmydata.com account and figure out how to point my local Tahoe node at it. In a couple of months I plan to set up a friendnet and switch to using that. What doesn't work yet is: 1. Difference-based backups. All of the infrastructure is in place to do them, and adding that functionality should be nearly trivial, but I want to focus on testing the basic backup functionality for a while. 2. Restores. Yeah, this is a rather important bit :-). It's my next major task. It also shouldn't be tremendously difficult, though, since it will use all of the code used to create the logs, read cap index, etc., to parse them. 3. Cross-platform support. All development and testing has been done on Linux, and while there is some stuff in place to handle Windows and OS X issues, testing, debugging and, I'm sure, fixing is required. 4. GUI. Right now the UI consists of two scripts (GridBackup and GridUpload) and a config file (~/.GridBackup/config.ini). The scripts don't even have any command line options; you have to configure everything through the config file. Somewhere in there I'm also going to do a major code restructuring. I've learned a lot about how Python code should be put together so the current codebase is an inconsistent mishmash of approaches. Other items that are even further away are: slick platform-appropriate installers that install and configure both GridBackup and Tahoe; integration of GridBackup and GridUpload into a single, daemonable (twistd) process that uses twisted to achieve some level of upload parallelism to improve performance; and other file system scanners, including one for OS X that uses fsevents, and a cross-platform scanner that operates remotely, to enable a 'backup server'. Anyway, I have lots more work to do, but the code actually successfully makes backups now. Anyone who is interested can grab the current code from my github-hosted repo, at: http://github.com/divegeek/GridBackup If you decide to play with it, you'll almost certainly run into problems. Feel free to ask questions here, or mail me directly. I'll try to actually pay a little attention to my IRC client which hangs out on #tahoe, too. Shawn. From trac at allmydata.org Thu Jul 2 23:32:55 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 06:32:55 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a647a6dd114542370e41c70cd1e727ac@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): It would be interesting if it were flakey hardware, since the same problem happens pretty uniformly across otherwise stable and independent machines that have similarities: they are x86, pretty old CPUs, etc. (Like, one is crazy rock-solid stable up for 600+ super busy days machinery, so solid I run a strata 2 ntp on it.) I downloaded and built crypto++ manually, and running the tests as bdew did I found 100% success so long as I kept the /dev/random entropy pool fed. (Else it would fail because it would wait too long for random data to appear.) I will attempt these other things that bdew seems so facile with. :) -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From shawn-tahoe at willden.org Fri Jul 3 07:18:49 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Fri, 3 Jul 2009 08:18:49 -0600 Subject: [tahoe-dev] Client/helper RAM usage Message-ID: <200907030818.49575.shawn-tahoe@willden.org> Are files being processed (FEC-encoded, etc.) loaded into RAM? Is it possible to upload a 2 GiB file on a machine with only 512 MiB RAM? Is there any way to estimate the memory consumption that will be required? I'm thinking about putting a client or maybe a helper on a Linux Virtual Server that I have, but it has only 512 MiB of RAM, and that is a hard figure. There is no swap. Shawn. From trac at allmydata.org Fri Jul 3 09:21:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 16:21:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #708: graceful handling of capabilities in a format from the future that you can't understand In-Reply-To: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> References: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> Message-ID: <046.51d70b736d3358965b036e89b856ad5e@allmydata.org> #708: graceful handling of capabilities in a format from the future that you can't understand -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: forward-compatibility Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by warner): * component: code-encoding => code-dirnodes Comment: The internal 'move' method does just that, and the JSON representation of a directory includes all the information we have about the unknown object (i.e. both the writecap and the readcap). What I don't know is how the CLI-side "tahoe cp" works, specifically if the put-lots-of-caps-at-once dirnode webapi operation will accept the same "unknown cap" structure that the JSON representation hands down. Also, I wanted to discourage people from adding new unknown caps to a directory (because they might just be adding complete junk, or a typo, or a blank string, and it'd be nice to detect that early), so the current code is liberal in what it accepts from the backend, but strict in what it accepts from the frontend, and this might prevent the frontend-based tools from doing this sort of copy. So yes, I think that approach would be safe, and it might already work. (of course we have no way to tell if the unknown-cap is a file or a directory, or something even more exotic, so we might be creating a hardlink to a mutable directoryish-thing when the rest of the copy operation was trying to make a deepcopy of individual files). The test would need to go in test_cli.py where it tests the "tahoe cp" operation. grep around the test suite for !UnknownNode, you have to be a bit sneaky to get the cap-from-the-future into a directory to start with. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 09:24:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 16:24:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.4656b18e8fb136c0a29393b4676f4337@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): I updated the debian packaging rules last night, and now the py2.4-based distributions (etch and edgy, I believe) add a python-sqlite2 dependency, and the more recent ones do not. I'd prefer to not force py2.5-based platforms to install an extra package. And I'd prefer that we use the stdlib {{{sqlite3}}} module in preference to an external {{{pysqlite2}}} module. Think about it this way: when py2.4 is long dead, do you still want us to be requiring people to install a third-party module to duplicate stdlib functionality? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 12:44:30 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 19:44:30 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.616b702ff82b2832be0c56e601e07d01@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): I have written a test program for crypto++ which demonstrates the same problem for a SHA256: the results differ based on whether the input is split up on an odd boundary or not. I have compiled my crypto++ library with and (I believe) without the optimized assembly routines. I will add my program to this ticket. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 12:51:28 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 19:51:28 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a31a76be36a536d1731865be6f5ca33c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): Replying to [comment:17 midnightmagic]: > I have written a test program for crypto++ which demonstrates the same problem for a SHA256: > the results differ based on whether the input is split up on an odd boundary or not. I have > compiled my crypto++ library with and (I believe) without the optimized assembly routines. I lied. Manually disabling the optimized assembly makes my sample program output the correct results. Here are the details: {{{ g++ -DCRYPTOPP_DISABLE_X86ASM -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3 -pipe -I. -c mytest2.cpp && g++ -o mytest2 mytest2.o -L. -lcryptopp && ./mytest2 }}} will work and results in: {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E }}} Just build crypto++ and then plunk mytest2.cpp in the same directory as the libcryptopp.a was built in, and run the above. {{{ g++ -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3 -pipe -I. -c mytest2.cpp && g++ -o mytest2 mytest2.o -L. -lcryptopp && ./mytest2 }}} will NOT work, and results in: {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: D159F4747C64249633829F8355DC84678A67CCCE92ABC8F770D33111AFB43E42 STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 13:15:12 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 20:15:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.bf5403de5324d1d822b8664ec82a8488@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): Way to go, MM (and BD)! I will report this to the Crypto++ maintainers. In the meantime, we should probably think about disabling optimized assembly (at least on some platforms?) so that we can go ahead with the Tahoe-1.5 release. Do you have a platform were the optimized assembly *does* work so you can see how much different it makes in performance? Thanks. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 13:22:33 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 20:22:33 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.933e95cb7bf3195ccba198a692cecebe@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): I posted to Crypto++ list: http://groups.google.com/group/cryptopp- users/browse_thread/thread/d3fb86b84a4793b7 I wouldn't be surprised if Wei Dai (author of Crypto++) can come up with a fix for this fast enough to go into Tahoe-LAFS v1.5.0. But in case he doesn't, Midnight Magic is writing a patch for pycryptopp's setup.py to turn off optimized assembly on some platforms. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 15:07:00 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 22:07:00 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.e2bfc308d52604665222aa1fbec77e6c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): I posted there too. Anyway, I've got a diff that allows pycryptopp to succeed with the chunktests Werner built. It's relatively specific, and it seems unfair. The older machinery is that which would most benefit from assembly optimizations. Anyway.. I'll attach the diff in just a moment. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 15:18:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 22:18:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.0d9e82740575998bee30bcf1a601413c@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by midnightmagic): My use-case is to specify which servers receive the lion's share of the blocks, for two reasons: 1. To ensure that known-reliable machines receive the most blocks. 2. To ensure that the space is used in a way I want to control manually. For example, even though one of my machines in my home storage grid has multiple TB of relatively reliable storage space on it, I would prefer to use that as sparingly as possible and instead dump most of them on machines that are allocated specifically to the task of '''being''' a Tahoe storage node. Also, out of those Tahoe nodes that are participating, I happen to know that machines X, Y, and Z are extremely reliable, whereas others have had flakey drive bus issues in the past. I want X, Y, and Z to definitely get blocks.. and the rest of them, maybe. So I'll be using this feature for sure. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:01:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 23:01:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.98a2f326e3b37c8fc702c77c78211217@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): SUCCESS! Rebuilding pycryptopp without ASM optimizations makes it pass the chunked SHA256 test, and setting PYTHONPATH to that top-level directory makes "tahoe ls" Just Work on an existing grid, and the failing command NOW WORKS PERFECTLY. So there is a patch in pycryptopp/ticket/24 for setup.py which detects the platform involved and turns off assembly optimizations in just that specific platform and bit-width (32bit). I would say that, if bdew could do the same and it works, we can put a platform detection for his also and likely close all these tickets out until crypto++ fixes their CPU features detection. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:06:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 23:06:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.4035bb4464404d4550053918341b6ad1@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Update: since the fix mentioned in #738 (disabling crypto++ assembly optimizations) was implemented in my local copy here, python2.5 setup.py test NOW COMPLETES THROUGH THE SPOT WHERE IT FROZE BEFORE!! So this specific issue I think, if those other ones can be closed, can be closed also once we verify that the same fix might work for bdew. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:43:49 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 23:43:49 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.86ee9d6f867e6aa499cc2044a3b151b4@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): It looks like this might be specific to AMD XP, which I don't have access to. I can't reproduce it on my AMD 64 machine. Can someone give me a shell account on an AMD XP machine so I can debug this? If so, please use http://www.ibiblio.org/weidai/mailme.php to email the login info to me. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 17:12:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 00:12:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.8c78605340b609db28c9977e1c06421b@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): Okay, I've got tests in place -- I'll attach them in a bit. One minor nit: I was working on the assumption that {{{_pack_contents}}} and {{{_unpack_contents}}} were inverses -- i.e., if I do {{{_pack_contents(_unpack_contents(string))}}}, I should get {{{string}}} back. I made this a condition in my test function, but it failed -- {{{_pack_contents}}} would return a different string, even when fed with the unmodified dict returned from {{{_unpack_contents}}}. But all of the data inside the dict seem the same -- at least the rocaps, rwcaps, and metadata. Is this normal? Also, the hardcoding of the base32-encoded representation of that directory tree is kind of horrific -- any ideas for a better way of doing that? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 17:12:56 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 00:12:56 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.c011d5435c295a51ff105e9dd2762105@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): Never mind, I reproduced it on my non-SSE2 Celeron processor. I'm working on a fix. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 18:25:12 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 01:25:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.d9295b0d82bbb508950e84028aa805e6@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): Replying to [comment:23 weidai]: > Never mind, I reproduced it on my non-SSE2 Celeron processor. I'm working on a fix. Perhaps not so coincidentally, a non-SSE2 Celeron is running on one of the other machines where this problem is also visible. (Amazing how fast you got on top of this.. thanks eh! I appreciate you helping me breathe more life into what might otherwise be my computer graveyard..) -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 20:05:58 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 03:05:58 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.9cbbb85fb8fd3c9adb82238742ba8677@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): I've attached a fix. Please try it out. I'll explain the bug on the Crypto++ list later if anyone is curious. I'll also add some additional testing code to the next release of Crypto++ to catch this type of bugs. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 20:14:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 03:14:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.22e809842c35e2b8b2e0476e82904e6c@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): [source:src/allmydata/dirnode.py#L201] The reason is that the encryption of the write-cap uses a random IV. There isn't an easy way to make that deterministic and still secure right now, so we can't rely on it for testing. (Hm, for future reference -- i.e. after Tahoe v1.5.0 -- maybe we ''could'' generate the random IV as the secure hash of the write cap itself. That would be make it deterministic without, as far as I can currently see, losing security.) As to the base32-encoded directory tree, that looks okay to me. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 20:58:46 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 03:58:46 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.81bb933202d9647205c57f74085366ad@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): Okay. Given that, do you think that the other tests in {{{test_unpack_and_pack_behavior()}}} are going to be enough? I've finished doing a few things. * I've changed the tests for {{{_pack_contents}}} and {{{_unpack_contents}}} in {{{test_dirnode.py}}} to not check the output of {{{_pack_contents}}} for equality, since that will always fail at the moment. I've also moved a misplaced comment. * I've added tests for {{{CachingDict}}}, the data structure described by warner above (cool idea, by the way). * I've added an implementation of {{{CachingDict}}} to {{{dirnode.py}}}. It basically does what warner suggested that it do. * I've changed {{{_pack_contents}}}, {{{_unpack_contents}}}, and {{{_create}}} to use {{{CachingDict}}}s instead of {{{dict}}}s. With these done (tests.txt, dict.txt, optimizations.txt applied), {{{python setup.py test}}} passes on my machine. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 22:53:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 05:53:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.393d376365e151302ae966d6e669a513@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): It turns out that the logic for the first point was already in {{{tahoe_mv.py}}} -- just not the behavior itself. From a cursory look at the webapi docs, appending something to a directory is as simple as sticking the name of the child after the name of the directory in the URL -- so if we want to add {{{tahoe:test.pdf}}} to {{{tahoe:testdir/}}}, we can make {{{ http://127.0.0.1:3456/uri//testdir/test.pdf }}} unless I'm misunderstanding something. That's basically what I did in this patch. I haven't looked into implementing the second part yet. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 00:37:41 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 07:37:41 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a7c615311fd882385ffa9af95e3fc0f2@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I'm getting a compile error with this patch: (gcc 4.3.3-12, binutils 2.19.1) {{{ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict- prototypes -fPIC -I. -I/usr/include/python2.5 -c cryptopp/sha.cpp -o build/temp.linux-i686-2.5/cryptopp/sha.o -w cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cryptopp/sha.cpp: Assembler messages: cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' error: command 'gcc' failed with exit status 1 }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Sat Jul 4 08:24:03 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 15:24:03 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d193d555dd3973788582d1e3af4a265e@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): I looked at your patch and it looks right to me. I think I would change the {{{if path.endswith("/"):}}} to {{{if to_url.endswith("/"):}}}. Also the patch description is not "Make mv move files into directories, instead of overrwriting directories.", because this patch doesn't (yet) achieve that in the case that the target directory name doesn't have a trailing slash. We need a test to go along with this patch -- perhaps just add some statements to {{{src/allmydata/test/test_system.py}}} to move a file into a directory-with-trailing-slash and assert that it works. Err, wait a second -- there is already a test of this in [source:src/allmydata/test/test_system.py#L973]. It says: {{{ d1.addCallback(self.log, "mv 'P/sekret data' P/personal/") d1.addCallback(lambda res: home.move_child_to(u"sekrit data", personal)) }}} This test currently passes, even without your patch that looks for a trailing slash and appends the filename to the target URL. Why is that? We should have a test that fails before we apply a patch to make the test pass. For the next part of the issue, we should add a test that tries to mv a file to a directory, without trailing slash. According to Kevin Reid's suggestion, Tahoe should refuse to do anything in that case, e.g. if 'sekret data' is a file and 'personal' is a directory, then {{{tahoe mv "sekret data" personal}}} should emit an error message and make no change. The test should make sure that tahoe emits a suitable error message, in addition to making sure that it doesn't move or remove the "sekret data" file. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 4 11:09:57 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 04 Jul 2009 20:09:57 +0200 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? Message-ID: <87hbxs7216.fsf@gnu.org> Hello! I noticed that http://allmydata.org/source/tahoe/releases/allmydata-tahoe-1.4.1.zip contains a file `allmydata-tahoe-1.4.1/src/allmydata/web/tahoe.css' that's only 7-line long, whereas the Debian package for Etch has `/usr/share/pycentral/allmydata-tahoe/site-packages/allmydata/web/tahoe.css', which is 136-line long. Am I missing something? Thanks, Ludo'. From zooko at zooko.com Sat Jul 4 14:16:29 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sat, 4 Jul 2009 15:16:29 -0600 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? In-Reply-To: <87hbxs7216.fsf@gnu.org> References: <87hbxs7216.fsf@gnu.org> Message-ID: <129A1BD4-0439-4E49-AD1A-5F83A2378343@zooko.com> Hello, Ludovic Court?s! Kevin Reid contributed an improved tahoe.css since the Tahoe v1.4.1 release. If the .deb for Etch that you are using is more recent than v1.4.1 then perhaps this explains it -- you have Kevin's new CSS file. Here's how to tell: is the web interface incredibly ugly and klunky? Then you're using the old CSS file! ;-) Oh, actually here is a better way to tell: if the version you are running is equal to or newer than v1.4.1-r3881 then you have the new CSS. Regards, Zooko From trac at allmydata.org Sat Jul 4 14:05:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:05:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #668: easy_install allmydata-tahoe is broken In-Reply-To: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> References: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> Message-ID: <049.b40fe6076cbee785fb62ca9701741d18@allmydata.org> #668: easy_install allmydata-tahoe is broken -----------------------+---------------------------------------------------- Reporter: francois | Owner: zooko Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.3.0 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by ndurner): From IRC: {{{ (20:58:05) ndurner: Hmpf, my build instructions for Windows don't work anymore (22:42:48) zooko: I think your report should go into ticket #668. }}} {{{ E:\allmydata-tahoe-1.4.0>python setup.py build Not found: tahoe-deps Not found: ../tahoe-deps Traceback (most recent call last): File "setup.py", line 346, in zip_safe=False, # We prefer unzipped for easier access. File "C:\Python25\lib\distutils\core.py", line 112, in setup _setup_distribution = dist = klass(attrs) File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 219, in __init__ File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 243, in fetch_build_eggs File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 522, in resolve File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 758, in best_match File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 770, in obtain File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 286, in fetch_build_egg File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 452, in easy_install File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 482, in install_item File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 661, in install_eggs File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 936, in build_and_install File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 927, in run_setup distutils.errors.DistutilsError: Setup script exited with error: Python was buil t with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 4 14:25:01 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 04 Jul 2009 23:25:01 +0200 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? References: <87hbxs7216.fsf@gnu.org> <129A1BD4-0439-4E49-AD1A-5F83A2378343@zooko.com> Message-ID: <873a9cnnte.fsf@gnu.org> Zooko Wilcox-O'Hearn writes: > Hello, Ludovic Court?s! Hi! > Kevin Reid contributed an improved tahoe.css since the Tahoe v1.4.1 > release. If the .deb for Etch that you are using is more recent than > v1.4.1 then perhaps this explains it -- you have Kevin's new CSS file. OK. > Oh, actually here is a better way to tell: if the version you are > running is equal to or newer than v1.4.1-r3881 then you have the new > CSS. Debian's package version is 1.4.1-r3916, which explains the difference. Thanks, Ludo'. From trac at allmydata.org Sat Jul 4 14:31:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:31:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #747: Files uploaded to FTP server never appear Message-ID: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> #747: Files uploaded to FTP server never appear ---------------------------+------------------------------------------------ Reporter: ndurner | Owner: zooko Type: defect | Status: new Priority: major | Milestone: undecided Component: code-frontend | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ While connected to Tahoe's builtin FTP server: {{{ ncftp / > ls personal-bzr/ ncftp / > put kv.tar kv.tar: 1,24 MB 983,36 kB/s ncftp / > ls personal-bzr/ }}} This is with Tahoe 1.3, Twisted 8.2.0 on Windows XP. Even with the patch from FTP-and-SFTP.txt applied, the assertion at ftpd.py:293 still failed, so I uncommented that line. The last hunk in the patch didn't apply because the file is missing (nothing unusual, see Ticket #688). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 14:44:07 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:44:07 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid Message-ID: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> #748: Exception: when trying to create directory on testgrid -------------------------+-------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- In a random attempt to see what the new CSS wui looks like, I connected to testgrid.allmydata.com:3567 just now. I clicked the "Create a directory" button and was presented with the exception listed above. I'm afraid I see no further information to provide but the error is reproducable. The actual HTML source returned is the following: {{{ Exception

<class 'allmydata.mutable.common.UncoordinatedWriteError'>:

<class 'allmydata.mutable.common.UncoordinatedWriteError'>:

}}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 14:54:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:54:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.fc62614d726b451f3523311e61d71392@allmydata.org> #748: Exception: when trying to create directory on testgrid -------------------------+-------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by zooko): Please look in your {{{~/.tahoe/logs/incidents}}} directory and attach any incident report files that occurred around this time. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Sat Jul 4 23:34:15 2009 From: warner at lothar.com (Brian Warner) Date: Sat, 4 Jul 2009 23:34:15 -0700 Subject: [tahoe-dev] Client/helper RAM usage In-Reply-To: <200907030818.49575.shawn-tahoe@willden.org> References: <200907030818.49575.shawn-tahoe@willden.org> Message-ID: <20090704233415.7f0ffab3@Cookies.lan> On Fri, 3 Jul 2009 08:18:49 -0600 Shawn Willden wrote: > Are files being processed (FEC-encoded, etc.) loaded into RAM? Is it > possible to upload a 2 GiB file on a machine with only 512 MiB RAM? > Is there any way to estimate the memory consumption that will be > required? Only in small pieces, yes, and yes, respectively :) We've carefully designed both the mutable/immutable encoding format and the immutable encode/decode algorithms to minimize the memory footprint they need while operating. They only ever work on one segment at a time, which (by default) is 128KiB. In general, the RAM footprint is a small multple of the segment size (maybe 3 or 4). We read in a segment's worth of plaintext from disk, encrypt it (then throw out the plaintext), encode it (then throw out the ciphertext), push it (then throw out the shares). It works the same way on the download side. The mutable publish/retrieve code does not yet do this: because we're still operating under "SDMF" rules, we only process one segment per mutable file. The maximum size limit for mutable files was recently removed, which means the maximum segment size limit was removed (segsize=filesize). So if you actually upload a 1GB *mutable* file, then you'll need 3 or 4 GB of ram. When we implement "MDMF", we'll change the mutable code to stream segments out the same way that immutable files do, and then switch back to a 128MiB segsize. There's a considerable static footprint, of course (this is python, after all). The buildbot performance page has a graph of this footprint over time. On a 32-bit machine, it's something like 40MB. > I'm thinking about putting a client or maybe a helper on a Linux > Virtual Server that I have, but it has only 512 MiB of RAM, and that > is a hard figure. There is no swap. I'd expect that to work just fine. cheers, -Brian From trac at allmydata.org Sat Jul 4 23:51:01 2009 From: trac at allmydata.org (pycryptopp) Date: Sun, 05 Jul 2009 06:51:01 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.05ad58ea6da4185fcca68b7f792dfc29@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Ok, the second patch works fine for me and fixes both this and http://allmydata.org/trac/tahoe/ticket/737 -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Sun Jul 5 07:00:15 2009 From: trac at allmydata.org (pycryptopp) Date: Sun, 05 Jul 2009 14:00:15 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.586311b89c1daa68da05fdd2edde260c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments --------------------------+------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: closed Priority: critical | Version: 0.5.1 Resolution: fixed | Keywords: integrity Launchpad_bug: | --------------------------+------------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Fixed by [20090705135508-92b7f-c3c3ad54fdfc52fa22c1c2fbd70479ba27802952]. Thanks, MM, BD, Wei Dai! -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From zookog at gmail.com Sun Jul 5 07:59:16 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 08:59:16 -0600 Subject: [tahoe-dev] Client/helper RAM usage In-Reply-To: <20090704233415.7f0ffab3@Cookies.lan> References: <200907030818.49575.shawn-tahoe@willden.org> <20090704233415.7f0ffab3@Cookies.lan> Message-ID: Those graphs are linked from and described on this page: http://allmydata.org/trac/tahoe/wiki/Performance I don't think those graphs are reliable, though, as described in #227. However, I do think that those graphs, which are measuring vmsize, should show an upper bound on the number that you care about, so since the vmsize on that 32-bit machine is shown as being around 40 MB on our Performance page graphs, your virtual machine with 512 MiB should work juuuust fine. :-) I used Tahoe-LAFS on Zandr's 266 MHz ARM NAS box yesterday. It has 128 MiB of RAM. It seemed to work okay, but I had to stop experimenting with it since both my ISP and Zandr's router seem to be flaky at the moment (so I could no longer open TCP connections to that box). Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/227 # our automated memory measurements might be measuring the wrong thing From zookog at gmail.com Sun Jul 5 08:14:49 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 09:14:49 -0600 Subject: [tahoe-dev] Tahoe on little ARM CPU (Re: Client/helper RAM usage) Message-ID: On Sun, Jul 5, 2009 at 8:59 AM, Zooko O'Whielacronx wrote: > I used Tahoe-LAFS on Zandr's 266 MHz ARM NAS box yesterday. ?It has > 128 MiB of RAM. ?It seemed to work okay, but I had to stop > experimenting with it since both my ISP and Zandr's router seem to be > flaky at the moment (so I could no longer open TCP connections to that > box). Here is the statistics output from uploading a file from my home computer, using Zandr's NAS box as the Tahoe-LAFS gateway. It shows that there was only one storage server reachable to the NAS box and accepting shares at that time (I wonder why). It also shows that the speed of CPU-intensive parts like erasure coding and encryption was faster than the speed of pushing shares over the network to the storage servers. Regards, Zooko Uploading File... done! Upload Results: * URI: URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 * Download link: /uri/URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 * Sharemap: o 0 -> placed on [6fyx5u4z] o 1 -> placed on [6fyx5u4z] o 2 -> placed on [6fyx5u4z] o 3 -> placed on [6fyx5u4z] o 4 -> placed on [6fyx5u4z] o 5 -> placed on [6fyx5u4z] o 6 -> placed on [6fyx5u4z] o 7 -> placed on [6fyx5u4z] o 8 -> placed on [6fyx5u4z] o 9 -> placed on [6fyx5u4z] * Servermap: o [6fyx5u4z] got shares: #0,#1,#2,#3,#4,#5,#6,#7,#8,#9 * Timings: o File Size: 371474 bytes o Total: 32 seconds (11.4kBps) + Storage Index: 1.51s (246.1kBps) + [Contacting Helper]: # [Helper Already-In-Grid Check]: + [Upload Ciphertext To Helper]: () + Peer Selection: 3.97s + Encode And Push: 27 seconds (19.3kBps) # Cumulative Encoding: 6.51s (57.1kBps) # Cumulative Pushing: 12 seconds (29.2kBps) # Send Hashes And Close: 6.75s + [Helper Total]: Return to the Welcome Page From zookog at gmail.com Sun Jul 5 09:03:32 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 10:03:32 -0600 Subject: [tahoe-dev] Tahoe on little ARM CPU (Re: Client/helper RAM usage) In-Reply-To: References: Message-ID: I just did it again and this time it uploaded different shares to different servers (on the Test Grid). Again, the encoding is faster than the pushing, which is encouraging -- it means that this little ARM 266 MHz NAS box is more limited by the speed of its network connection to the storage servers than by its CPU power. (Although it is being limited by both, and this is a very light load -- only one client.) Regards, Zooko # URI: URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 # Download link: /uri/URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 # Sharemap: * 0 -> placed on [fcmlx6em] * 1 -> placed on [xiktf6ok] * 2 -> placed on [jfdpabh3] * 3 -> placed on [qvqv7jmm] * 4 -> placed on [7tlov7eg] * 5 -> placed on [lwkv6cji] * 6 -> placed on [wfninubk] * 7 -> placed on [trjdor3o] # Servermap: * [fcmlx6em] got share: #0 * [jfdpabh3] got share: #2 * [lwkv6cji] got share: #5 * [qvqv7jmm] got share: #3 * [trjdor3o] got share: #7 * [wfninubk] got share: #6 * [xiktf6ok] got share: #1 * [7tlov7eg] got share: #4 # Timings: * File Size: 371474 bytes * Total: 16 seconds (22.1kBps) o Storage Index: 509ms (729.3kBps) o [Contacting Helper]: + [Helper Already-In-Grid Check]: o [Upload Ciphertext To Helper]: () o Peer Selection: 994ms o Encode And Push: 15 seconds (34.2kBps) + Cumulative Encoding: 3.41s (109.0kBps) + Cumulative Pushing: 7.46s (49.8kBps) + Send Hashes And Close: 3.79s From shawn-tahoe at willden.org Sun Jul 5 11:34:17 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 5 Jul 2009 12:34:17 -0600 Subject: [tahoe-dev] Verifying file contents Message-ID: <200907051234.17821.shawn-tahoe@willden.org> Is there an easy/inexpensive way to check that a local file matches a remote (immutable) file? I could extract the Tahoe code that calculates the key from the local file, given the current convergence secret, and then compare that with the key in the URI. Or perhaps modify Tahoe to expose a "key calculation" service through the web API. Another thing I could do is upload the local file as an immutable file. If the file is already in the grid, this shouldn't take very long, and the returned URI should match. But if the local file doesn't match the remote file, the local file will be uploaded to the grid, which I don't really want. Is there a way (through the web API) to "fake" a PUT? That is, do everything except actually communicate with storage servers? A problem with all of these methods is that they break if the convergence key doesn't match, but I suppose that's a feature, not a bug. Thanks, Shawn From zooko at zooko.com Sun Jul 5 12:07:56 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 5 Jul 2009 13:07:56 -0600 Subject: [tahoe-dev] Verifying file contents In-Reply-To: <200907051234.17821.shawn-tahoe@willden.org> References: <200907051234.17821.shawn-tahoe@willden.org> Message-ID: <72EC76A2-59D8-4559-BEA7-ED1439D86C2E@zooko.com> On Jul 5, 2009, at 12:34 PM, Shawn Willden wrote: > Is there an easy/inexpensive way to check that a local file matches > a remote (immutable) file? I think this is pretty much the same thing as #713. If we had #713 then you could run that and then compare it to the cap to the remote file. Perhaps you should paste your new/related use case from this thread into that ticket. Regards, Zooko Tickets mentions in this e-mail: http://allmydata.org/trac/tahoe/ticket/713 # tahoe make-verify-cap From trac at allmydata.org Sun Jul 5 13:25:58 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 20:25:58 -0000 Subject: [tahoe-dev] [tahoe-lafs] #713: tahoe make-verify-cap In-Reply-To: <037.8ce6933cfa6fcba192d994ce27bd7c31@allmydata.org> References: <037.8ce6933cfa6fcba192d994ce27bd7c31@allmydata.org> Message-ID: <046.40e60990f6ab2a08db001a259b1e7366@allmydata.org> #713: tahoe make-verify-cap -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-frontend-cli | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by swillden): Replying to [comment:6 swillden]: > The need to have read cap index files has driven me to find other uses for them as well, uses that I'd have to find some other way to implement if I didn't have them, so between the performance issue and that one, I'm not sure I'd use a "dry run" option, even if it existed. As zooko pointed out in response to a recent mailing list question, this would be a good solution for a need I have to verify that a grid-based immutable file matches a local file. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 15:33:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 22:33:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.186de45a6287ad75e32b1076a66e53a8@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): I've run the benchmarks, and am attaching the results. To get the {{{unmodifed_results}}} file, I simply ran {{{bench_dirnode.py}}} against an unmodified checkout from trunk. To get the {{{modified_results}}} file, I needed to change a call to {{{dict}}} in {{{bench_dirnode.py}}} to refer to {{{dirnode.CachingDict}}} (otherwise {{{_pack_contents}}} raises an {{{AssertionError}}}). I'm attaching a patch that duplicates that change. I then applied {{{dict.txt}}}, {{{optimizations.txt}}}, and {{{tests.txt}}} to an otherwise unmodified source tree, and ran {{{bench_dirnode.py}}} against it. The results seem to favor the optimized implementation. I'm also updating dict.txt again -- hopefully for the last time -- to fix another *args related issue that I found. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 15:39:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 22:39:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #327: performance measurement of directories In-Reply-To: <037.99b82637bbb101b1ebc214b8c8630a60@allmydata.org> References: <037.99b82637bbb101b1ebc214b8c8630a60@allmydata.org> Message-ID: <046.060503d43e096893edacfcbbf4b9145d@allmydata.org> #327: performance measurement of directories ------------------------------------------+--------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: assigned Priority: major | Milestone: eventually Component: code-dirnodes | Version: 0.8.0 Keywords: test performance scalability | Launchpad_bug: ------------------------------------------+--------------------------------- Comment(by kevan): (this is in response to the first iteration of the benchmarking script; the issue is addressed in r3965) Cool! I've run it on my machine, and noticed that it actually shows slower results for the optimized code. I think that's a matter of methodology, though. From what I understand (having never used benchutil), to test pack_contents you're building a list of (name, child) tuples, feeding that into a dict, and then feeding the dict to pack_contents, and you're testing how long that takes for increasing numbers of tuples. To test unpack_contents, you're doing that, but saving the result of pack_contents in a global variable, then feeding that to unpack_contents to see how long it takes. If I'm right, we aren't seeing any speed improvements because the benchmark isn't actually testing the optimizations. In order to do that, we need to feed pack_contents a dictionary that was actually output from unpack_contents (or else built with set_both_items instead of __setitem__) -- that way, the set_both_items method of the dict wrapper will have been used to set the serialized value, and pack_contents will find and use that value, thus (ideally) making it faster. One way to do this might be to stick the logic for child creation into a more general setup method -- something which, when called, would generate + pack the list of children, and return the results of pack_contents on that list, for example. Then the init method for the unpack test could store that value in the global packstr variable, and work as it does now, while the init method for the pack test could unpack that, then feed the resulting dictionary into pack_contents again, where the optimizations would start working. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 22:07:00 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 05:07:00 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.6a3e6c4fbc8a28f92529d8660378cbc0@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Wow, Wei Dai is fast. Check it out, he's fixed the problem already in SVN: [http://cryptopp.svn.sourceforge.net/viewvc/cryptopp?view=rev&sortby=date&revision=470 Crypto++ SVN Rev 470] Impressive. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Mon Jul 6 19:31:14 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 6 Jul 2009 20:31:14 -0600 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> Message-ID: <51864C9C-C63D-4BA1-A6ED-21C203113440@zooko.com> On Jul 6, 2009, at 17:15 PM, tahoe-lafs wrote: > #737: python2.5 setup.py test runs CPU to 100% on 32-bit single- > core NetBSD "4" ... > Comment(by midnightmagic): ... > Other things fail, but.. looks like it's just because it's trunk and > certainly not the same problem this ticket references. Do tell! :-) Regards, Zooko From trac at allmydata.org Mon Jul 6 13:55:12 2009 From: trac at allmydata.org (pycryptopp) Date: Mon, 06 Jul 2009 20:55:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.b7ec91724c4574716b5e330c06005a99@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments --------------------------+------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: closed Priority: critical | Version: 0.5.1 Resolution: fixed | Keywords: integrity Launchpad_bug: | --------------------------+------------------------------------------------- Comment(by midnightmagic): Just wanted to chime in and state that trunk as of the patch committed, above, works perfectly on all machines of mine (NetBSD all, all Athlon XP, Celeron, Pentium III) and the problem pycryptopp test now passes perfectly: {{{ test_chunksize (pycryptopp.test.test_sha256.SHA256) ... ok }}} The following machines now report success, and this apparently with full ASM optimizations: __quickie__ NetBSD quickie 4.99.7 NetBSD 4.99.7 (quickie) #0: Tue Jan 2 14:47:23 PST 2007 root at shoggoth:/v/src/sys/arch/i386/compile/quickie i386 CPU: AMD Athlon XP 2500+ (686-class), 1837.62 MHz, id 0x6a0 __warp__ NetBSD warp 4.0_STABLE NetBSD 4.0_STABLE (warp) #0: Sat Jun 20 23:42:00 PDT 2009 root at shoggoth:/v/src/sys/arch/i386/compile/obj/warp i386 CPU: Intel Celeron (686-class), 1196.44 MHz, id 0x6b1 __draco__ NetBSD draco 3.99.23 NetBSD 3.99.23 (GENERIC) #0: Thu Jul 27 08:22:27 PDT 2006 root at shoggoth:/usr/netbsd- build/netbsd-3.99.23/i386/OBJ/usr/v/src/sys/arch/i386/compile/GENERIC i386 CPU: Intel Pentium III (686-class), 731.29 MHz, id 0x683 -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Mon Jul 6 16:15:39 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:15:39 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Using trunk from pycryptopp and trunk tahoe (via PYTHONPATH and deleting tahoe/support/lib/site-packages/pycrypto*) the test no longer fails and pushes through the spot where it failed before. .. with full crypto optimizations. Other things fail, but.. looks like it's just because it's trunk and certainly not the same problem this ticket references. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 16:18:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:18:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.e6f675ea1334573718e1bc6b68dc3311@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): pycryptopp in trunk now works perfectly (well the tests don't fail anyway) on all three machines, as listed in [http://allmydata.org/trac/pycryptopp/ticket/24 pycryptopp ticket 24]. Using pycryptopp trunk, I now have apparently perfectly-working tahoe nodes where before they were only remotely usable. Therefore, I believe this ticket can be closed, from my perspective. If 1.5.0 is going to include these fixes, then all's well! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 16:43:00 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:43:00 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.5261c72d223550638298d700da75f064@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by bdew): Same here, with fixed pycryptopp all the tests pass without hangs/rashes: http://allmydata.org/buildbot/builders/BlackDew%20debian- unstable-i386/builds/40/steps/test/logs/stdio -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 19:39:49 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 02:39:49 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings Message-ID: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings --------------------+------------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------+------------------------------------------------------- These six unit tests fail if optimization is turned on: {{{ $ PYTHONOPTIMIZE=2 PYTHONPATH=./support/lib/python2.6/site-packages/ trial allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte allmydata.test.test_uri.Constraint.test_constraint allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte allmydata.test.test_uri.Constraint.test_constraint allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named }}} ... {{{ allmydata.test.test_uri Constraint test_constraint ... [FAIL] allmydata.test.test_mutable Roundtrip test_corrupt_all_verbyte ... [FAIL] allmydata.test.test_web Web test_GET_unhandled_URI ... [FAIL] [ERROR] test_GET_unhandled_URI_named ... [FAIL] [ERROR] =============================================================================== [FAIL]: allmydata.test.test_uri.Constraint.test_constraint Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_uri.py", line 189, in test_constraint self.failUnlessRaises(AssertionError, uri.NewDirectoryURI.init_from_string, good) twisted.trial.unittest.FailTest: raised instead of AssertionError: Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 752, in _run self.getSuppress(), method) File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/utils.py", line 191, in runWithWarningsSuppressed result = f(*a, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_uri.py", line 189, in test_constraint self.failUnlessRaises(AssertionError, uri.NewDirectoryURI.init_from_string, good) --- --- File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 235, in failUnlessRaises result = f(*args, **kwargs) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/uri.py", line 337, in init_from_string bits = uri[mo.end():] exceptions.AttributeError: 'NoneType' object has no attribute 'end' =============================================================================== [FAIL]: allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_mutable.py", line 1033, in _do_retrieve self.failUnless(substring in "".join(allproblems)) twisted.trial.unittest.FailTest: None =============================================================================== [FAIL]: allmydata.test.test_web.Web.test_GET_unhandled_URI Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_web.py", line 341, in done % (which, substring, str(res))) twisted.trial.unittest.FailTest: test_GET_unhandled_URI: substring '400 Bad Request' not in '[Failure instance: Traceback (failure with no frames): : 500 Internal Server Error ]' =============================================================================== [FAIL]: allmydata.test.test_web.Web.test_GET_unhandled_URI_named Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_web.py", line 341, in done % (which, substring, str(res))) twisted.trial.unittest.FailTest: GET_unhandled_URI_named: substring '400 Bad Request' not in '[Failure instance: Traceback (failure with no frames): : 500 Internal Server Error ]' =============================================================================== [ERROR]: allmydata.test.test_web.Web.test_GET_unhandled_URI Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/common.py", line 220, in renderHTTP return m(ctx) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/filenode.py", line 181, in render_GET if self.node.is_mutable(): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/common.py", line 183, in is_mutable return self.my_uri.is_mutable() exceptions.AttributeError: CHKFileVerifierURI instance has no attribute 'is_mutable' =============================================================================== [ERROR]: allmydata.test.test_web.Web.test_GET_unhandled_URI_named Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/common.py", line 220, in renderHTTP return m(ctx) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/filenode.py", line 181, in render_GET if self.node.is_mutable(): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/common.py", line 183, in is_mutable return self.my_uri.is_mutable() exceptions.AttributeError: CHKFileVerifierURI instance has no attribute 'is_mutable' ------------------------------------------------------------------------------- Ran 4 tests in 0.586s FAILED (failures=4, errors=2) }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 23:08:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 06:08:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.ddd46174d6a278a2e034761bf7a49644@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): The test passes because it's testing the filesystem layer, while {{{tahoe_mv.py}}} uses the webapi. Basically, we want {{{tahoe mv}}} to do the following: * If we're doing {{{tahoe mv file1 file2}}}, {{{file1}}} should replace {{{file2}}}. * If we're doing {{{tahoe mv file1 folder1/}}}, {{{file1}}} should be inserted into {{{folder1}}}. * If we're doing {{{tahoe mv file1 folder1}}}, {{{tahoe mv}}} should print an error. To the user, this looks more or less the same regardless of how we do it on the backend -- {{{tahoe mv}}} should output "OK" in the first two cases, and an error message in the second. I'm attaching some tests that make sure that happens. I guess there are a few ways to implement this sort of functionality. * We could rely on the CLI (i.e., {{{tahoe mv}}}) to be smart enough to distinguish between files and folders, and structure HTTP requests appropriately. By using the methods described in {{{=== Get Information About A File Or Directory (as JSON) ===}}}, we can easily (and rather laboriously) determine whether the target is a file or a directory, and then act accordingly. * Alternatively, we could maybe bake some sort of functionality into the webapi that covers this. I'm not sure of the best way to do this, though. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 02:09:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 09:09:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.338ef5136a06df0ae201cc50d41d20cd@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): could you summarize the benchmark numbers in terms of A+Bx factors? Also, it's not clear to me what those numbers point to as being the expensive part: is it the netstring parsing? or the encryption? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 07:38:33 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 14:38:33 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.68c41b34520b39dc0adc967c3ee833b9@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Summarizing into {{{A+Bx}}} is not the way I do it. Instead I look at a handful of data points, like this: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.09, worst: 0.10 (of 5), reps/s: 11, ave rate: 732 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.36, 2th- worst: 0.35, worst: 0.39 (of 5), reps/s: 2, ave rate: 714 N: 1024, time: best: 1.51, 2th-best: 1.51, ave: 1.55, 2th- worst: 1.51, worst: 1.72 (of 5), reps/s: 0, ave rate: 659 N: 4096, time: best: 10.48, 2th-best: 10.49, ave: 10.66, 2th- worst: 10.53, worst: 11.29 (of 5), reps/s: 0, ave rate: 384 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 747 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.35, 2th- worst: 0.35, worst: 0.35 (of 5), reps/s: 2, ave rate: 728 N: 1024, time: best: 1.52, 2th-best: 1.53, ave: 1.53, 2th- worst: 1.53, worst: 1.53 (of 5), reps/s: 0, ave rate: 670 N: 4096, time: best: 10.48, 2th-best: 10.49, ave: 10.50, 2th- worst: 10.52, worst: 10.53 (of 5), reps/s: 0, ave rate: 390 benchmarking N: 64, time: best: 0.08, 2th-best: 0.09, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 744 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.35, 2th- worst: 0.35, worst: 0.35 (of 5), reps/s: 2, ave rate: 727 N: 1024, time: best: 1.52, 2th-best: 1.52, ave: 1.54, 2th- worst: 1.53, worst: 1.60 (of 5), reps/s: 0, ave rate: 665 N: 4096, time: best: 10.49, 2th-best: 10.50, ave: 10.51, 2th- worst: 10.51, worst: 10.52 (of 5), reps/s: 0, ave rate: 390 }}} This is a good example of why I do it this way: because it isn't linear! So any {{{A+Bx}}} summary would be off. I already have a patch which fixes the observable non-linearity there. I guess I might as well commit that one so that everyone can see it. There -- [3969], entitled "directories: keep track of your position as you decode netstring after netstring from an input buffer instead of copying the trailing part | This makes decoding linear in the number of netstrings instead of O(N^2^).". After this patch, the same benchmark on the same machine says: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.08, worst: 0.10 (of 5), reps/s: 11, ave rate: 750 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.34, 2th- worst: 0.33, worst: 0.37 (of 5), reps/s: 2, ave rate: 755 N: 1024, time: best: 1.28, 2th-best: 1.28, ave: 1.32, 2th- worst: 1.28, worst: 1.48 (of 5), reps/s: 0, ave rate: 776 N: 4096, time: best: 4.83, 2th-best: 4.83, ave: 4.97, 2th- worst: 4.83, worst: 5.51 (of 5), reps/s: 0, ave rate: 824 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 760 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.33, 2th- worst: 0.33, worst: 0.33 (of 5), reps/s: 2, ave rate: 767 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.29, 2th- worst: 1.29, worst: 1.29 (of 5), reps/s: 0, ave rate: 794 N: 4096, time: best: 4.83, 2th-best: 4.83, ave: 4.83, 2th- worst: 4.83, worst: 4.84 (of 5), reps/s: 0, ave rate: 847 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 759 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.33, 2th- worst: 0.33, worst: 0.33 (of 5), reps/s: 2, ave rate: 767 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.30, 2th- worst: 1.29, worst: 1.36 (of 5), reps/s: 0, ave rate: 786 N: 4096, time: best: 4.82, 2th-best: 4.83, ave: 4.85, 2th- worst: 4.87, worst: 4.90 (of 5), reps/s: 0, ave rate: 844 }}} As to your question about what's the expensive part, after the optimization patch from Kevan plus a few that I have here in my sandbox, benchmarking shows: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 849 N: 256, time: best: 0.29, 2th-best: 0.30, ave: 0.31, 2th- worst: 0.32, worst: 0.32 (of 5), reps/s: 3, ave rate: 832 N: 1024, time: best: 1.24, 2th-best: 1.24, ave: 1.30, 2th- worst: 1.27, worst: 1.52 (of 5), reps/s: 0, ave rate: 787 N: 4096, time: best: 4.28, 2th-best: 4.29, ave: 4.40, 2th- worst: 4.31, worst: 4.82 (of 5), reps/s: 0, ave rate: 931 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.08, worst: 0.08 (of 5), reps/s: 13, ave rate: 843 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.31, worst: 0.32 (of 5), reps/s: 3, ave rate: 840 N: 1024, time: best: 1.14, 2th-best: 1.14, ave: 1.16, 2th- worst: 1.17, worst: 1.20 (of 5), reps/s: 0, ave rate: 883 N: 4096, time: best: 4.30, 2th-best: 4.31, ave: 4.53, 2th- worst: 4.64, worst: 4.99 (of 5), reps/s: 0, ave rate: 904 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 860 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 880 N: 1024, time: best: 1.14, 2th-best: 1.14, ave: 1.15, 2th- worst: 1.14, worst: 1.21 (of 5), reps/s: 0, ave rate: 889 N: 4096, time: best: 4.28, 2th-best: 4.28, ave: 4.29, 2th- worst: 4.29, worst: 4.30 (of 5), reps/s: 0, ave rate: 955 }}} and profiling shows: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.08, worst: 0.10 (of 5), reps/s: 11, ave rate: 738 N: 256, time: best: 0.34, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.38 (of 5), reps/s: 2, ave rate: 744 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.48, 2th- worst: 1.46, worst: 2.04 (of 5), reps/s: 0, ave rate: 693 N: 4096, time: best: 4.84, 2th-best: 4.85, ave: 4.97, 2th- worst: 4.87, worst: 5.45 (of 5), reps/s: 0, ave rate: 824 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.10, 2th- worst: 0.09, worst: 0.18 (of 5), reps/s: 9, ave rate: 611 N: 256, time: best: 0.33, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.34 (of 5), reps/s: 2, ave rate: 764 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.30, 2th- worst: 1.30, worst: 1.30 (of 5), reps/s: 0, ave rate: 790 N: 4096, time: best: 4.85, 2th-best: 4.85, ave: 4.88, 2th- worst: 4.87, worst: 4.96 (of 5), reps/s: 0, ave rate: 840 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 754 N: 256, time: best: 0.33, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.34 (of 5), reps/s: 2, ave rate: 764 N: 1024, time: best: 1.30, 2th-best: 1.30, ave: 1.31, 2th- worst: 1.31, worst: 1.37 (of 5), reps/s: 0, ave rate: 779 N: 4096, time: best: 4.85, 2th-best: 4.87, ave: 4.89, 2th- worst: 4.91, worst: 4.96 (of 5), reps/s: 0, ave rate: 837 674783 function calls in 4.909 CPU seconds Ordered by: internal time List reduced from 75 to 32 due to restriction <32> ncalls tottime percall cumtime percall filename:lineno(function) 36830 2.422 0.000 2.591 0.000 base32.py:60(b2a_l) 7366 0.527 0.000 1.606 0.000 base32.py:208(a2b_l) 54026 0.154 0.000 0.154 0.000 netstring.py:3(netstring) 20879 0.149 0.000 0.149 0.000 hashutil.py:26(digest) 7366 0.111 0.000 0.111 0.000 hashutil.py:163(_xor) 36830 0.093 0.000 0.093 0.000 string.py:306(join) 3683 0.088 0.000 0.088 0.000 encoder.py:205(iterencode) 58928 0.080 0.000 0.080 0.000 string.py:480(translate) 1 0.076 0.076 1.905 1.905 dirnode.py:248(_pack_contents) 7366 0.066 0.000 0.066 0.000 netstring.py:7(split_netstring) 29464 0.060 0.000 2.141 0.000 base32.py:48(b2a) 49124 0.058 0.000 0.058 0.000 hashutil.py:23(update) 3683 0.056 0.000 0.386 0.000 dirnode.py:196(_encrypt_rwcap) 7366 0.055 0.000 0.199 0.000 hashutil.py:48(tagged_pair_hash) 13513 0.054 0.000 0.134 0.000 hashutil.py:38(tagged_hasher) 3683 0.050 0.000 0.050 0.000 decoder.py:341(raw_decode) 1 0.046 0.046 2.999 2.999 dirnode.py:218(_unpack_contents) 3683 0.042 0.000 0.155 0.000 dirnode.py:207(_decrypt_rwcapdata) 7366 0.040 0.000 1.697 0.000 base32.py:199(a2b) 125222 0.040 0.000 0.040 0.000 assertutil.py:30(precondition) 3683 0.040 0.000 0.151 0.000 hashutil.py:166(hmac) 13513 0.037 0.000 0.286 0.000 hashutil.py:43(tagged_hash) 20879 0.035 0.000 0.035 0.000 hashutil.py:19(__init__) }}} Whoops, time for me to go to work! I hope to measure and commit the rest of my optimization patches today after work. There is one that I really want other people's input on before I commit it... That one generates the per-entry IV with a secure hash of the writecap instead of with {{{os.urandom(16)}}}. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 08:01:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 15:01:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries Message-ID: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> #750: deterministic IV for writecaps for dir entries -----------------------------+---------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Keywords: confidentiality | Launchpad_bug: -----------------------------+---------------------------------------------- Okay, here is a patch which replaces {{{os.urandom(16)}}} with a (tagged) secure hash of the write cap. This protects the directory entries against the "VM rollback problem", which otherwise might expose the writecaps of entries to someone who has only the readcap to the dir in the case that the writer of the dir suffered a vm rollback. In addition, this makes packing deterministic, so that pack(unpack(packeddir)) == packeddir. (Kevan's original unit tests for this ticket assumed that would be the case.) Is there any cryptographic problem that making this change would raise? It means that encryption of the writecaps doesn't have "semantic security" (which I think corresponds to IND-CPA?), but they actually didn't anyway since each one is accompanied by its readcap. Perhaps surprisingly, this patch appears to reduce the CPU usage a little bit for packing directories (at least on Mac OS X). Please examine this patch, all cryptographers and security experts! Before this patch: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.09 (of 5), reps/s: 13, ave rate: 837 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.30, worst: 0.32 (of 5), reps/s: 3, ave rate: 851 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.18, 2th- worst: 1.15, worst: 1.31 (of 5), reps/s: 0, ave rate: 866 N: 4096, time: best: 4.37, 2th-best: 4.37, ave: 4.49, 2th- worst: 4.41, worst: 4.93 (of 5), reps/s: 0, ave rate: 912 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 862 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 872 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.15, 2th- worst: 1.15, worst: 1.15 (of 5), reps/s: 0, ave rate: 889 N: 4096, time: best: 4.36, 2th-best: 4.38, ave: 4.39, 2th- worst: 4.38, worst: 4.45 (of 5), reps/s: 0, ave rate: 933 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 857 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.30 (of 5), reps/s: 3, ave rate: 870 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.15, 2th- worst: 1.15, worst: 1.15 (of 5), reps/s: 0, ave rate: 888 N: 4096, time: best: 4.37, 2th-best: 4.38, ave: 4.38, 2th- worst: 4.38, worst: 4.39 (of 5), reps/s: 0, ave rate: 935 }}} after this patch: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 844 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.29, worst: 0.32 (of 5), reps/s: 3, ave rate: 858 N: 1024, time: best: 1.11, 2th-best: 1.11, ave: 1.15, 2th- worst: 1.11, worst: 1.27 (of 5), reps/s: 0, ave rate: 894 N: 4096, time: best: 4.25, 2th-best: 4.26, ave: 4.37, 2th- worst: 4.27, worst: 4.79 (of 5), reps/s: 0, ave rate: 938 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 859 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 874 N: 1024, time: best: 1.11, 2th-best: 1.12, ave: 1.12, 2th- worst: 1.12, worst: 1.12 (of 5), reps/s: 0, ave rate: 917 N: 4096, time: best: 4.26, 2th-best: 4.26, ave: 4.27, 2th- worst: 4.28, worst: 4.28 (of 5), reps/s: 0, ave rate: 959 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 856 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 871 N: 1024, time: best: 1.11, 2th-best: 1.11, ave: 1.13, 2th- worst: 1.12, worst: 1.18 (of 5), reps/s: 0, ave rate: 908 N: 4096, time: best: 4.25, 2th-best: 4.25, ave: 4.25, 2th- worst: 4.26, worst: 4.26 (of 5), reps/s: 0, ave rate: 963 }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From tahoe-devPOST at quake.ca Tue Jul 7 13:00:32 2009 From: tahoe-devPOST at quake.ca (Marc Tooley) Date: Tue, 7 Jul 2009 13:00:32 -0700 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <51864C9C-C63D-4BA1-A6ED-21C203113440@zooko.com> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> <51864C9C-C63D-4BA1-A6ED-21C203113440@zooko.com> Message-ID: <200907071300.32918.tahoe-devPOST@quake.ca> On Monday 06 July 2009, Zooko Wilcox-O'Hearn wrote: > > Do tell! :-) > > Regards, > > Zooko Sure. On what appears to be trunk tahoe, running: python2.5 setup.py test Returns the attached test results on one of my problematic systems. Hey out of curiosity, is that bad hash tests part of the data block I sent in? That would be pretty cool.. :) -------------- next part -------------- running darcsver darcs failed: Time.toClockTime: invalid input running develop Not found: tahoe-deps Not found: ../tahoe-deps running egg_info writing requirements to src/allmydata_tahoe.egg-info/requires.txt writing src/allmydata_tahoe.egg-info/PKG-INFO writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' running build_ext Processing setuptools-0.6c12dev.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg' (and everything under it) creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg Extracting setuptools-0.6c12dev.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages setuptools 0.6c12dev is already the active version in easy-install.pth Installing easy_install script to support/bin Installing easy_install-2.5 script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg Creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/allmydata-tahoe.egg-link (link to src) allmydata-tahoe 0.0.0 is already the active version in easy-install.pth Installing tahoe script to support/bin Installed /v/tahoe/tahoe/src Processing dependencies for allmydata-tahoe==0.0.0 Searching for pycryptopp==0.5.15 Best match: pycryptopp 0.5.15 Adding pycryptopp 0.5.15 to easy-install.pth file Using /v/tahoe/pycryptopp Searching for Nevow==0.9.33-r17222 Best match: Nevow 0.9.33-r17222 Processing Nevow-0.9.33_r17222-py2.5.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/Nevow-0.9.33_r17222-py2.5.egg' (and everything under it) Copying Nevow-0.9.33_r17222-py2.5.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages Nevow 0.9.33-r17222 is already the active version in easy-install.pth Installing nevow-xmlgettext script to support/bin Installing nit script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/Nevow-0.9.33_r17222-py2.5.egg Searching for foolscap==0.4.2 Best match: foolscap 0.4.2 Processing foolscap-0.4.2-py2.5.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/foolscap-0.4.2-py2.5.egg' (and everything under it) creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/foolscap-0.4.2-py2.5.egg Extracting foolscap-0.4.2-py2.5.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages foolscap 0.4.2 is already the active version in easy-install.pth Installing flogtool script to support/bin Installing flappserver script to support/bin Installing flappclient script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/foolscap-0.4.2-py2.5.egg Searching for Twisted==8.2.0 Best match: Twisted 8.2.0 Processing Twisted-8.2.0-py2.5-netbsd-4.99.7-i386.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-netbsd-4.99.7-i386.egg' (and everything under it) Copying Twisted-8.2.0-py2.5-netbsd-4.99.7-i386.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages Twisted 8.2.0 is already the active version in easy-install.pth Installing manhole script to support/bin Installing trial script to support/bin Installing mktap script to support/bin Installing pyhtmlizer script to support/bin Installing twistd script to support/bin Installing tap2rpm script to support/bin Installing tap2deb script to support/bin Installing tapconvert script to support/bin Installing im script to support/bin Installing tkconch script to support/bin Installing conch script to support/bin Installing cftp script to support/bin Installing ckeygen script to support/bin Installing lore script to support/bin Installing mailmail script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-netbsd-4.99.7-i386.egg Searching for zope.interface==3.5.1 Best match: zope.interface 3.5.1 Processing zope.interface-3.5.1-py2.5-netbsd-4.99.7-i386.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/zope.interface-3.5.1-py2.5-netbsd-4.99.7-i386.egg' (and everything under it) Copying zope.interface-3.5.1-py2.5-netbsd-4.99.7-i386.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages zope.interface 3.5.1 is already the active version in easy-install.pth Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/zope.interface-3.5.1-py2.5-netbsd-4.99.7-i386.egg Searching for simplejson==2.0.9 Best match: simplejson 2.0.9 Processing simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg' (and everything under it) creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg Extracting simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages simplejson 2.0.9 is already the active version in easy-install.pth Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg Searching for zfec==1.4.5 Best match: zfec 1.4.5 Processing zfec-1.4.5-py2.5-netbsd-4.99.7-i386.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/zfec-1.4.5-py2.5-netbsd-4.99.7-i386.egg' (and everything under it) Copying zfec-1.4.5-py2.5-netbsd-4.99.7-i386.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages zfec 1.4.5 is already the active version in easy-install.pth Installing zfec script to support/bin Installing zunfec script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/zfec-1.4.5-py2.5-netbsd-4.99.7-i386.egg Searching for pyOpenSSL==0.9 Best match: pyOpenSSL 0.9 Processing pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg' (and everything under it) creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg Extracting pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages pyOpenSSL 0.9 is already the active version in easy-install.pth Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg Searching for pyutil==1.3.30 Best match: pyutil 1.3.30 Processing pyutil-1.3.30-py2.5.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/pyutil-1.3.30-py2.5.egg' (and everything under it) Copying pyutil-1.3.30-py2.5.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages pyutil 1.3.30 is already the active version in easy-install.pth Installing randfile script to support/bin Installing adler32sum script to support/bin Installing sha256dsum script to support/bin Installing lines script to support/bin Installing find_trial script to support/bin Installing sha1sum_py script to support/bin Installing sha256sum_py script to support/bin Installing randcookie script to support/bin Installing sha256dcppsum script to support/bin Installing tailx script to support/bin Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/pyutil-1.3.30-py2.5.egg Searching for argparse==0.9.1 Best match: argparse 0.9.1 Processing argparse-0.9.1-py2.5.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/argparse-0.9.1-py2.5.egg' (and everything under it) creating /v/tahoe/tahoe/support/lib/python2.5/site-packages/argparse-0.9.1-py2.5.egg Extracting argparse-0.9.1-py2.5.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages argparse 0.9.1 is already the active version in easy-install.pth Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/argparse-0.9.1-py2.5.egg Searching for zbase32==1.1.1 Best match: zbase32 1.1.1 Processing zbase32-1.1.1-py2.5.egg removing '/v/tahoe/tahoe/support/lib/python2.5/site-packages/zbase32-1.1.1-py2.5.egg' (and everything under it) Copying zbase32-1.1.1-py2.5.egg to /v/tahoe/tahoe/support/lib/python2.5/site-packages zbase32 1.1.1 is already the active version in easy-install.pth Installed /v/tahoe/tahoe/support/lib/python2.5/site-packages/zbase32-1.1.1-py2.5.egg Finished processing dependencies for allmydata-tahoe==0.0.0 running make_executable running build running build_py copying src/allmydata/_auto_deps.py -> build/lib/allmydata running trial running build_ext allmydata.test.test_backupdb BackupDB test_basic ... [ERROR] test_check ... [ERROR] test_fail ... [ERROR] test_wrong_version ... [ERROR] allmydata.test.test_base62 T test_ende_0x00 ... [OK] test_ende_0x000000 ... [OK] test_ende_0x01 ... [OK] test_ende_0x0100 ... [OK] test_ende_0x010000 ... [OK] test_ende_longrandstr ... [OK] test_ende_randstr ... [OK] test_num_octets_that_encode_to_this_many_chars ... [OK] test_odd_sizes ... [OK] allmydata.test.test_checker WebResultsRendering test_check ... [OK] test_check_and_repair ... [OK] test_literal ... [OK] allmydata.test.test_cli Backup test_backup ... [ERROR] test_exclude_options ... [OK] CLI test_alias ... [OK] test_catalog_shares_error ... [OK] test_dump_cap_chk ... [OK] test_dump_cap_directory ... [OK] test_dump_cap_lit ... [OK] test_dump_cap_ssk ... [OK] test_options ... [OK] Check test_check ... [OK] test_deep_check ... [OK] Cp test_dangling_symlink_vs_recursion ... [OK] test_not_enough_args ... [OK] test_unicode_filename ... [SUCCESS!?!] CreateAlias test_create ... [OK] Errors test_check ... [OK] Help test_add_aliases ... [OK] test_backup ... [OK] test_check ... [OK] test_create_alias ... [OK] test_deep_check ... [OK] test_get ... [OK] test_ln ... [OK] test_manifest ... [OK] test_mv ... [OK] test_put ... [OK] test_rm ... [OK] test_stats ... [OK] test_webopen ... [OK] List test_list ... [OK] Put test_immutable_from_file ... [OK] test_mutable ... [OK] test_mutable_unlinked ... [OK] test_unlinked_immutable_from_file ... [OK] test_unlinked_immutable_stdin ... [OK] allmydata.test.test_client Basic test_loadable ... [OK] test_loadable_old_config_bits ... [OK] test_loadable_old_storage_config_bits ... [OK] test_loadable_without_vdrive ... [OK] test_permute ... [OK] test_reserved_1 ... [OK] test_reserved_2 ... [OK] test_reserved_3 ... [OK] test_reserved_4 ... [OK] test_reserved_bad ... [OK] test_secrets ... [OK] test_versions ... [FAIL] NodeMaker test_maker ... [OK] Run test_loadable ... [OK] test_reloadable ... [OK] allmydata.test.test_codec T test_encode ... [OK] test_encode1 ... [OK] test_encode2 ... [OK] allmydata.test.test_consolidate Consolidate test_arg_parsing ... [OK] allmydata.test.test_crawler Basic test_empty_subclass ... [OK] test_immediate ... [OK] test_oneshot ... [OK] test_paced ... [OK] test_paced_service ... [OK] test_service ... [OK] allmydata.test.test_deepcheck DeepCheckWebBad test_bad ... [OK] DeepCheckWebGood test_good ... [OK] Large test_lots_of_lits ... [OK] MutableChecker test_corrupt ... [OK] test_delete_share ... [OK] test_good ... [OK] allmydata.test.test_dirnode DeepStats test_stats ... [OK] Deleter test_retry ... [OK] Dirnode test_basic ... [OK] test_check ... [OK] test_create ... [OK] test_deepcheck ... [OK] test_deepcheck_and_repair ... [OK] test_deepcheck_problems ... [OK] test_readonly ... [OK] Dirnode2 test_from_future ... [OK] allmydata.test.test_download DownloadTest test_download ... [OK] allmydata.test.test_encode Encode test_send_100 ... [OK] test_send_101 ... [OK] test_send_124 ... [OK] test_send_125 ... [OK] test_send_51 ... [OK] test_send_74 ... [OK] test_send_75 ... [OK] test_send_76 ... [OK] test_send_99 ... [OK] Roundtrip test_100 ... [OK] test_101 ... [OK] test_124 ... [OK] test_125 ... [OK] test_51 ... [OK] test_74 ... [OK] test_75 ... [OK] test_76 ... [OK] test_99 ... [OK] test_bad_blockhashes ... [OK] test_bad_blockhashes_failure ... [OK] test_bad_blocks ... [OK] test_bad_blocks_failure ... [OK] test_bad_crypttext_hashes ... [OK] test_bad_crypttext_hashes_failure ... [OK] test_bad_crypttext_hashroot ... [OK] test_bad_sharehashes ... [OK] test_bad_sharehashes_failure ... [OK] test_bad_uri_extension ... [OK] test_good ... [OK] test_lost_all_shareholders ... [OK] test_lost_many_shareholders ... [OK] test_lost_one_shareholder ... [OK] test_lost_one_shareholder_early ... [OK] test_missing_sharehashes ... [OK] test_missing_sharehashes_failure ... [OK] test_not_enough_shares ... [OK] test_one_share_per_peer ... [OK] test_pause ... [OK] test_pause_then_stop ... [OK] test_stop ... [OK] ValidatedExtendedURIProxy test_accept_minimal ... [OK] test_accept_optional ... [OK] test_reject_insufficient ... [OK] test_reject_optional ... [OK] allmydata.test.test_filenode LiteralChecker test_literal_filenode ... [OK] Node test_chk_filenode ... [OK] test_literal_filenode ... [OK] test_mutable_filenode ... [OK] allmydata.test.test_hashtree Complete test_create ... [OK] test_dump ... [OK] test_needed_hashes ... [OK] Incomplete test_check ... [OK] test_create ... [OK] test_depth_of ... [OK] test_large ... [OK] test_needed_hashes ... [OK] allmydata.test.test_helper AssistedUpload test_already_uploaded ... [OK] test_one ... [OK] test_previous_upload_failed ... [OK] allmydata.test.test_immutable Test test_download ... [OK] test_download_abort_if_too_many_corrupted_shares ... [OK] test_download_abort_if_too_many_missing_shares ... [OK] test_download_from_only_3_remaining_shares ... [OK] test_download_from_only_3_shares_with_good_crypttext_hash ... [OK] test_test_code ... [OK] allmydata.test.test_introducer DecodeFurl test_decode ... [OK] Introducer test_create ... [OK] test_duplicate ... [OK] test_listen ... [OK] Node test_loadable ... [OK] NonV1Server test_failure ... [OK] SystemTest test_system ... [OK] allmydata.test.test_iputil ListAddresses test_get_local_ip_for ... [OK] test_list_async ... [FAIL] allmydata.test.test_keygen KeyGenService test_key_gen_service ... [OK] allmydata.test.test_mutable Checker test_check_all_bad_blocks ... [OK] test_check_all_bad_sig ... [OK] test_check_good ... [OK] test_check_no_shares ... [OK] test_check_not_enough_shares ... [OK] test_verify_all_bad_sig ... [OK] test_verify_good ... [OK] test_verify_one_bad_block ... [OK] test_verify_one_bad_encprivkey ... [OK] test_verify_one_bad_encprivkey_uncheckable ... [OK] test_verify_one_bad_sharehash ... [OK] test_verify_one_bad_sig ... [OK] Exceptions test_repr ... [OK] Filenode test_create ... [OK] test_create_with_initial_contents ... [OK] test_create_with_too_large_contents ... [OK] test_modify ... [OK] test_modify_backoffer ... [OK] test_serialize ... [OK] test_upload_and_download ... [OK] test_upload_and_download_full_size_keys ... [OK] MakeShares test_encrypt ... [OK] test_generate ... [OK] MultipleEncodings test_multiple_encodings ... [OK] MultipleVersions test_multiple_versions ... [OK] test_replace ... [OK] Problems test_bad_server ... [OK] test_bad_server_overlap ... [OK] test_privkey_query_error ... [OK] test_privkey_query_missing ... [OK] test_publish_all_servers_bad ... [OK] test_publish_no_servers ... [OK] test_publish_surprise ... [OK] test_retrieve_surprise ... [OK] test_unexpected_shares ... [OK] Repair test_check_and_repair_readcap ... [OK] test_merge ... [OK] test_non_merge ... [OK] test_repair_nop ... [OK] Roundtrip test_all_shares_vanished ... [OK] test_basic ... [OK] test_basic_pubkey_at_end ... [OK] test_corrupt_all_IV ... [OK] test_corrupt_all_N ... [OK] test_corrupt_all_R ... [OK] test_corrupt_all_block ... [OK] test_corrupt_all_block_hash_tree ... [OK] test_corrupt_all_block_hash_tree_late ... [OK] test_corrupt_all_block_late ... [OK] test_corrupt_all_datalen ... [OK] test_corrupt_all_encprivkey ... [OK] test_corrupt_all_k ... [OK] test_corrupt_all_pubkey ... [OK] test_corrupt_all_segsize ... [OK] test_corrupt_all_seqnum ... [OK] test_corrupt_all_seqnum_late ... [OK] test_corrupt_all_share_hash_chain_hash ... [OK] test_corrupt_all_share_hash_chain_number ... [OK] test_corrupt_all_sig ... [OK] test_corrupt_all_verbyte ... [OK] test_corrupt_some ... [OK] test_download_fails ... [OK] test_no_servers ... [OK] test_no_servers_download ... [OK] Servermap test_basic ... [OK] test_fetch_privkey ... [OK] test_mark_bad ... [OK] test_no_shares ... [OK] test_not_quite_enough_shares ... [OK] Utils test_cache ... [OK] test_cache_inside ... [OK] test_cache_overlap ... [OK] allmydata.test.test_netstring Netstring test_extra ... [OK] test_nested ... [OK] test_split ... [OK] allmydata.test.test_nevow Web test_read_default_css ... [SUCCESS!?!] allmydata.test.test_no_network Harness test_create ... [OK] test_upload ... [OK] allmydata.test.test_node TestCase test_location ... [OK] test_location2 ... [OK] test_secrets_dir ... [OK] test_secrets_dir_protected ... [OK] test_timestamp ... [OK] allmydata.test.test_observer Observer test_lazy_oneshot ... [OK] test_observerlist ... [OK] test_oneshot ... [OK] test_oneshot_fireagain ... [OK] allmydata.test.test_provisioning Provisioning test_load ... [OK] test_provisioning_math ... [OK] Reliability test_basic ... [SKIPPED] allmydata.test.test_repairer DownUpConnector test_deferred_satisfaction ... [OK] test_extra ... [OK] test_short_reads_1 ... [OK] test_short_reads_2 ... [OK] test_short_reads_3 ... [OK] Repairer test_harness ... [OK] test_repair_from_deletion_of_1 ... [OK] test_repair_from_deletion_of_7 ... [OK] Verifier test_check_without_verify ... [OK] test_verify_no_problem_1 ... [OK] test_verify_no_problem_2 ... [OK] test_verify_no_problem_3 ... [OK] test_verify_no_problem_4 ... [OK] test_verify_server_invisible_corruption_1 ... [OK] test_verify_server_invisible_corruption_2 ... [OK] test_verify_server_invisible_corruption_3 ... [OK] test_verify_server_invisible_corruption_4 ... [OK] test_verify_server_invisible_corruption_5 ... [OK] test_verify_server_invisible_corruption_6 ... [OK] test_verify_server_invisible_corruption_block_hash_tree_TODO ... [SUCCESS!?!] test_verify_server_invisible_corruption_cryptext_hash_tree_TODO ... [TODO] test_verify_server_invisible_corruption_offset_of_block_hashtree_TODO ... [SUCCESS!?!] test_verify_server_invisible_corruption_offset_of_block_hashtree_to_truncate_crypttext_hashtree_TODO ... [SUCCESS!?!] test_verify_server_invisible_corruption_offset_of_ciphertext_hashtree_TODO ... [TODO] test_verify_server_invisible_corruption_offset_of_share_hashtree_TODO ... [SUCCESS!?!] test_verify_server_invisible_corruption_share_hash_tree_TODO ... [SUCCESS!?!] test_verify_server_invisible_corruption_sharedata_plausible_version ... [OK] test_verify_server_visible_corruption ... [OK] test_verify_share_version_incompatibility ... [OK] allmydata.test.test_runner CreateNode test_client ... [OK] test_introducer ... [OK] test_key_generator ... [OK] test_stats_gatherer ... [OK] test_subcommands ... [OK] RunNode test_baddir ... [OK] test_client ... [OK] test_introducer ... [OK] test_keygen ... [OK] TheRightCode test_path ... [FAIL] allmydata.test.test_stats CPUUsage test_monitor ... [OK] allmydata.test.test_storage Bucket test_create ... [OK] test_readwrite ... [OK] BucketCounter test_bucket_counter ... [OK] test_bucket_counter_cleanup ... [OK] test_bucket_counter_eta ... [OK] BucketProxy test_create ... [OK] test_readwrite_v1 ... [OK] test_readwrite_v2 ... [OK] LeaseCrawler test_bad_mode ... [OK] test_basic ... [OK] test_expire_age ... [OK] test_expire_cutoff_date ... [OK] test_limited_history ... [OK] test_no_st_blocks ... [OK] test_only_immutable ... [OK] test_only_mutable ... [OK] test_parse_date ... [OK] test_parse_duration ... [OK] test_share_corruption ... [OK] test_unpredictable_future ... [OK] MutableServer test_allocate ... [OK] test_bad_magic ... [OK] test_container_size ... [OK] test_create ... [OK] test_leases ... [OK] test_operators ... [OK] test_readv ... [OK] test_remove ... [OK] Server test_advise_corruption ... [OK] test_allocate ... [OK] test_bad_container_version ... [OK] test_create ... [OK] test_discard ... [OK] test_disconnect ... [OK] test_dont_overfill_dirs ... [OK] test_large_share ... [SKIPPED] test_leases ... [OK] test_readonly ... [OK] test_remove_incoming ... [OK] test_reserved_space ... [OK] test_seek ... [OK] Stats test_latencies ... [OK] WebStatus test_huge_reserved ... [OK] test_no_server ... [OK] test_readonly ... [OK] test_reserved ... [OK] test_status ... [OK] test_status_no_statvfs ... [OK] test_util ... [OK] allmydata.test.test_system SystemTest test_mutable ... [OK] test_upload_and_download_convergent ... [OK] test_upload_and_download_random_key ... [OK] test_vdrive ... GET http://127.0.0.1:50052/ output was: Introducer Status - AllMyData Tahoe

Welcome To The Tahoe Introducer

My Code

My nodeid: njeaxgfqknd2as37fxdkl4skyybywey6
My versions: allmydata-tahoe: 0.0.0, foolscap: 0.4.2, pycryptopp: 0.5.15, zfec: 1.4.5, Twisted: 8.2.0, Nevow: 0.9.33-r17222, zope.interface: 3.5.1, python: 2.5.2, platform: NetBSD-4.99.7-i386-32bit-ELF, sqlite: None, simplejson: 2.0.9, argparse: 0.9.1, pyOpenSSL: 0.9, pyutil: 1.3.30, zbase32: 1.1.1, setuptools: 0.6c12dev, pysqlite: None
Tahoe code imported from: <module 'allmydata' from '/ v/ tahoe/ tahoe/ src/ allmydata/ __init__.pyc'>

Announcement Summary: storage: 5, stub_client: 5
Subscription Summary: storage: 5

Service Announcements

PeerID / Nickname Advertised IPs Announced Version Service Name
ncieuno4yjvioe7fz5y7tqhl3nfrmawh <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
z3g3meylekyi7yjo4azff2xbqwjbuioa <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
iwlrbdlr67sspxhbp7xpeku6rxe5zasm <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
a2niqogjbeorxkcqxx3yy24q77rstxc7 <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
bduf2tsgfciu2j4zrs7oxvt52h5ikgtj <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage

Subscribed Clients

PeerID / Nickname Advertised IPs Connected From Since Version Subscribed To
iwlrbdlr67sspxhbp7xpeku6rxe5zasm <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 127.0.0.2:50032 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
a2niqogjbeorxkcqxx3yy24q77rstxc7 <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 127.0.0.2:50020 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
bduf2tsgfciu2j4zrs7oxvt52h5ikgtj <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 127.0.0.2:50014 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
ncieuno4yjvioe7fz5y7tqhl3nfrmawh <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 10.3.0.1:50041 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
z3g3meylekyi7yjo4azff2xbqwjbuioa <unspecified> 10.2.0.12 10.3.0.1 127.0.0.2 10.1.0.4 10.2.0.12:50024 12:28:54 07-Jul-2009 allmydata-tahoe/unknown storage
[FAIL] allmydata.test.test_upload EncodingParameters test_configure_parameters ... [OK] FullServer test_data_large ... [OK] GoodServer test_data_large ... [OK] test_data_large_odd_segments ... [OK] test_data_small ... [OK] test_data_zero ... [OK] test_filehandle_large ... [OK] test_filehandle_small ... [OK] test_filehandle_zero ... [OK] test_filename_large ... [OK] test_filename_small ... [OK] test_filename_zero ... [OK] test_too_large ... [OK] PeerSelection test_four_each ... [OK] test_one_each ... [OK] test_one_each_plus_one_extra ... [OK] test_some_big_some_small ... [OK] test_three_of_ten ... [OK] test_two_each ... [OK] StorageIndex test_params_must_matter ... [OK] Uploadable test_data ... [OK] test_filehandle_convergent_encryption ... [OK] test_filehandle_random_key ... [OK] test_filename ... [OK] allmydata.test.test_uri CHKFile test_pack ... [OK] test_pack_badly ... [OK] Compare test_compare ... [OK] test_is_uri ... [OK] Constraint test_constraint ... [OK] Extension test_pack ... [OK] Invalid test_from_future ... [OK] Literal test_empty ... [OK] test_nonascii ... [OK] test_pack ... [OK] Mutable test_pack ... [OK] NewDirnode test_pack ... [OK] allmydata.test.test_util Abbreviate test_parse_space ... [OK] test_space ... [OK] test_time ... [OK] Asserts test_assert ... [OK] test_postcondition ... [OK] test_precondition ... [OK] Base32 test_a2b ... [OK] test_b2a ... [OK] test_b2a_matches_Pythons ... [OK] test_b2a_or_none ... [OK] CacheDir test_basic ... [OK] DeferredUtilTests test_failure ... [OK] test_gather_results ... [OK] test_success ... [OK] DictUtil test_all ... [OK] test_del_if_present ... [OK] test_dict_of_sets ... [OK] test_move ... [OK] test_numdict ... [OK] test_subtract ... [OK] test_utildict ... [OK] test_valueordereddict ... [OK] FileUtil test_NamedTemporaryDirectory ... [OK] test_du ... [OK] test_open_or_create ... [OK] test_remove_if_possible ... [OK] test_rename ... [OK] test_rm_dir ... [OK] HashUtilTests test_chk ... [OK] test_constant_time_compare ... [OK] test_hashers ... [OK] test_known_answers ... [OK] test_random_key ... [OK] test_sha256d ... [OK] test_sha256d_truncated ... [OK] HumanReadable test_repr ... [OK] IDLib test_nodeid_b2a ... [OK] Limiter test_errors ... [OK] test_limiter ... [OK] Math test_ave ... [OK] test_div_ceil ... [OK] test_is_power_of_k ... [OK] test_next_multiple ... [OK] test_next_power_of_k ... [OK] test_pad_size ... [OK] test_round_sigfigs ... [OK] Pipeline test_basic ... [OK] test_errors ... [OK] test_errors2 ... [OK] PollMixinTests test_PollMixin_False_then_True ... [OK] test_PollMixin_True ... [OK] test_timeout ... [OK] Statistics test_binomial_coeff ... [OK] test_binomial_distribution_pmf ... [OK] test_convolve ... [OK] test_find_k ... [OK] test_pr_backup_file_loss ... [OK] test_pr_file_loss ... [OK] test_repair_cost ... [OK] test_repair_count_pmf ... [OK] test_survival_pmf ... [OK] TimeFormat test_epoch ... [OK] test_epoch_in_London ... [OK] allmydata.test.test_web Grid test_add_lease ... [OK] test_deep_add_lease ... [OK] test_deep_check ... [OK] test_deep_check_and_repair ... [OK] test_exceptions ... [OK] test_filecheck ... [OK] test_repair_html ... [OK] test_repair_json ... [OK] test_unknown ... [OK] Util test_abbreviate_rate ... [OK] test_abbreviate_size ... [OK] test_abbreviate_time ... [OK] test_plural ... [OK] Web test_DELETE_DIRURL ... [OK] test_DELETE_DIRURL_missing ... [OK] test_DELETE_DIRURL_missing2 ... [OK] test_DELETE_FILEURL ... [OK] test_DELETE_FILEURL_missing ... [OK] test_DELETE_FILEURL_missing2 ... [OK] test_GET_DIRURL ... [OK] test_GET_DIRURL_badtype ... [OK] test_GET_DIRURL_json ... [OK] test_GET_DIRURL_named_bad ... [OK] test_GET_DIRURL_readonly_uri ... [OK] test_GET_DIRURL_uri ... [OK] test_GET_FILEURL ... [OK] test_GET_FILEURL_badtype ... [OK] test_GET_FILEURL_json ... [OK] test_GET_FILEURL_json_missing ... [OK] test_GET_FILEURL_missing ... [OK] test_GET_FILEURL_named ... [OK] test_GET_FILEURL_partial_range ... [OK] test_GET_FILEURL_range ... [OK] test_GET_FILEURL_range_bad ... [OK] test_GET_FILEURL_save ... [OK] test_GET_FILEURL_uri ... [OK] test_GET_FILEURL_uri_missing ... [OK] test_GET_FILE_URI ... [OK] test_GET_FILE_URI_badchild ... [OK] test_GET_URI_URL ... [OK] test_GET_URI_URL_dir ... [OK] test_GET_URI_URL_missing ... [OK] test_GET_URI_form ... [OK] test_GET_URI_form_bad ... [OK] test_GET_rename_form ... [OK] test_GET_slash_file_bad ... [OK] test_GET_unhandled_URI ... [OK] test_GET_unhandled_URI_named ... [OK] test_HEAD_FILEURL ... [OK] test_HEAD_FILEURL_partial_range ... [OK] test_HEAD_FILEURL_range ... [OK] test_POST_DIRURL_check ... [OK] test_POST_DIRURL_check_and_repair ... [OK] test_POST_DIRURL_deepcheck ... [OK] test_POST_DIRURL_deepcheck_and_repair ... [OK] test_POST_DIRURL_deepcheck_no_ophandle ... [OK] test_POST_DIRURL_deepsize ... [OK] test_POST_DIRURL_deepsize_no_ophandle ... [OK] test_POST_DIRURL_deepstats ... [OK] test_POST_DIRURL_deepstats_no_ophandle ... [OK] test_POST_DIRURL_manifest ... [OK] test_POST_DIRURL_manifest_no_ophandle ... [OK] test_POST_DIRURL_stream_manifest ... [OK] test_POST_FILEURL_bad_t ... [OK] test_POST_FILEURL_check ... [OK] test_POST_FILEURL_check_and_repair ... [OK] test_POST_bad_t ... [OK] test_POST_check ... [OK] test_POST_delete ... [OK] test_POST_mkdir ... [OK] test_POST_mkdir_2 ... [OK] test_POST_mkdir_no_parentdir_noredirect ... [OK] test_POST_mkdir_no_parentdir_redirect ... [OK] test_POST_mkdir_no_replace_field ... [OK] test_POST_mkdir_no_replace_queryarg ... [OK] test_POST_mkdir_replace ... [OK] test_POST_mkdir_whendone_field ... [OK] test_POST_mkdir_whendone_queryarg ... [OK] test_POST_mkdirs_2 ... [OK] test_POST_noparent_bad ... [OK] test_POST_put_uri ... [OK] test_POST_put_uri_no_replace_field ... [OK] test_POST_put_uri_no_replace_queryarg ... [OK] test_POST_put_uri_replace ... [OK] test_POST_rename_dir ... [OK] test_POST_rename_file ... [OK] test_POST_rename_file_no_replace_field ... [OK] test_POST_rename_file_no_replace_queryarg ... [OK] test_POST_rename_file_redundant ... [OK] test_POST_rename_file_replace ... [OK] test_POST_rename_file_slash_fail ... [OK] test_POST_set_children ... [OK] test_POST_upload ... [OK] test_POST_upload_mutable ... [OK] test_POST_upload_mutable_toobig ... [OK] test_POST_upload_named ... [OK] test_POST_upload_named_badfilename ... [OK] test_POST_upload_no_link ... [OK] test_POST_upload_no_link_mutable ... [OK] test_POST_upload_no_link_mutable_toobig ... [OK] test_POST_upload_no_link_whendone ... [OK] test_POST_upload_no_link_whendone_results ... [OK] test_POST_upload_no_replace_field ... [OK] test_POST_upload_no_replace_ok ... [OK] test_POST_upload_no_replace_queryarg ... [OK] test_POST_upload_replace ... [OK] test_POST_upload_unicode ... [OK] test_POST_upload_unicode_named ... [OK] test_POST_upload_whendone ... [OK] test_PUT_DIRURL_bad_t ... [OK] test_PUT_DIRURL_uri ... [OK] test_PUT_DIRURL_uri_noreplace ... [OK] test_PUT_FILEURL_named_bad ... [OK] test_PUT_FILE_URI_badchild ... [OK] test_PUT_NEWDIRURL ... [OK] test_PUT_NEWDIRURL_blocked ... [OK] test_PUT_NEWDIRURL_exists ... [OK] test_PUT_NEWDIRURL_mkdir_p ... [OK] test_PUT_NEWDIRURL_mkdirs ... [OK] test_PUT_NEWFILEURL ... [OK] test_PUT_NEWFILEURL_bad_t ... [OK] test_PUT_NEWFILEURL_blocked ... [OK] test_PUT_NEWFILEURL_mkdirs ... [OK] test_PUT_NEWFILEURL_mutable ... [OK] test_PUT_NEWFILEURL_mutable_toobig ... [OK] test_PUT_NEWFILEURL_no_replace ... [OK] test_PUT_NEWFILEURL_not_mutable ... [OK] test_PUT_NEWFILEURL_range_bad ... [OK] test_PUT_NEWFILEURL_replace ... [OK] test_PUT_NEWFILEURL_uri ... [OK] test_PUT_NEWFILEURL_uri_no_replace ... [OK] test_PUT_NEWFILEURL_uri_replace ... [OK] test_PUT_NEWFILE_URI ... [OK] test_PUT_NEWFILE_URI_mutable ... [OK] test_PUT_NEWFILE_URI_not_mutable ... [OK] test_PUT_NEWFILE_URI_only_PUT ... [OK] test_PUT_mkdir ... [OK] test_bad_method ... [OK] test_create ... [OK] test_incident ... [OK] test_ophandle_bad ... [OK] test_ophandle_cancel ... [OK] test_ophandle_release_after_complete ... [OK] test_ophandle_retainfor ... [OK] test_provisioning ... [OK] test_reliability_tool ... [SKIPPED] test_short_url ... [OK] test_static ... [OK] test_status ... [OK] test_status_numbers ... [OK] test_welcome ... [OK] test_welcome_page_mkdir_button ... [OK] =============================================================================== [SKIPPED]: allmydata.test.test_provisioning.Reliability.test_basic reliability model requires NumPy =============================================================================== [SKIPPED]: allmydata.test.test_storage.Server.test_large_share This test can spuriously fail if you have less than 4 GiB free on your filesystem, and if your filesystem doesn't support efficient sparse files then it is very expensive (Mac OS X is the only system I know of in the desktop/server area that doesn't support efficient sparse files). =============================================================================== [SKIPPED]: allmydata.test.test_web.Web.test_reliability_tool reliability tool requires NumPy =============================================================================== [TODO]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_cryptext_hash_tree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_repairer.py", line 114, in _check judgement(vr) File "/v/tahoe/tahoe/src/allmydata/test/test_repairer.py", line 206, in judge_invisible_corruption self.failIf(vr.is_healthy(), (vr, vr.is_healthy(), vr.get_data())) twisted.trial.unittest.FailTest: (, True, {'list-incompatible-shares': [], 'count-shares-expected': 10, 'count-incompatible-shares': 0, 'count-unrecoverable-versions': 0, 'count-shares-needed': 3, 'sharemap': {0: set(['\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&']), 1: set(['\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80']), 2: set(['\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b']), 3: set(['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i']), 4: set(["\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc"]), 5: set(['+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc']), 6: set(["\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6"]), 7: set(['!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T']), 8: set(["\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4"]), 9: set(['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t'])}, 'count-recoverable-versions': 1, 'servers-responding': ['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i', '\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b', "\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4", '\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&', '+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc', "\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6", '!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T', "\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc", '\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t', '\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80'], 'count-good-share-hosts': 10, 'count-wrong-shares': 0, 'count-shares-good': 10, 'count-corrupt-shares': 0, 'list-corrupt-shares': []}) vr.data is: {'list-incompatible-shares': [], 'count-shares-expected': 10, 'count-incompatible-shares': 0, 'count-unrecoverable-versions': 0, 'count-shares-needed': 3, 'sharemap': {0: set(['\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&']), 1: set(['\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80']), 2: set(['\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b']), 3: set(['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i']), 4: set(["\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc"]), 5: set(['+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc']), 6: set(["\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6"]), 7: set(['!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T']), 8: set(["\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4"]), 9: set(['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t'])}, 'count-recoverable-versions': 1, 'servers-responding': ['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i', '\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b', "\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4", '\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&', '+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc', "\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6", '!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T', "\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc", '\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t', '\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80'], 'count-good-share-hosts': 10, 'count-wrong-shares': 0, 'count-shares-good': 10, 'count-corrupt-shares': 0, 'list-corrupt-shares': []} =============================================================================== [TODO]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_offset_of_ciphertext_hashtree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_repairer.py", line 114, in _check judgement(vr) File "/v/tahoe/tahoe/src/allmydata/test/test_repairer.py", line 206, in judge_invisible_corruption self.failIf(vr.is_healthy(), (vr, vr.is_healthy(), vr.get_data())) twisted.trial.unittest.FailTest: (, True, {'list-incompatible-shares': [], 'count-shares-expected': 10, 'count-incompatible-shares': 0, 'count-unrecoverable-versions': 0, 'count-shares-needed': 3, 'sharemap': {0: set(['\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&']), 1: set(['\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80']), 2: set(['\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b']), 3: set(['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i']), 4: set(["\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc"]), 5: set(['+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc']), 6: set(["\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6"]), 7: set(['!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T']), 8: set(["\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4"]), 9: set(['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t'])}, 'count-recoverable-versions': 1, 'servers-responding': ['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t', '\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b', "\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4", '\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&', '+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc', "\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6", '!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T', "\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc", 'W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i', '\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80'], 'count-good-share-hosts': 10, 'count-wrong-shares': 0, 'count-shares-good': 10, 'count-corrupt-shares': 0, 'list-corrupt-shares': []}) vr.data is: {'list-incompatible-shares': [], 'count-shares-expected': 10, 'count-incompatible-shares': 0, 'count-unrecoverable-versions': 0, 'count-shares-needed': 3, 'sharemap': {0: set(['\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&']), 1: set(['\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80']), 2: set(['\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b']), 3: set(['W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i']), 4: set(["\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc"]), 5: set(['+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc']), 6: set(["\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6"]), 7: set(['!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T']), 8: set(["\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4"]), 9: set(['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t'])}, 'count-recoverable-versions': 1, 'servers-responding': ['\xb9\xa3N\x80u\x9c_\xf7\x97FSS\xa7\xbd\x02\xf9f$:\t', '\xfcxa\xfa\xf4z\xf3I)\xe5\xc3z\xfa\x14\xb7\xaa\x9a\xbd\x90\x8b', "\x0e\xd6\xb3>\xd6\x85\x9d\x94')'\xf03:R\x88\xf1\x04\x1b\xa4", '\x8de\x1cqM\xba\xc3\x0b\x80\x9aC<5\xfc$\xdc\xd5\xd3\x8b&', '+\x83\xfa\xc0\xc3\xff\x16%X\xfcv\x96=w\xde\x8d\x15\xd1\xc0\xcc', "\xf2G'\xa0\xe0\xf3m\xa0g\xc5\x81\x97b\xe44\xd0\xb5J\xb4\xa6", '!\x92"\x0b\xf0\xdcF\x1c\xa9\xbe\xb6\xa8\xdfg\x08\xe1Zs+T', "\x02\xcd\xea\x10,'\xf7p'\xc2\x1a\xf0\xf7\xd6\xe3XT\x89*\xdc", 'W\xaa\x1d\x05\xe3l\x19}\xd2:\xa7\xae(\xc5\x7fh\xc3} i', '\xc4\x83\x9eJ\x7f\xac| .\xc90\xf4b\xe4\x92\xbe\xaa\xe6\t\x80'], 'count-good-share-hosts': 10, 'count-wrong-shares': 0, 'count-shares-good': 10, 'count-corrupt-shares': 0, 'list-corrupt-shares': []} =============================================================================== [FAIL]: allmydata.test.test_client.Basic.test_versions Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_client.py", line 169, in test_versions self.failIfEqual(str(allmydata.__version__), "unknown") twisted.trial.unittest.FailTest: 'unknown' == 'unknown' =============================================================================== [FAIL]: allmydata.test.test_iputil.ListAddresses.test_list_async Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_iputil.py", line 18, in _check self.failUnless("127.0.0.1" in addresses, addresses) twisted.trial.unittest.FailTest: ['10.2.0.12', '10.3.0.1', '127.0.0.2', '10.1.0.4'] =============================================================================== [FAIL]: allmydata.test.test_runner.TheRightCode.test_path Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_runner.py", line 37, in _cb self.failUnless(out.startswith(required_ver_and_path), (out, err, rc_or_sig, required_ver_and_path)) twisted.trial.unittest.FailTest: ('allmydata-tahoe: 0.0.0 (/v/tahoe/tahoe/src), foolscap: 0.4.2 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/foolscap-0.4.2-py2.5.egg), pycryptopp: 0.5.15 (/v/tahoe/pycryptopp), zfec: 1.4.5 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/zfec-1.4.5-py2.5-netbsd-4.99.7-i386.egg), Twisted: 8.2.0 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-netbsd-4.99.7-i386.egg), Nevow: 0.9.33-r17222 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/Nevow-0.9.33_r17222-py2.5.egg), zope.interface: 3.5.1 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/zope.interface-3.5.1-py2.5-netbsd-4.99.7-i386.egg), python: 2.5.2 (/usr/pkg/bin/python2.5), platform: NetBSD-4.99.7-i386-32bit-ELF (None), sqlite: None (unknown), simplejson: 2.0.9 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/simplejson-2.0.9-py2.5-netbsd-4.99.7-i386.egg), argparse: 0.9.1 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/argparse-0.9.1-py2.5.egg), pyOpenSSL: 0.9 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/pyOpenSSL-0.9-py2.5-netbsd-4.99.7-i386.egg), pyutil: 1.3.30 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/pyutil-1.3.30-py2.5.egg), zbase32: 1.1.1 (/v/tahoe/tahoe/support/lib/python2.5/site-packages/zbase32-1.1.1-py2.5.egg), setuptools: 0.6c12dev (/v/tahoe/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg), pysqlite: None (None)\n', '', 0, 'allmydata-tahoe: unknown (/v/tahoe/tahoe/src)') =============================================================================== [FAIL]: allmydata.test.test_system.SystemTest.test_vdrive Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_system.py", line 792, in _check in res) twisted.trial.unittest.FailTest: None =============================================================================== [ERROR]: allmydata.test.test_backupdb.BackupDB.test_basic Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 22, in test_basic bdb = self.create_or_skip(dbfile) File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 12, in create_or_skip bdb = backupdb.get_backupdb(dbfile, stderr=stderr) File "/v/tahoe/tahoe/src/allmydata/scripts/backupdb.py", line 50, in get_backupdb from pysqlite2 import dbapi2 exceptions.ImportError: No module named pysqlite2 =============================================================================== [ERROR]: allmydata.test.test_backupdb.BackupDB.test_check Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 74, in test_check bdb = self.create_or_skip(dbfile) File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 12, in create_or_skip bdb = backupdb.get_backupdb(dbfile, stderr=stderr) File "/v/tahoe/tahoe/src/allmydata/scripts/backupdb.py", line 50, in get_backupdb from pysqlite2 import dbapi2 exceptions.ImportError: No module named pysqlite2 =============================================================================== [ERROR]: allmydata.test.test_backupdb.BackupDB.test_fail Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 36, in test_fail stderr_f) File "/v/tahoe/tahoe/src/allmydata/scripts/backupdb.py", line 50, in get_backupdb from pysqlite2 import dbapi2 exceptions.ImportError: No module named pysqlite2 =============================================================================== [ERROR]: allmydata.test.test_backupdb.BackupDB.test_wrong_version Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 139, in test_wrong_version bdb = self.create_or_skip(where) File "/v/tahoe/tahoe/src/allmydata/test/test_backupdb.py", line 12, in create_or_skip bdb = backupdb.get_backupdb(dbfile, stderr=stderr) File "/v/tahoe/tahoe/src/allmydata/scripts/backupdb.py", line 50, in get_backupdb from pysqlite2 import dbapi2 exceptions.ImportError: No module named pysqlite2 =============================================================================== [ERROR]: allmydata.test.test_cli.Backup.test_backup Traceback (most recent call last): File "/v/tahoe/tahoe/src/allmydata/test/test_cli.py", line 868, in test_backup hush) File "/v/tahoe/tahoe/src/allmydata/scripts/backupdb.py", line 50, in get_backupdb from pysqlite2 import dbapi2 exceptions.ImportError: No module named pysqlite2 =============================================================================== [SUCCESS!?!]: allmydata.test.test_cli.Cp.test_unicode_filename Reason: 'This behavior is not yet supported, although it does happen to work (for reasons that are ill-understood) on many platforms. See issue ticket #534.' =============================================================================== [SUCCESS!?!]: allmydata.test.test_nevow.Web.test_read_default_css Reason: 'This patch that we submitted to Nevow fixes this issue: http://www.divmod.org/trac/ticket/2527' =============================================================================== [SUCCESS!?!]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_block_hash_tree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." =============================================================================== [SUCCESS!?!]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_offset_of_block_hashtree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." =============================================================================== [SUCCESS!?!]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_offset_of_block_hashtree_to_truncate_crypttext_hashtree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." =============================================================================== [SUCCESS!?!]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_offset_of_share_hashtree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." =============================================================================== [SUCCESS!?!]: allmydata.test.test_repairer.Verifier.test_verify_server_invisible_corruption_share_hash_tree_TODO Reason: "Verifier doesn't yet properly detect this kind of corruption." ------------------------------------------------------------------------------- Ran 602 tests in 458.155s FAILED (skips=3, expectedFailures=2, failures=4, errors=5, unexpectedSuccesses=7, successes=581) From trac at allmydata.org Tue Jul 7 12:09:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 19:09:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.7f2155e9763af9cb279f6bf7c648e2ba@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" -------------------------------+-------------------------------------------- Reporter: midnightmagic | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Fixed by [3966] which increases the requirement on pycryptopp to >= 0.5.15. Note that if you are building pycryptopp against an external libcryptopp, however, then you may still have this bug if your libcryptopp has it. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 14:06:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 21:06:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #747: Files uploaded to FTP server never appear In-Reply-To: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> References: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> Message-ID: <048.92c84d26d442f40aa1dd413fd4dcf068@allmydata.org> #747: Files uploaded to FTP server never appear -------------------------------+-------------------------------------------- Reporter: ndurner | Owner: zooko Type: defect | Status: closed Priority: major | Milestone: undecided Component: code-frontend | Version: 1.3.0 Resolution: invalid | Keywords: Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by ndurner): * status: new => closed * resolution: => invalid Comment: False alarm, sorry. I patched in the wrong directory, Tahoe's complaints were correct. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Tue Jul 7 15:59:05 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Wed, 08 Jul 2009 00:59:05 +0200 Subject: [tahoe-dev] Tahoe packaged for NixOS Message-ID: <87k52krtfq.fsf@gnu.org> Hello! I packaged Tahoe [0] and its dependencies [1] for Nixpkgs/NixOS. NixOS (http://nixos.org/) is a GNU/Linux distribution, which could well be dubbed "the least-authority GNU/Linux distro" given that it follows POLA for build processes. The build farm is currently in a bad shape, but one can normally see builds at, e.g., [2] and [3]. For zfec, pycryptopp, pyutil, etc., tests are run and all pass. However, trying to run Tahoe's tests currently fails: --8<---------------cut here---------------start------------->8--- running develop Checking .pth file support in support/lib/python2.5/site-packages error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: 'support/lib/python2.5/site-packages/test-easy-install-20210.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: support/lib/python2.5/site-packages This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). --8<---------------cut here---------------end--------------->8--- I haven't yet investigated, but I wonder where the offending path comes from. Thanks, Ludo'. [0] https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/tools/networking/p2p/tahoe/default.nix [1] https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/top-level/python-packages.nix [2] http://hydra.nixos.org/job/nixpkgs/trunk/tahoe [4] http://hydra.nixos.org/job/nixpkgs/trunk/pythonPackages.zfec From zooko at zooko.com Tue Jul 7 16:06:13 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 7 Jul 2009 17:06:13 -0600 Subject: [tahoe-dev] Tahoe packaged for NixOS In-Reply-To: <87k52krtfq.fsf@gnu.org> References: <87k52krtfq.fsf@gnu.org> Message-ID: On Jul 7, 2009, at 16:59 PM, Ludovic Court?s wrote: > I packaged Tahoe [0] and its dependencies [1] for Nixpkgs/NixOS. > NixOS (http://nixos.org/) is a GNU/Linux distribution, which could > well be dubbed "the least-authority GNU/Linux distro" given that it > follows POLA for build processes. Cool! :-) > Checking .pth file support in support/lib/python2.5/site-packages > error: can't create or remove files in install directory Oh, this should be fixed when I finally fix ticket #668. For what it is worth, this is also setuptools issue 54 (be more like distutils with regard to --prefix=), but it doesn't look like it is going to be fixed in setuptools anytime soon, so I intend to patch our version of setuptools to do this. Regards, Zooko tickets mentioned in this email: http://allmydata.org/trac/tahoe/ticket/668 # easy_install allmydata- tahoe is broken From zooko at zooko.com Wed Jul 8 06:00:46 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 8 Jul 2009 07:00:46 -0600 Subject: [tahoe-dev] buildbot news: Shawn's jaunty-amd64 demoted, BlackDew's debian-i386 and Zandr's lenny-armel promoted, Arthur's lenny-c7 added. Zooko's mac-powerpc dropped In-Reply-To: References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> Message-ID: <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Folks: Shawn's jaunty-amd64 machine had mysterious failures and has been demoted from "Supported" to "Unsupported" status. BlackDew's debian-i386 and Zandr's lenny-armel are green and have been promoted to "Supported". This makes for the first "embedded" CPU in the Supported category! :-) Arthur Lutz has contributed a C7 box running Debian, which as been added in the Unsupported category. Finally, we must say a sad farewell, at least temporarily, to my PowerPC G4 867 MHz laptop "draco". My four-year old son dropped draco and now it won't power up. http://allmydata.org/buildbot/ Regards, Zooko From trac at allmydata.org Wed Jul 8 21:28:44 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 04:28:44 -0000 Subject: [tahoe-dev] [tahoe-lafs] #752: speed up directories more Message-ID: <037.22675d737680f7eb9a51c0982f43738e@allmydata.org> #752: speed up directories more ---------------------------+------------------------------------------------ Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-dirnodes | Version: 1.4.1 Keywords: performance | Launchpad_bug: ---------------------------+------------------------------------------------ Even after [3969], [3971], [3972], ''and'' the not-yet-committed patch to generate IVs with hashes of writecaps (#750) ''and'' a not-yet-committed patch to run with {{{-OO}}} optimization mode, directories are still too slow. This set of five patches speeds directories up significantly: on my amd64 workstation yukyuk: before these five optimization patches: unpack and repack a 64-entry directory: 0.10 seconds, a 1024-entry directory: 1.75s, a 4096-entry directory: 7.75s, a 65536-entry directory: 441.65s. after these five optimization patches: unpack and repack a 64-entry directory: 0.08s, a 1024-entry directory: 1.3s, a 4096-entry directory: 4.95s, a 65536-entry directory: 115.4s on Zandr's 266 MHz armel box: before these five optimization patches: unpack and repack a 64-entry directory: 2.19 seconds, a 1024-entry directory: 35.48s, a 4096-entry directory: 183.05s (65536-entry directory not measured). after these five optimization patches: unpack and repack a 64-entry directory: 2.16s, a 1024-entry directory: 32.55s, a 4096-entry directory: 124.28s (a 65536-entry directory not measured). So this is a significant speed-up. Except wait -- 5s to open a directory with 4096 entries on an amd64 workstation, or 2 minutes to open it on a 266 MHz armel NAS? We can do better than that! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 8 21:43:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 04:43:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.a7e6b0a188b493d34e90d5419d82b1c8@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Okay, I went ahead and committed this in [3976]. Brian: I hope you sneak away from your vacation and log into the internet on your cell phone and review this patch before I release v1.5... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 09:40:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 16:40:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.e25adef058d655bd447afe4035145ebc@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): I think this is ok.. we're always using the same data with the same key, so no worries there.. we'd be enabling readers to mount a dictionary attack if we weren't already doing so by putting the readcap in the dirnode contents, so there's no increase in exposure there either. I think it would be more accurate to call this a "salt" instead of an "IV" though, since 1) we're using CTR mode, and 2) we're hashing this value to generate and AES key, not an IV. I don't know what I was thinking when I named that value "IV" in the original code. Also, looking at the code, we're using two hashes and not using the intermediate value anywhere. So how about using just one hash? Instead of: {{{ IV = hashutil.mutable_rwcap_iv_hash(self._node.get_writekey()) key = hashutil.mutable_rwcap_key_hash(IV, self._node.get_writekey()) cryptor = AES(key) }}} we could use something like: {{{ wk = self._node.get_writekey() key = hashutil.mutable_rwcap_key_hash2(wk) cryptor = AES(key) }}} where {{{mutable_rwcap_key_hash2}}} is a one-argument function that hashes a fixed prefix with the writekey. I too am surprised by the apparent speedup. Maybe the kernel context- switching time needed to read from /dev/urandom (or the refill-/dev/urandom code that gets invoked when you drain few kb from the entropy pool) is larger than the SHA256 code that gets invoked by the extra hash. In any case, using just one hash ought to be faster than either the original code or the attached patch. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 11:08:41 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 18:08:41 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.4e2f0a24ed98123f013138428a75740f@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Whoops, I had forgotten that the name "IV" was wrong, and was misled by it into thinking it ought to be separate from the key. Your suggested patch sounds better to me. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 18:59:41 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 01:59:41 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.528f3fb8d5c2d5b706aeb8737cb177d0@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by kevan): Note that r3971 didn't actually commit the optimization code -- just the dictionary that it uses. If you want that, you should also apply the optimizations.txt patch and the tests.txt patch (which also contains tests for CachingDict). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 19:21:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 02:21:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.e12a05813c4a679c54f783cde7161200@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Oh how embarassing that I thought that was the optimization patch. I ''was'' actually going to ask you what you thought about the mysterious fact that [3971] seems to moderately speed up pack and unpack but not so much unpack-and-repack! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 20:24:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 03:24:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.ee27b173168b26831182b358912c67f7@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Now that I've applied your actual optimization patch, time to unpack and repack a 4096-entry directory dropped substantially. However, time to pack and time to unpack also changed, which suggests that my benchmark script is accidentally using your cache, or that extraneous factors on my system are screwing up the results or something. Before {{{optimizations.txt}}}: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.09, ave: 0.10, 2th- worst: 0.10, worst: 0.15 (of 5), reps/s: 9, ave rate: 632 N: 256, time: best: 0.35, 2th-best: 0.37, ave: 0.38, 2th- worst: 0.39, worst: 0.40 (of 5), reps/s: 2, ave rate: 675 N: 1024, time: best: 1.40, 2th-best: 1.42, ave: 1.63, 2th- worst: 1.74, worst: 1.88 (of 5), reps/s: 0, ave rate: 629 N: 4096, time: best: 5.29, 2th-best: 5.90, ave: 6.16, 2th- worst: 6.36, worst: 7.15 (of 5), reps/s: 0, ave rate: 665 benchmarking N: 64, time: best: 0.09, 2th-best: 0.09, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 705 N: 256, time: best: 0.36, 2th-best: 0.36, ave: 0.39, 2th- worst: 0.37, worst: 0.51 (of 5), reps/s: 2, ave rate: 650 N: 1024, time: best: 1.38, 2th-best: 1.43, ave: 1.51, 2th- worst: 1.51, worst: 1.80 (of 5), reps/s: 0, ave rate: 676 N: 4096, time: best: 5.22, 2th-best: 5.77, ave: 6.08, 2th- worst: 6.33, worst: 6.79 (of 5), reps/s: 0, ave rate: 674 benchmarking N: 64, time: best: 0.10, 2th-best: 0.10, ave: 0.14, 2th- worst: 0.18, worst: 0.19 (of 5), reps/s: 7, ave rate: 473 N: 256, time: best: 0.36, 2th-best: 0.37, ave: 0.39, 2th- worst: 0.40, worst: 0.42 (of 5), reps/s: 2, ave rate: 664 N: 1024, time: best: 1.41, 2th-best: 1.42, ave: 1.46, 2th- worst: 1.48, worst: 1.52 (of 5), reps/s: 0, ave rate: 702 N: 4096, time: best: 5.22, 2th-best: 5.58, ave: 5.81, 2th- worst: 6.09, worst: 6.20 (of 5), reps/s: 0, ave rate: 705 }}} after {{{optimizations.txt}}}: {{{ benchmarking N: 64, time: best: 0.05, 2th-best: 0.05, ave: 0.06, 2th- worst: 0.06, worst: 0.07 (of 5), reps/s: 17, ave rate: 1091 N: 256, time: best: 0.21, 2th-best: 0.21, ave: 0.23, 2th- worst: 0.24, worst: 0.27 (of 5), reps/s: 4, ave rate: 1093 N: 1024, time: best: 0.83, 2th-best: 0.84, ave: 0.88, 2th- worst: 0.87, worst: 1.00 (of 5), reps/s: 1, ave rate: 1170 N: 4096, time: best: 3.29, 2th-best: 3.35, ave: 3.54, 2th- worst: 3.58, worst: 4.09 (of 5), reps/s: 0, ave rate: 1156 benchmarking N: 64, time: best: 0.06, 2th-best: 0.06, ave: 0.06, 2th- worst: 0.06, worst: 0.06 (of 5), reps/s: 17, ave rate: 1106 N: 256, time: best: 0.22, 2th-best: 0.22, ave: 0.22, 2th- worst: 0.22, worst: 0.23 (of 5), reps/s: 4, ave rate: 1158 N: 1024, time: best: 0.85, 2th-best: 0.85, ave: 0.86, 2th- worst: 0.87, worst: 0.88 (of 5), reps/s: 1, ave rate: 1188 N: 4096, time: best: 3.27, 2th-best: 3.28, ave: 3.36, 2th- worst: 3.38, worst: 3.60 (of 5), reps/s: 0, ave rate: 1218 benchmarking N: 64, time: best: 0.06, 2th-best: 0.06, ave: 0.07, 2th- worst: 0.09, worst: 0.10 (of 5), reps/s: 13, ave rate: 881 N: 256, time: best: 0.22, 2th-best: 0.22, ave: 0.28, 2th- worst: 0.26, worst: 0.46 (of 5), reps/s: 3, ave rate: 929 N: 1024, time: best: 0.85, 2th-best: 0.90, ave: 1.00, 2th- worst: 1.02, worst: 1.34 (of 5), reps/s: 0, ave rate: 1022 N: 4096, time: best: 3.25, 2th-best: 3.33, ave: 3.49, 2th- worst: 3.67, worst: 3.74 (of 5), reps/s: 0, ave rate: 1172 }}} I've applied your patches plus one from me to fix a conflict between one of my optimizations and one of yours. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From kpreid at mac.com Thu Jul 9 20:40:37 2009 From: kpreid at mac.com (Kevin Reid) Date: Thu, 9 Jul 2009 23:40:37 -0400 Subject: [tahoe-dev] Mac users: Publish your iCal calendars over Tahoe Message-ID: <282AB964-F9F1-4B01-AF20-C78866C53073@mac.com> How to publish iCal calendars over Tahoe: 1. Create a directory and get its http-URL cap, e.g. http://localhost:9797/uri/URI%3ADIR2%3Ablahblahblah/ 2. In the iCal publishing settings choose "Publish on: a Private Server". Paste the the aforementioned directory cap into the Base URL field. Enter anything in the Login and Password fields, because iCal insists on them. (Note also that if you reenter the publish dialog ("Change Location...") the port number will be missing. This is a bug in iCal.) 3. Get the read-only directory cap by going to the directory in a browser and clicking "Read-Only Version", e.g. http://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/ Append to it whatever you entered in the "Publish calendar as:" field, followed by ".ics". This is the URL to subscribe to. http://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/foo.ics (Or, if you aren't terribly annoyed by URL-scheme-abuse-as-disposition- instructions, that's webcal://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/foo.ics ) (Note: I'm using a directory instead of a mutable file because iCal insists on having a url of the format /.ics, and I didn't immediately find anything in the tahoe web interface that would be in that format and also PUTable.) -- Kevin Reid From trac at allmydata.org Fri Jul 10 02:41:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 09:41:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance Message-ID: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ As we work on new encoding schemes (like DSA-based mutable files), I'm thinking that we want to put a lower bound on the cap/SI length to maintain reasonable margin against collisions. 256 bits would be more than enough. 128 is ok, but a bit tight. 92 bits would make me nervous. robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation before it) because, for something that is meant as a backup system, even the slightest possibility of the CHK-based indexing scheme mapping two documents to the same storage index was too high for him. (I believe he would be more satisfied with a scheme that used centrally-allocated guaranteed- unique storage indices, which we could do but would require more coordination and longer caps, since we could no longer use a randomly-generated readkey to derive the storage index. In exchange for a controllable but non-zero probability of collision, we get to avoid central coordination and use smaller caps). The specific places where collisions could occur are: * mapping from file contents to CHK-derived readkey * mapping from readkey (CHK-derived or randomly generated) to storage index * mapping from randomly-generated mutable writekey to storage index The "birthday paradox" determines the chance of collision. If I'm doing my math right, if you want less than '{{{p}}}' chance of getting any collisions when selecting items out of a namespace of size '{{{N}}}', then you can't select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" (where "surprise" would be a better term) because that square root causes C to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to C=19. In the Tahoe context, {{{C}}} is the number of files you can add to the grid. In the current case, our 128-bit storage index (N=2**128) means that p=0.5 gets us a nice large 2**64 number of files, except that p=0.5 is insufficient margin: we'd much prefer a vanishingly small chance of collision, like p=2**-64. Fortunately we get two bits of margin for every one bit we reduce from C. The table looks like: ||N||numfiles||prob(collision)|| ||96||2**48||2**-1 (0.5)|| ||96||2**40||2**-16|| ||96||2**32||2**-32|| ||96||2**24||2**-48|| ||128||2**64||2**-1 (0.5)|| ||128||2**56||2**-16|| ||128||2**48||2**-32|| ||128||2**32||2**-64|| ||256||2**128||2**-1 (0.5)|| ||256||2**96||2**-64|| Note that our {{{N}}} is the minimum of the storage-index size and the top-most cap value (i.e. the readkey for immutable files, or the writekey for mutable files). So a DSA-based mutable file with a 92-bit writecap gives us an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 or 256 bits. Also note that the allmydata.com grid currently has something like 10M objects in it, about C=2**23. So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap would be nice, it's too short to provide enough collision margin. I want to be able to put trillions of files into a grid, and I want a the chance of collision to be so small that I don't ever need to worry about it, and 96 bits isn't really there. 128 bits is probably good enough, but doesn't have enough margin to be obviously and unquestionably safe (C=2**32 is a lot of files but you can imagine people wanting more, p=2**-64 is a tiny probability but you can imagine people wanting a bit better). 256 would be plenty (but of course I want my filecaps to be shorter than that). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 07:14:43 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 14:14:43 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.c8e38efecda693263f3db9c04b64358b@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Thanks for this analysis. I like your comments at the end that we want a bit of "overkill" in number of files and chance of collision. People who don't want to rely on the collision-resistance of secure hash functions at ''all'' are probably not part of our market, but people who are willing to do so in principle, but would feel better with a nice fat margin of safety are definitely in our market. Note that if you generate a new write cap (private key) and then check and it turns out that the same write cap has been generated by someone else then now you have gained the ability to write to their mutable file or directory! That's why I have been thinking that 96-bits was too few for write caps. Originally I had been thinking something like "It would probably not be worthwhile for any attacker to spend 2^96^ computer power trying to forge a write cap.". But this way of thinking discounts two important factors: chance of success and number of targets. If there are 2^40^ writecaps in use, then if an attacker does a mere 2^36^ work (excluding the cost of checking whether each writecap that they generate is already out there), then they have a 2^-20^ chance of forging ''someone's'' writecap. (Thanks to Daniel J. Bernstein's papers and mailing list postings for helping me understand this important point. http://cr.yp.to/snuffle/bruteforce-20050425.pdf ) However, a storage-index collision doesn't sound nearly as serious to me. No integrity or confidentiality is necessarily lost due to storage-index collision, right? Well, it could complicate the question of "which servers are handling which shares of this mutable file" -- an issue that is already not well managed. Anyway, nowadays I think of storage-indexes as a separate layer built on top of the crypto layer. People can define their storage indexes as secure hashes of some pieces of the capabilities if they want, or choose random storage indices, or hierarchical ones based on the DNS names, or just not have storage indices at all and require every downloader to query every server. It shouldn't impact the security of the crypto layer, if the crypto layer includes integrity checking using the verifycap itself on the storage-server side. I think we should write a document called something like "crypto failure modes (What could possibly go wrong?)" that explains what the consequences are of each different type of failure. (As requested by Nathan: http://allmydata.org/pipermail/tahoe-dev/2009-April/001661.html .) The one discussed above is caused by "two people choose the same write cap (signing key) seed (possibly due to malice)". That one leads to an integrity failure, where one of the people thinks that they are the only one with a write-cap to a specific file or directory, but actually someone else also has the same write-cap. So I think that is the worst one (because I value integrity highly). Another similar integrity failure could come about from a failure of the digital signature algorithm -- i.e. if someone were able to forge digital signatures even without the writecap. (Note that a collision in the hash function used by the digital signature algorithm could cause this. People who don't want to rely on collision-resistance of secure hash functions at ''all'' can't even rely on rsa, dsa, ssh, ssl, or gpg, although I should hasten to add that those algorithms typically include some randomness in the input to their secure hash function, to make it that much harder for attackers to cause collisions.) After those integrity failures, there are confidentiality failures. The obvious one is someone being able to crack AES-128-CTR without having the readkey. Another one is if the content-hash-key encryption where to generate the same readkey for two different immutable files. I suppose that's another reason why using an {{{added convergence secret}}} is safer (although I hasten to add that I see no reason to distrust the collision- resistance of SHA-256d-truncated-to-128-bits at the present). Note that of course in practice the dangers from bugs and from operator error (i.e. misplacing your keys) are a lot greater than these algorithmic crypto risks. So much greater that the former are pretty much guaranteed to happen and the latter will probably never. Nonetheless, I value getting the crypto part right so that it is secure and also so that everyone who is willing to rely on crypto in principle is willing to rely on our crypto, so thanks for you help with this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 07:27:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 14:27:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.2a61e7ade88853f48bbfd3a29371d39e@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): P.S. Oh, what I wrote about people choosing whatever storage index they want is possibly confused because I'm conflating two ideas into the word "storage index". Let's talk about those two uses as "server selector" and "file identifier". The former is how you decide which servers you're going to use (either when uploading a file for the first time or when using a cap to find a file that's already up there). The latter is how you tell a storage server which of the shares that it has are the shares that you are currently interested in. I currently think that the verify cap is a good thing to use for the latter role, but perhaps not for the former. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 22:48:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 05:48:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d626ddfe413e7f8ad4b64c6c22b2bd6e@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): I'm attaching a patch that implements + tests for the behavior described above. However, when testing, I came upon another maybe-bug. If, using the trunk build from last night, I do {{{ tahoe mv tahoe:file1 file1 }}} with the expectation that it will copy the remote file to my system, it purports to succeed (i.e.: no error message) but doesn't actually copy anything to my system, and unlinks the remote file. Is this what we want to happen? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 03:25:36 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 10:25:36 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.f3fe64637b23c67b7e4c95e8b79c2ab4@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Old description: > As we work on new encoding schemes (like DSA-based mutable files), I'm > thinking that we want to put a lower bound on the cap/SI length to > maintain > reasonable margin against collisions. 256 bits would be more than enough. > 128 > is ok, but a bit tight. 92 bits would make me nervous. > > robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation > before it) because, for something that is meant as a backup system, even > the > slightest possibility of the CHK-based indexing scheme mapping two > documents > to the same storage index was too high for him. (I believe he would be > more satisfied with a scheme that used centrally-allocated guaranteed- > unique > storage indices, which we could do but would require more coordination > and > longer caps, since we could no longer use a randomly-generated readkey to > derive the storage index. In exchange for a controllable but non-zero > probability of collision, we get to avoid central coordination and use > smaller caps). > > The specific places where collisions could occur are: > > * mapping from file contents to CHK-derived readkey > * mapping from readkey (CHK-derived or randomly generated) to storage > index > * mapping from randomly-generated mutable writekey to storage index > > The "birthday paradox" determines the chance of collision. If I'm doing > my > math right, if you want less than '{{{p}}}' chance of getting any > collisions > when selecting items out of a namespace of size '{{{N}}}', then you can't > select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" > (where "surprise" would be a better term) because that square root causes > C > to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to > C=19. > In the Tahoe context, {{{C}}} is the number of files you can add to the > grid. > > In the current case, our 128-bit storage index (N=2**128) means that > p=0.5 > gets us a nice large 2**64 number of files, except that p=0.5 is > insufficient > margin: we'd much prefer a vanishingly small chance of collision, like > p=2**-64. Fortunately we get two bits of margin for every one bit we > reduce > from C. The table looks like: > > ||N||numfiles||prob(collision)|| > ||96||2**48||2**-1 (0.5)|| > ||96||2**40||2**-16|| > ||96||2**32||2**-32|| > ||96||2**24||2**-48|| > ||128||2**64||2**-1 (0.5)|| > ||128||2**56||2**-16|| > ||128||2**48||2**-32|| > ||128||2**32||2**-64|| > ||256||2**128||2**-1 (0.5)|| > ||256||2**96||2**-64|| > > Note that our {{{N}}} is the minimum of the storage-index size and the > top-most cap value (i.e. the readkey for immutable files, or the writekey > for > mutable files). So a DSA-based mutable file with a 92-bit writecap gives > us > an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 > or > 256 bits. > > Also note that the allmydata.com grid currently has something like 10M > objects in it, about C=2**23. > > So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap > would be nice, it's too short to provide enough collision margin. I want > to > be able to put trillions of files into a grid, and I want a the chance of > collision to be so small that I don't ever need to worry about it, and 96 > bits isn't really there. 128 bits is probably good enough, but doesn't > have > enough margin to be obviously and unquestionably safe (C=2**32 is a lot > of > files but you can imagine people wanting more, p=2**-64 is a tiny > probability > but you can imagine people wanting a bit better). 256 would be plenty > (but of > course I want my filecaps to be shorter than that). New description: As we work on new encoding schemes (like DSA-based mutable files), I'm thinking that we want to put a lower bound on the cap/SI length to maintain reasonable margin against collisions. 256 bits would be more than enough. 128 is ok, but a bit tight. 92 bits would make me nervous. robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation before it) because, for something that is meant as a backup system, even the slightest possibility of the CHK-based indexing scheme mapping two documents to the same storage index was too high for him. (I believe he would be more satisfied with a scheme that used centrally-allocated guaranteed- unique storage indices, which we could do but would require more coordination and longer caps, since we could no longer use a randomly-generated readkey to derive the storage index. In exchange for a controllable but non-zero probability of collision, we get to avoid central coordination and use smaller caps). The specific places where collisions could occur are: * mapping from file contents to CHK-derived readkey * mapping from readkey (CHK-derived or randomly generated) to storage index * mapping from randomly-generated mutable writekey to storage index The "birthday paradox" determines the chance of collision. If I'm doing my math right, if you want less than '{{{p}}}' chance of getting any collisions when selecting items out of a namespace of size '{{{N}}}', then you can't select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" (where "surprise" would be a better term) because that square root causes C to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to C=19. In the Tahoe context, {{{C}}} is the number of files you can add to the grid. In the current case, our 128-bit storage index (N=2**128) means that p=0.5 gets us a nice large 2**64 number of files, except that p=0.5 is insufficient margin: we'd much prefer a vanishingly small chance of collision, like p=2**-64. Fortunately we get two bits of margin for every one bit we reduce from C. The table looks like: ||N||numfiles|| ||prob(collision)|| ||96||2**48||->||2**-1 (0.5)|| ||96||2**40||->||2**-17|| ||96||2**32||->||2**-33|| ||96||2**24||->||2**-49|| ||128||2**64||->||2**-1 (0.5)|| ||128||2**56||->||2**-17|| ||128||2**48||->||2**-33|| ||128||2**32||->||2**-65|| ||192||2**96||->||2**-1|| ||192||2**80||->||2**-33|| ||192||2**64||->||2**-65|| ||256||2**128||->||2**-1 (0.5)|| ||256||2**96||->||2**-65|| Note that our {{{N}}} is the minimum of the storage-index size and the top-most cap value (i.e. the readkey for immutable files, or the writekey for mutable files). So a DSA-based mutable file with a 92-bit writecap gives us an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 or 256 bits. Also note that the allmydata.com grid currently has something like 10M objects in it, about C=2**23. So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap would be nice, it's too short to provide enough collision margin. I want to be able to put trillions of files into a grid, and I want a the chance of collision to be so small that I don't ever need to worry about it, and 96 bits isn't really there. 128 bits is probably good enough, but doesn't have enough margin to be obviously and unquestionably safe (C=2**32 is a lot of files but you can imagine people wanting more, p=2**-64 is a tiny probability but you can imagine people wanting a bit better). 256 would be plenty (but of course I want my filecaps to be shorter than that). -- Comment(by warner): as before, I think I'd like to continue using "storage index" for what you're calling the "file identifier", but yeah split out "server selector" or "peer-selection index" or some similar term for the purpose of determining which servers you're going to be talking to. One way of describing this would be "we used to use the storage-index as the peer-selection index, but these days we put two separate values in the filecap". I am also starting to think of these as separate concepts, but remember that we've yet to actually implement such a split. Sylvan's concern was about availability: he considered a backup system to be broken if its design has a built-in probability of file unrecoverability. It's easier to see the problem if we set the encryption-key and hash lengths to infinity, but restrict the storage index to say four bits. Then upload two files, and try to download one of them.. you've got a 1/16 chance of getting a download failure because your two files had the same storage-index, you downloaded the wrong bits, and now they won't pass the integrity check. Also, when we talk about this, we should be careful to distinguish between the failure modes of mutable files versus immutable files.. they're very distinct. And, collisions at different levels have very different consequences: if the storage index is too small, we'll get availability failures; if the immutable encryption key or mutable writekey is too small, we'll get confidentiality failures. I've been assuming that we'll keep the security parameters sufficiently large.. this ticket was specifically about the availability concerns resulting from a too-small storage index. If we compress the filecap by deriving the storage-index from the writekey, then clearly we're limited by {{{min(len(writekey),len(storage-index))}}}. Mostly I ticketed this issue because it's something I want to keep in mind as we design the next revision of the filecap format. If we don't already have a wiki page for it, I'll add one to organize our ideas.. I think they're currently spread across half a dozen tickets. I updated the table in the description: I think 192-bit caps will let us have an effectively-infinite number of files (2**64) with an effectively-zero chance of collision (2**-65). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:23:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:23:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #674: controlled access to your WUI In-Reply-To: <037.61baae0405033f54a68bcd066354261f@allmydata.org> References: <037.61baae0405033f54a68bcd066354261f@allmydata.org> Message-ID: <046.3a4f3df4508ab893b934394af8ff3988@allmydata.org> #674: controlled access to your WUI -------------------------+-------------------------------------------------- Reporter: zooko | Owner: nobody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.3.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by warner): good point. I suppose that means making the $WUI_SECRET pages come from a completely different origin, by using a separate port number or something. We'd still need $WUI_SECRET, of course. We'll have to have a rule that says we never emit unescaped external content on any page served below $WUI_SECRET . That means not using this namespace for delivering files or directories, and it also means being careful about escaping node nicknames and log messages and anything else that we might want to serve from that space. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:31:12 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:31:12 -0000 Subject: [tahoe-dev] [tahoe-lafs] #495: zfec crashes and incorrect precondition values In-Reply-To: <036.ff54e5856c0f9cdb10822c30c5bdfb32@allmydata.org> References: <036.ff54e5856c0f9cdb10822c30c5bdfb32@allmydata.org> Message-ID: <045.313f56345acbe2af07d39a3724f7f52d@allmydata.org> #495: zfec crashes and incorrect precondition values ------------------------+--------------------------------------------------- Reporter: greg | Owner: zooko Type: defect | Status: closed Priority: minor | Milestone: undecided Component: code | Version: unknown Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed * component: unknown => code Comment: This appears to be fixed in zfec-1.4.5 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:44:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:44:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #714: Not possible to renew leases from the WUI. In-Reply-To: <038.6fc6800acc928e5cdcc1648750d676f0@allmydata.org> References: <038.6fc6800acc928e5cdcc1648750d676f0@allmydata.org> Message-ID: <047.18158e3309b645d3981aae535ee269be@allmydata.org> #714: Not possible to renew leases from the WUI. -----------------------------------+---------------------------------------- Reporter: kpreid | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | -----------------------------------+---------------------------------------- Changes (by warner): * status: assigned => closed * resolution: => fixed * milestone: eventually => 1.5.0 Comment: done, in [3937], so it should be in 1.5.0 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 06:11:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 13:11:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #432: writing down filecaps: revise URI scheme In-Reply-To: <038.2bb9b2fab7dfea20676c145eca6fdd40@allmydata.org> References: <038.2bb9b2fab7dfea20676c145eca6fdd40@allmydata.org> Message-ID: <047.255569cba90686b8ef9709c62b2a34b4@allmydata.org> #432: writing down filecaps: revise URI scheme ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: task | Status: new Priority: major | Milestone: eventually Component: code-encoding | Version: 1.0.0 Keywords: caps | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): NewCapDesign is the place to design the new cap format. It contains a detailed list of design criteria. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 11:03:36 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 18:03:36 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.43305e40bb1caae498925ff30b9bd733@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): I was just trying to make my storage servers for [wiki:VolunteerGrid the volunteer grid] more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. So this is just a note to remind myself that I wish someone would implement this. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 11:12:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 18:12:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location Message-ID: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- I'm configuring a laptop which is behind a home NAT to have a specified {{{tub.location}}} which says the IP address of my home network. (As described in [source:docs/configuration.txt].) However, sometimes I take this laptop to other networks, and it would be nice if it would do the process of auto-detecting its current IP address and advertise both that ''and'' the manually-configured IP address that I am putting in, so that if clients can connect to it either way then they win. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 14:00:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 21:00:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information Message-ID: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- If I do a deep-check on a directory, I start getting results reported on the web page showing the files and subdirectories within that directory. Reloading (or waiting for the automatic self-reload) shows more and more results. Until one of the subdirectories is unrecoverable, in which case the web page containing the deep check results is replaced with a web page saying only this: {{{ UnrecoverableFileError: the directory (or mutable file) could not be retrieved, because there were insufficient good shares. This might indicate that no servers were connected, insufficient servers were connected, the URI was corrupt, or that shares have been lost due to server departure, hard drive failure, or disk corruption. You should perform a filecheck on this object to learn more. }}} To close this ticket, make it so that I can still see all the other result that have already been generated, plus further results about other files and subdirectories that haven't yet been checked, even while there is an unrecoverable subdirectory present. I'm using the current trunk: 1.4.1-r3982. Brian: are you willing to take this ticket? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 11 14:50:43 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 11 Jul 2009 15:50:43 -0600 Subject: [tahoe-dev] calling all VolunteerGrid operators Message-ID: Folks: If you are contributing a storage server to The VolunteerGrid, would you please update the VolunteerGrid page on the wiki and describe your server a little? Thanks. After this mail, which is going out to tahoe-dev, I'll probably send future mails about the VolunteerGrid only to people whose contact info is listed on that page. Regards, Zooko From trac at allmydata.org Sat Jul 11 15:56:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 22:56:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location In-Reply-To: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> References: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> Message-ID: <046.20cbce44f4693fd5521fc7f1a9b06b77@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): maybe we could use a magic value, like {{{tub.location = auto+1.2.3.4:1234}}}, which is interpreted to mean "merge {{{1.2.3.4:1234}}} with the auto-detected location". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 16:20:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 23:20:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #612: build binary packages of pysqlite In-Reply-To: <037.c2244a0bcbee3a7fb6fe0ac856505fe0@allmydata.org> References: <037.c2244a0bcbee3a7fb6fe0ac856505fe0@allmydata.org> Message-ID: <046.951885ef896789f5ce82b5319acf0e5e@allmydata.org> #612: build binary packages of pysqlite ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: undecided Component: packaging | Version: 1.2.0 Resolution: wontfix | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Changes (by zooko): * status: new => closed * resolution: => wontfix Comment: Since Python >= 2.5 comes with sqlite built in and since I'm hoping to drop support for Python 2.4 after the 1.5.0 release, I'm closing this ticket as wontfix. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 16:28:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 23:28:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information In-Reply-To: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> References: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> Message-ID: <046.f6ea262473f49ac4940020aded70f0a7@allmydata.org> #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Changes (by warner): * status: new => assigned * component: code-frontend-web => code-dirnodes Comment: yeah, I'll work on this. Basically traversal failures during a deep-check or deep-repair operation should increment a counter and move on, instead of throwing an exception and stopping the walker. I don't know if I can finish it in time for 1.5.0 though. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 03:21:43 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 10:21:43 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c078d69a8cda56882185c8bf057d9e49@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by warner): Hm, good question. "tahoe mv" was indeed meant for tahoe-to-tahoe moves, so that an unqualified "file1" really means "tahoe:file1". (the fact that "tahoe mv alias:file1 alias:file1" results in deleting a file is certainly a bug, probably in the dirnode code). I don't know if it's better to add docs/etc to teach people to expect that "tahoe mv" doesn't touch the local disk, or to add code/tests to make "tahoe mv" behave more like people's existing expectations. There's value in having all tahoe CLI commands that happen to overlap regular unix commands (cp, ln, mv, rm) be prepared to handle both local-disk and tahoe- filesystem arguments. But it also adds code, complication, and redundancy (e.g. why would you ever use "tahoe rm ~/.emacs" instead of regular rm?). Overall, I guess 'mv' should handle both local and tahoe-side files, and should behave like 'cp'. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 03:35:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 10:35:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.ac39bf40ae408a50fc2ccda8b205d12a@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): oh, I'm being stupid and the new code is bad. What we're doing is encrypting a list of child writecaps in such a way that the containing dirnode's writecap is required to retrieve them. Obviously each child writecap must be encrypted with a different key. It is reasonable to use the hash of the child writecap as a salt (since there's no requirement that the constant child writecap be encrypted differently each time), but it is not ok to use the hash of the parent dirnode writecap as a salt (because that's the same for all children). My suggested change is wrong too. The code needs to be more like: {{{ salt = hashutil.mutable_rwcap_iv_hash(rwcap) key = hashutil.mutable_rwcap_key_hash(salt, self._node.get_writekey()) crypto = AES(key) }}} I'll make this change soon. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 07:21:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 14:21:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.45dfcecb759e1003826585a2cd0d1b98@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Whoops. Ouch. I'm sure glad you caught that! I was thinking when I wrote the patch that I was using the writecap of the entry as the input, but obviously I was using the key of the directory! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 07:24:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 14:24:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.4e59327da6bbf1dfd45a03760d3e5f8b@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): By the way, a more traditional way to do something like this ''would'' be to use the same key (the one for the dir) to encrypt each entry and use a unique IV for each entry. We are in the habit of instead generating a unique key for each thing we want to encrypt and typically just leaving the IV at 0, which seems fine to me, too. Your proposed fix is in the latter tradition. Please hurry up and commit it so that nobody uses trunk to write directories insecurely! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Sun Jul 12 11:31:17 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 12:31:17 -0600 Subject: [tahoe-dev] Is anyone using GridBackup? Message-ID: <200907121231.17312.shawn-tahoe@willden.org> I hope not, because I'm about to rev the read cap index format, and I'd rather not have to maintain backward compatibility, yet. I doubt anyone is, but I thought I'd better toss this out just in case. If you are using it, let me know. I still won't keep support for the previous format in the code, but I'll put together a script to update your read caps -- but only if it's really important to you. Going forward, I have infrastructure in place to allow the index format to evolve, so future changes will be able to maintain backward compatibility. Shawn. From trac at allmydata.org Sun Jul 12 16:09:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 23:09:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.3d3a709ec974f427a0816ba1953dff20@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): Huh, I wasn't even aware of pycryptopp offering IV!=0. I'll test the patch tomorrow, but I'm not sure I'll be able to to commit it until about 24h from now. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 16:52:08 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 23:52:08 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.543b7669887a86d5f8525534fbf7393d@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by warner): * status: reopened => closed * resolution: => fixed Comment: pushed, in [3984]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Sun Jul 12 17:45:32 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 18:45:32 -0600 Subject: [tahoe-dev] UEB hash size Message-ID: <200907121845.32680.shawn-tahoe@willden.org> Hi guys, What's the rationale for including the full 256-bit UEB hash in the CHK URI? Those URIs could be shortened considerably by truncating it to, say, 128 bits. How difficult would it be to allow Tahoe to operate with either full UEB hashes or abbreviated hashes? I'm thinking I could shorten my read cap index entries significantly if I could store only half of the UEB hash. Along those lines in the context of small, trusted grids, it might even be reasonable to omit the UEB hash entirely (in my case I have a separate content hash, and it should also be possible to hash the retrieved and decrypted file to regenerate the read key and check integrity that way, assuming an unchanged convergence secret). What is the bare minimum data needed to retrieve, reassemble and decrypt an immutable file? Just the AES read key? The web API doesn't like CHK URIs that conform to the full specification, but I'm sure that could be changed easily enough, if it made sense. Shawn. From trac at allmydata.org Sun Jul 12 18:02:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 01:02:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.033e3dad16b4dc934a1420c44e7a83fa@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:7 zooko]: > I was just trying to make my storage servers for [wiki:VolunteerGrid the volunteer grid] more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. > > So this is just a note to remind myself that I wish someone would implement this. :-) I've fiddled a little bit with miniupnpc, which ghazel already mentioned. It's a C library but includes a Python wrapper. It's pretty simple. The following code was all it took to tell my gateway (a Linux box running linux-igd) to set up a forwarding for my Tahoe node: {{{ import miniupnpc u = miniupnpc.UPnP() u.discoverdelay = 200 u.discover() u.selectigd() external_port = 3456 r = u.getspecificportmapping(external_port, 'TCP') while r != None and external_port < 65536: external_port += 1 r = u.getspecificportmapping(external_port, 'TCP') u.addportmapping(external_port, 'TCP', u.lanaddr, 3456, "Tahoe %u" % external_port, '') }}} Obviously, that needs a lot of error checking, but it's still really straightforward. I ran that script from the command line, and watched the log on my gateway to verify that the mapping was set up. I then verified that I can connect to my local Tahoe server from outside. Works fine. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Sun Jul 12 18:37:12 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 19:37:12 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907121845.32680.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> Message-ID: <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> I really don't like the idea of adding an insecure mode, if we can help it. Could you explain more -- perhaps even quantify -- what impact the size of the cap has on your system? Hopefully there's a solution which is both secure and efficient. Regards, Zooko From trac at allmydata.org Sun Jul 12 19:18:55 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 02:18:55 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.ac182060dc9195fbdaf7c7f1d82efe02@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Wait, if you you have 4-bit storage indexes, ''and servers verify integrity of shares'', then you don't get an unrecoverable file, you get a failure to upload the second one, right? I really like the idea of servers verifying the integrity of shares, because without that then even if you have a nice fat storage index, the first server that you upload to could quickly turn around and upload garbage to all the other servers, under the same storage index that you just told him. A failure to upload (backup) is a much better problem to have than a failure to download (recover), as long as it can be detected, understood, and fixed by the operators. I.e. "Oh, we need to add servers", or "Oh, we need to start using longer storage indexes". :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Sun Jul 12 19:50:43 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 20:50:43 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907121845.32680.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> Message-ID: <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> And in answer to your questions: On Jul 12, 2009, at 18:45 PM, Shawn Willden wrote: > What's the rationale for including the full 256-bit UEB hash in the > CHK URI? Those URIs could be shortened considerably by truncating > it to, say, 128 bits. It is that the integrity of an immutable file cap is the "exactly one file matches this cap" guarantee. To ensure this requires 2K bits in the immutable cap to guarantee K bits of security, because of a birthday-surprise attack in which an attacker generates two (or more) files with the same immutable file cap so that they have the ability to undetectably swap in the alternate files substituted for the original file, after they've distributed the cap to other people. To generate such files requires only about K bits of work to find multiple matching files for a 2K-bit immutable file cap. > How difficult would it be to allow Tahoe to operate with either > full UEB hashes or abbreviated hashes? It is a neat idea. We've discussed it before, but I can't find the reference. I seem to recall that Brian had a good summary of the risk of publishing a shortened immutable cap. Perhaps he just pointed out that in the future people may come to distrust whether the file that they get by retrieving with that cap is really the only file that could have matched. If your shortened cap is sufficiently, let's say 192-bits, this risk doesn't sound like a big issue as far as brute computer power goes -- even if people in the future have vastly improved computation technology, 2^96 computations will probably still be very, perhaps even prohibitively, expensive. However, the possibility of people uncovering algorithmic weaknesses in the hash algorithm that we are using (currently SHA-256d, hopefully in the future SHA-3) can reduce the effective strength. By the way, I'm sitting on a good idea that I haven't finished writing up yet for how to combine the encryption key and the integrity-checking hash together so that you have only one value (perhaps of size 256 bits) instead of two values -- one for the key and one for the hash. Perhaps that would solve most of your performance issues? As I mentioned in my previous mail, I'd like to understand more about what the performance implications are in GridBackup. > What is the bare minimum data needed to retrieve, reassemble and > decrypt an immutable file? Just the AES read key? That, and some way to find the shares, which we currently call the "storage index". That would omit not only the integrity check on the ciphertext (to guarantee that the immutable cap you started with could match only one file) but also the integrity check on the shares (to identify which servers are responsible for serving up corrupted shares, in the case that the resulting file was corrupted). Regards, Zooko From shawn-tahoe at willden.org Sun Jul 12 19:56:31 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 20:56:31 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> References: <200907121845.32680.shawn-tahoe@willden.org> <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> Message-ID: <200907122056.31646.shawn-tahoe@willden.org> On Sunday 12 July 2009 07:37:12 pm Zooko Wilcox-O'Hearn wrote: > I really don't like the idea of adding an insecure mode, if we can > help it. Could you explain more -- perhaps even quantify -- what > impact the size of the cap has on your system? Hopefully there's a > solution which is both secure and efficient. I'm not suggesting an "insecure" mode; I don't think reducing the UEB hash size from 32 bytes to 16 bytes (or even smaller) affects security at all. In fact, as far as I can see the UEB hash is entirely redundant -- it's a sort of an integrity optimization that allows shares to be checked more quickly, but as far as ensuring the integrity of the file as a whole, it's redundant, because the read key is derived from the content hash, so you can use it to verify integrity. And including the whole 32 bytes is further redundant, when a truncated version would do just as well. As for the impact of the size, the smaller the URI data, the more read cap index entries I can pack into an index file in the grid-based burst trie (without making the files so big they're slow to upload and retrieve), and since those files have to get uploaded and downloaded frequently, tighter packing improves performance. Shawn. From zooko at zooko.com Sun Jul 12 20:31:28 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 21:31:28 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907122056.31646.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> <200907122056.31646.shawn-tahoe@willden.org> Message-ID: On Jul 12, 2009, at 20:56 PM, Shawn Willden wrote: > In fact, as far as I can see the UEB hash is entirely redundant -- > it's a sort of an integrity optimization that allows shares to be > checked more quickly, but as far as ensuring the integrity of the > file as a whole, it's redundant, because the read key is derived > from the content hash, so you can use it to verify integrity. Hey -- you're starting to come up with the same sort of thing that I came up with as I mentioned in my previous note -- a way to do both decryption and integrity checking on the file with only one "strong crypto worth" of bits. :-) However, you're not quite there yet, because the downloader wouldn't have the ability to verify the integrity of the individual shares, only of the final plaintext, so if the integrity check fails you wouldn't know which server(s) were serving corrupted shares. Also that scheme wouldn't allow incremental integrity checking of the file before the whole thing was downloaded. Finally, that would not work if the uploader used a randomly-chosen (not-content-hash-key) or an added-convergence secret. Still, I like the way you think... > As for the impact of the size, the smaller the URI data, the more > read cap index entries I can pack into an index file in the grid- > based burst trie (without making the files so big they're slow to > upload and retrieve), and since those files have to get uploaded > and downloaded frequently, tighter packing improves performance. Hm, this is definitely the kind of reason why I greatly value small capabilities. Can you characterize it as something like "the difference between a 24-byte immutable read cap and a 32-byte and a 64-byte is X times as many trie node fetches for an average file download"? Regards, Zooko From shawn-tahoe at willden.org Sun Jul 12 20:51:19 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 21:51:19 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> References: <200907121845.32680.shawn-tahoe@willden.org> <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> Message-ID: <200907122151.20033.shawn-tahoe@willden.org> On Sunday 12 July 2009 08:50:43 pm Zooko Wilcox-O'Hearn wrote: > By the way, I'm sitting on a good idea that I haven't finished > writing up yet for how to combine the encryption key and the > integrity-checking hash together so that you have only one value > (perhaps of size 256 bits) instead of two values -- one for the key > and one for the hash. I can see several ways to do that, and it makes a lot of sense, particularly since in the case of a content-hash-key, the key is another integrity verifier, as well as a decryption key. Not so in the case of a random key, of course. Given a content-hash-key, I think the combined key and UEB hash could be even shorter than 256 bits. 192 bits would give you strong integrity assurance (since all of the 192 bits are used for that), as well as strong encryption (128 bits). > Perhaps that would solve most of your > performance issues? As I mentioned in my previous mail, I'd like to > understand more about what the performance implications are in > GridBackup. Me too. So far, I just know that read cap index uploading is significantly slowing backups. Most of this is due to the fact that the burst trie structure consists of a large number of files, and they're mutable, which makes them slower. I chose to make them mutable because they do mutate, but perhaps it would help to make them immutable. If I could reduce the size of an index entry by 30%, I could reduce the need to update trie nodes by the same amount. For the moment I've switched to uploading the indices as a set of sorted flat files, and I just have to binary search each of them in turn when looking for a specific cap. That makes uploading fast and searching slow (somewhat; the files are cached locally, so the search is performed locally). I think after I've accumulated a few hundred thousand entries in the flat files, I'll then push them all into the trie. Bulk updates of the trie are fairly quick on a per-entry basis. In general, I view the read cap index as overhead and I'd like to minimize it. Currently, the CHK URI portion of each record consumes 60 of 78 total bytes, of which only the 16-byte AES key is really essential for retrieval. > > What is the bare minimum data needed to retrieve, reassemble and > > decrypt an immutable file? Just the AES read key? > > That, and some way to find the shares, which we currently call the > "storage index". But the SID is derived from the read key, so that's implied. > That would omit not only the integrity check on the > ciphertext (to guarantee that the immutable cap you started with > could match only one file). Well, if you regenerated the read key from the plaintext and compared with the original read key, you'd know that it matches with very high probability. The exact probability depends on the number of files in the system, but even with trillions of files, the odds of a false match are on the order of winning Powerball twice in a row. > but also the integrity check on the shares > (to identify which servers are responsible for serving up corrupted > shares, in the case that the resulting file was corrupted). If the shares have different UEBs, it tells you which of them is wrong. In the case of any sort of random corruption, you'd be able to tell that anyway. If the UEBs are the same, then the UEB contains the information to tell you which share is corrupted. Assuming the read key is the primary integrity verification mechanism (and I know that's not the assumption, but it could be, and it works), then the birthday paradox is irrelevant with respect to the UEB hash, so it only needs to be long enough to resist an attacker's attempt to produce an otherwise-valid UEB with the same abbreviated hash, i.e. a second pre-image attack against a tagged SHA-256d hash (abbreviated). Shawn. From shawn-tahoe at willden.org Sun Jul 12 21:25:42 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 22:25:42 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: References: <200907121845.32680.shawn-tahoe@willden.org> <200907122056.31646.shawn-tahoe@willden.org> Message-ID: <200907122225.43008.shawn-tahoe@willden.org> On Sunday 12 July 2009 09:31:28 pm Zooko Wilcox-O'Hearn wrote: > Can you characterize it as something like "the difference between a > 24-byte immutable read cap and a 32-byte and a 64-byte is X times as > many trie node fetches for an average file download"? Actually, it's not an issue on the download side. The trie structure ensures that only a single trie node will be downloaded (even when the local cache is empty; with a fully-populated cache, there will be no Tahoe traffic). The issue is on the upload side, and it's a little complicated to quantify, as it depends heavily on the trie parameters and the "fullness" of trie nodes, which determines the number of trie nodes that need to be updated to add a set of entries. I don't fully understand the mathematical characteristics of the burst trie yet -- the one paper I found on the structure didn't really analyze that in detail, and my intuition about node fullness has proven to be way off -- nodes seem to be on average much less full that I'd expected, meaning there are more nodes than I'd expected. If there are m nodes (including empty nodes on already-burst levels), and n entries are inserted and if none of the nodes has to "burst", then on average it will be necessary to update m(1-(1-1/m)**n) nodes. As m grows, that tends toward n. For small m, it tends toward m. What I can't do is characterize m as a function of the number of entries in the trie and the set of trie parameters (bits per level and max entries per node). I wish I could, because besides helping me quantify this issue, it would probably also let me optimize the choice of trie parameters. I guess I need to sit down and think hard about how the trie behaves as it grows. It seems like it should be easy to get a handle on... Shawn. From trac at allmydata.org Mon Jul 13 05:37:16 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 12:37:16 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.a292665351a053e79e43595821d22934@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 07:02:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 14:02:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.9ddcebc1eee9f16a9191090706b8334f@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:9 zooko]: > Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-) Writing tests for it might be challenging, since good tests would require the presence of a UPnP IGD to respond to the registration/de-registration, which would require providing a rudimentary (at least) IGD daemon in the code base just for testing. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From rufus.pollock at okfn.org Mon Jul 13 09:29:31 2009 From: rufus.pollock at okfn.org (Rufus Pollock) Date: Mon, 13 Jul 2009 17:29:31 +0100 Subject: [tahoe-dev] How does file deletion work? Message-ID: How does file deletion work? webapi.txt says: > DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME [...] > Note that this does not actually delete the file or directory that the name > points to from the tahoe grid -- it only removes the named reference from > this directory. If there are other names in this directory or in other > directories that point to the resource, then it will remain accessible > through those paths. Even if all names pointing to this object are removed > from their parent directories, then someone with possession of its read-cap > can continue to access the object through that cap. Right: so deleting from a directory means unlinking from that directory. If it is linked from elsewhere it won't be deleted and even if i unlink from all directories if there's a read-cap out there it will still stay so how do I *really* delete it for sure: > The object will only become completely unreachable once 1: there are no > reachable directories that reference it, and 2: nobody is holding a read- > or write- cap to the object. (This behavior is very similar to the way > hardlinks and anonymous files work in traditional unix filesystems). But how do I (or tahoe) know if someone is holding a read or write-cap? (And when we say someone we can't mean anything I've passed the read-cap to, we've got to mean something that's connected to the grid like another node right?) Also how quickly once all read or write-caps have gone does deletion happen (is this a GC thing?). Thanks in advance for any help. Rufus Pollock From shawn-tahoe at willden.org Mon Jul 13 11:14:12 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 13 Jul 2009 12:14:12 -0600 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: Message-ID: <200907131214.12791.shawn-tahoe@willden.org> On Monday 13 July 2009 10:29:31 am Rufus Pollock wrote: > Right: so deleting from a directory means unlinking from that > directory. If it is linked from elsewhere it won't be deleted and even > if i unlink from all directories if there's a read-cap out there it > will still stay so how do I *really* delete it for sure You can't! > But how do I (or tahoe) know if someone is holding a read or > write-cap? (And when we say someone we can't mean anything I've passed > the read-cap to, we've got to mean something that's connected to the > grid like another node right?) Tahoe can't in general know about all read or write caps held outside of Tahoe dirnodes. If you're holding an unlinked read cap which you don't want to get deleted, you have to periodically contact the servers holding the shares to "renew the lease". Without that, the servers will probably eventually reclaim the space. > Also how quickly once all read or write-caps have gone does deletion > happen (is this a GC thing?). It's a GC thing, and I'm not sure what the state of the GC implementation is. I probably should have just let someone who does know respond. Shawn. From ludo at gnu.org Mon Jul 13 12:50:50 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Mon, 13 Jul 2009 21:50:50 +0200 Subject: [tahoe-dev] How does file deletion work? References: <200907131214.12791.shawn-tahoe@willden.org> Message-ID: <873a90mkf9.fsf@gnu.org> Hi Shawn Willden writes: > On Monday 13 July 2009 10:29:31 am Rufus Pollock wrote: [...] >> But how do I (or tahoe) know if someone is holding a read or >> write-cap? (And when we say someone we can't mean anything I've passed >> the read-cap to, we've got to mean something that's connected to the >> grid like another node right?) > > Tahoe can't in general know about all read or write caps held outside of Tahoe > dirnodes. If you're holding an unlinked read cap which you don't want to get > deleted, you have to periodically contact the servers holding the shares > to "renew the lease". Without that, the servers will probably eventually > reclaim the space. Does that mean that storage servers are able to list the capabilities contained in a directory object? Thanks, Ludo'. From shawn-tahoe at willden.org Mon Jul 13 13:42:08 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 13 Jul 2009 14:42:08 -0600 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: <873a90mkf9.fsf@gnu.org> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> Message-ID: <200907131442.08602.shawn-tahoe@willden.org> On Monday 13 July 2009 01:50:50 pm Ludovic Court?s wrote: > Does that mean that storage servers are able to list the capabilities > contained in a directory object? Hehe. Hadn't thought that through quite far enough. No, the storage servers can't know what caps are in the directories. So those leases will have to be renewed periodically, too. Shawn. From trac at allmydata.org Mon Jul 13 19:01:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 02:01:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.076ef86795c023679c6e563abe565549@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: undecided => 1.5.0 Comment: Hm, I guess we should look into this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 19:02:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 02:02:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.0a3048016366d2980d709f60f3d4cd36@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Soultcer just reported the same thing on IRC. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 20:46:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 03:46:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" Message-ID: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Jonathan Ellis says that he was telling a co-worker: "There are a lot of docs on Tahoe, but there's nothing that says ''which operations are efficient''." -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:19:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:19:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maxmimum recursion depth exceeded in Tahoe2PeerSelector Message-ID: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> #758: maxmimum recursion depth exceeded in Tahoe2PeerSelector --------------------------------+------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- I just got this traceback from a node using the volunteergrid: {{{ /usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py, line 328 in _runCallbacks 326 self._runningCallbacks = True 327 try: 328 self.result = callback(self.result, *args, **kw) 329 finally: Locals callback > self args (, set([19, 20]), []) kw {} /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 384 in _got_response 382 383 # now loop 384 return self._loop() 385 Locals self /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 284 in _loop 282 self.contacted_peers.extend(self.contacted_peers2) 283 self.contacted_peers[:] = [] 284 return self._loop() 285 else: Locals self /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 284 in _loop 282 self.contacted_peers.extend(self.contacted_peers2) 283 self.contacted_peers[:] = [] 284 return self._loop() 285 else: Locals self }}} (And so forth until maximum recursion depth exceeded.) There are only 15 servers on the volunteergrid right now. The clause that is shown, around [source:src/allmydata/immutable/upload.py#L279 279 of upload.py] is for the case that all servers have been asked to hold a share, and then all servers have been asked to hold a second share, and this clause is to iterate and go on to ask them to hold yet a third-or- greater share. It appears that this loop never terminated before the recursion depth was exceeded. We have [source:src/allmydata/tahoe/test/test_upload.py at 20090625021809-4233b- 9cdbf53c54025466fea8ab97bed668cd0017b142#L483 tests of this case], but... Hey waitaminute! That code in upload.py says: {{{ elif self.contacted_peers2: # we've finished the second-or-later pass. Move all the remaining # peers back into self.contacted_peers for the next pass self.contacted_peers.extend(self.contacted_peers2) self.contacted_peers[:] = [] return self._loop() }}} That can't be right. It probably means to say: {{{ self.contacted_peers.extend(self.contacted_peers2) del self.contacted_peers2[:] }}} Why does that test catch this bug? But it is too late at night for me to be messing with such stuff. If someone in a different timezone or a different sleep schedule wants to fix the test to catch this bug while I sleep, that would be great! :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:34:17 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:34:17 -0000 Subject: [tahoe-dev] [tahoe-lafs] #651: handle MemoryError by failing quickly and loudly In-Reply-To: <037.129716934a290747156300e1e2c25ac5@allmydata.org> References: <037.129716934a290747156300e1e2c25ac5@allmydata.org> Message-ID: <046.0158885167c0586a2f0e4f06036e2b86@allmydata.org> #651: handle MemoryError by failing quickly and loudly --------------------------+------------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-mutable | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): People are reporting !UncoordinatedWriteErrors on the test grid (#748), and so I checked some of the test grid storage servers and saw that one -- tahoebs5.allmydata.com bs5c1 -- has 1.5 GB of virt, 475 MB res. That is running some very old versions of foolscap and TahoeLAFS: {{{ allmydata-tahoe: 1.3.0-r3747, foolscap: 0.3.2, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.26, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c8 }}} Hm... Yep, that one has !MemoryError in its twisted.log. I wonder if we could monkeypatch Python's !MemoryError class to do a {{{sys.exit()}}} or the like. I suppose this also suggests that we ought to be pushing this issue upstream to Python itself... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:36:25 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:36:25 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.4453153a4c189abb2be34fecf5bd8878@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Okay, so this could be caused by one or more of the following known issues: #651 (handle !MemoryError by failing quickly and loudly) (I added a note about this issue on that issue just now, #548 (mutable publish sends queries to servers that have already been asked), #547 (mapupdate(MODE_WRITE) triggers on a false boundary), #546 (mutable-file surprise shares raise inappropriate UCWE), #540 (inappropriate "uncoordinated write error" after handling a server failure). I will investigate more in the evening after work and next weekend. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:09:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:09:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #760: the command to create a server is named "create-client" Message-ID: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> #760: the command to create a server is named "create-client" ----------------------------+----------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- I just had to explain to someone on the phone that the way to create a server is to run the command 'create-client'. Maybe we should rename that command to 'create-node'? It is used for creating clients, storage servers, and web gateways, but not for creating introducers. Alternately I suppose we could split it into several commands: 'create-client', 'create-storage-server', 'create-web-gateway'? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:10:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:10:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" In-Reply-To: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> References: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> Message-ID: <046.098df38048e1a27f4794ec7a20a4ceeb@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): To close this ticket, add notes to [wiki:Performance the Performance page] about the general efficiency of all of the major operations, and link to the Performance page from [source:docs/frontends/webapi.txt the webapi.txt doc]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:11:10 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:11:10 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.67f58e10fc5c7451ad9413f1e46945bf@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Attached is the foolscap logtool incident report. The critical excerpt of that report looks like this: {{{ # 05:32:47.692 [22866716]: entering loop # 05:32:47.693 [22866717]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.693 [22866718]: we are planning to push new seqnum=#1 # 05:32:47.693 [22866719]: 10 queries still outstanding # 05:32:47.697 [22866720]: 10 shares sent # 05:32:47.744 [22866724]: _got_write_answer from qvqv7jmm # 05:32:47.744 [22866725]: entering loop # 05:32:47.745 [22866726]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.745 [22866727]: we are planning to push new seqnum=#1 # 05:32:47.745 [22866728]: 9 queries still outstanding # 05:32:47.746 [22866729]: _got_write_answer from 6fyx5u4z # 05:32:47.746 [22866730]: entering loop # 05:32:47.747 [22866731]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.747 [22866732]: we are planning to push new seqnum=#1 # 05:32:47.747 [22866733]: 8 queries still outstanding # 05:32:47.748 [22866734]: _got_write_answer from trjdor3o # 05:32:47.748 [22866735]: entering loop # 05:32:47.749 [22866736]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.749 [22866737]: we are planning to push new seqnum=#1 # 05:32:47.749 [22866738]: 7 queries still outstanding # 05:32:47.750 [22866739]: _got_write_answer from fcmlx6em # 05:32:47.751 [22866740]: entering loop # 05:32:47.751 [22866741]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.751 [22866742]: we are planning to push new seqnum=#1 # 05:32:47.752 [22866743]: 6 queries still outstanding # 05:32:47.753 [22866744]: _got_write_answer from fcmlx6em * 05:32:47.753 [22866745]: WEIRD our testv failed, so the write did not happen [INCIDENT-TRIGGER] * 05:32:51.776 [22866746]: somebody modified the share on us: shnum=5: I thought they had #1:R=7ahx, but testv reported #1:R=7ahx }}} Hm, so far I don't understand how the client was surprised by this chain of events. But, now it is time to go to work so I'll wonder about it later. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From rufus.pollock at okfn.org Tue Jul 14 09:01:00 2009 From: rufus.pollock at okfn.org (Rufus Pollock) Date: Tue, 14 Jul 2009 17:01:00 +0100 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: <200907131442.08602.shawn-tahoe@willden.org> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: Thanks for the responses. So as I understand it: 1. a node will keep (its chunks of) a file based on renewal requests. 2. If these don't come in the file will be marked for garbage collection. 3. If a file is listed in a directory then this will lead automatically to renewal of the relevant leases 4. However if it is no longer linked from any directory explicit renewal requests would have to be sent. 5. So by unlinking a file from all directories, in the absence, of any explicit renewal requests, it will be deleted from the grid. Does this sound right? Rufus 2009/7/13 Shawn Willden : > On Monday 13 July 2009 01:50:50 pm Ludovic Court?s wrote: >> Does that mean that storage servers are able to list the capabilities >> contained in a directory object? > > Hehe. ?Hadn't thought that through quite far enough. ?No, the storage servers > can't know what caps are in the directories. > > So those leases will have to be renewed periodically, too. > > ? ? ? ?Shawn. > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > -- Join our "100 supporters pledge": http://www.pledgebank.com/support-okfn Promoting Open Knowledge in a Digital Age - http://www.okfn.org/ From david at triendl.name Tue Jul 14 09:19:15 2009 From: david at triendl.name (David Triendl) Date: Tue, 14 Jul 2009 18:19:15 +0200 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: <20090714161915.GA5130@triendl.name> On Tue, Jul 14, 2009 at 05:01:00PM +0100, Rufus Pollock wrote: > 1. a node will keep (its chunks of) a file based on renewal requests. Yes, storage nodes keep their data until all leases expired (typically they expire after 30 days). > 2. If these don't come in the file will be marked for garbage collection. The storage nodes regularly perform scans of all data and all data with no leases (leases are stored together with chunk data) will be removed. > 3. If a file is listed in a directory then this will lead > automatically to renewal of the relevant leases No, the files in the directory need to have their leases renewed as well. This can simply be done with a single deep-check with &add-lease=true call, which will reach all directories and files beneath it and renew their leases at the currently reachable servers. > 4. However if it is no longer linked from any directory explicit > renewal requests would have to be sent. Explicit renawal requests always need to be sent, if garbage collection is enabled on the storage servers. > 5. So by unlinking a file from all directories, in the absence, of any > explicit renewal requests, it will be deleted from the grid. Yes, but you can also just abandon the whole directory, which will yield the same results. Cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090714/9ac0409c/attachment.pgp From ludo at gnu.org Tue Jul 14 16:27:47 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Wed, 15 Jul 2009 01:27:47 +0200 Subject: [tahoe-dev] How does file deletion work? References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090714161915.GA5130@triendl.name> Message-ID: <87eisi26bw.fsf@gnu.org> Hello, David Triendl writes: > On Tue, Jul 14, 2009 at 05:01:00PM +0100, Rufus Pollock wrote: [...] >> 5. So by unlinking a file from all directories, in the absence, of any >> explicit renewal requests, it will be deleted from the grid. > Yes, but you can also just abandon the whole directory, which will yield the > same results. Or, if you don't trust storage servers for being available, or for faithfully implementing the lease mechanism, anything may vanish at any time. :-) Thanks, Ludo'. From trac at allmydata.org Tue Jul 14 18:24:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 01:24:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #760: the command to create a server is named "create-client" In-Reply-To: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> References: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> Message-ID: <046.0feb8c36d2bb91380d823fdae7102225@allmydata.org> #760: the command to create a server is named "create-client" ----------------------------+----------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- Comment(by warner): My plan was to rename it 'create', and then add some options to specify which services you want the new node to provide. Since we want to do away with the Introducer eventually, I plan to retain 'create-introducer' as a separate command (rather than add 'tahoe create --introducer' and then get rid of it again later). The default node will have client functionality (i.e. the ability to upload and download files), a webapi server, and maybe storage functionality (I forget if we currently default to --with- storage or --no-storage). But yeah, 'create-node' has the benefits of * being longer, since creating a new client/server node is an infrequent operation and thus, according to Zipf's Law, deserves a longer name * being harder to interpret as intended for creating other things, like directories, or files I'm always on the fence as to what word to use for this client node thingy of ours. In a distributed-system sense, "node" is the right word: it's just one node out of many. If you think of things in asymmetric terms and really want to emphasize the difference between "clients" and "servers", then the node you're creating might have one or both features: sometimes you are creating a client-only node, sometimes a server-only node, and (more commonly) you are creating something that can do both. "node" has the bad quality of also being the most convenient term for elements of the file/directory DAG structure, but there we usually lean towards the longer terms "filenode" and "dirnode" (and only use the unqualified term when we need a plural: "nodes" instead of "filenodes/dirnodes" or "file/directory nodes"). So, I don't know what's the best term for the "instance of the Tahoe program (with working directory and saved state and a persistent nodeid) that you're creating for eventual launch with 'tahoe start'" thingy which would serve as the direct object in "tahoe create-XYZ". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 18:30:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 01:30:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" In-Reply-To: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> References: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> Message-ID: <046.1f2276b581311e021047dff226365936@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): Also, include a copy of these notes in the source tree somewhere. Adding docs to the wiki is great and all, but I'd like people to be able to grab a source tree and know that they don't need to get anything else. The biggest benefits (in my mind) of having docs on the wiki is that 1) everybody can edit them, add questions, etc, and 2) you can reference them from other wiki pages with nice short !WikiNames. The benefit of having docs in the source tree are 1) long-term persistence, 2) no version skew between the tahoe version you're looking at and the docs, and 3) local availability despite allmydata.org and/or trac being down. Maybe each file in docs/ could have a corresponding wiki page, and the file could have a note at the bottom saying "for the latest version of this document, please see URL". And we could write a script that would fetch all the docs and copy them into the source tree. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:34:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:34:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.14aacb083aebc19d4268203ab97bf7ea@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): Well, "It Depends". Current servers do no checking of shares. (I agree, I like server-side share-verification, except for the load and complexity it adds to the storage servers) == Immutable Files == Current code has uploaders quietly+simplisticly believing servers who say "I already have a share by that SI", without additional checking. (they don't search very far for existing shares, but if the server list is the same as it was for the first file, they'll get complete overlap). This results in the second file appearing to upload correctly but being completely unrecoverable (all shares for that SI are for the first file, so a download of the second file will get shares that fail the hash checks and fail). The uploader that we haven't written yet will probably search further for existing shares (especially once it's seen evidence of one), and will do some amount of verification of those shares (at the very least pulling down the UEB to make sure it's for the same file). This uploader will see evidence of an SI collision and probably fail the upload (with an exception that basically says "please pick a different SI, perhaps a random one"). Unless, of course, you're unlucky and wind up talking to a different servers during the two uploads. The subsequent downloader will either succeed but notice a lot of "corrupted" shares (if they can talk to the second uploader's servers) or fail due to a lot of corrupted shares (if they can't). If we further improve the servers to verify the integrity of immutable shares upon receipt, and we change the protocol to split server-selection-index from storage-index, and assume that the SI is required to be derived from the verifycap (so that servers can validate the share all the way up to the SI), then we're still in the same boat as the previous paragraph: servers who don't yet have a share will accept either the first upload or the second (since we're assuming a too-short SI, and these two files are colliding), and if the second upload succeeded due to disjoint peersets, then second-file downloaders will succeed or fail depending upon which servers they're able to talk to. == Mutable Files == We get the same issues here: if the storage-index is too small, then there will be multiple RSA keys which map to the same SI. A validating server which has accepted a share for key1 will reject an update for key2 (because it's looking at the full pubkey inside the share), but other servers might accept key2 (if they don't already have a share). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:51:05 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:51:05 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d78f331b13587674eeba6d5bd9f3cb19@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): We need someone to review Kevan's patch. Do it now and this patch can go into TahoeLAFS v1.5! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:57:27 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:57:27 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.c7a289efb8eab4563c9b529aad537e89@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Comment(by zooko): [20090715025814-92b7f-d4af644430e5daef6d6ad57cc550c8faceaeb2cf] fixes "* make the Title and/or Label of the Directory page indicate that it is part of TahoeLAFS", and "* Change the wording of the 'Attach' form", but not "* Add a hyperlink with anchor text "What is this?" next to the "mutable" check-box which explains what mutable files are and how you never want to use them. Hm, maybe this also means hide the "mutable" check-box behind some sort of expert-user-mode? :-/ " -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:58:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:58:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.c70c263d88230cb487ad5d43908cc6fc@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: eventually Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: We're not going to finish this for v1.5. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 21:09:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:09:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.0e721b68d60064598187b9878415835c@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings --------------------+------------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------+------------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Not urgent for TahoeLAFS v1.5, so I'm bumping it. Note that aside from the general "curiousness" that our code (or more likely our unit tests) rely on assertion-checking to be correct, turning on Python's {{{-OO}}} optimization mode might reduce CPU load a bit. I thought I observed that when benchmarking for #329/#752, but I'm not sure if it was real or an artifact of noisy benchmarking. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Tue Jul 14 22:29:41 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Tue, 14 Jul 2009 23:29:41 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: Message-ID: Folks: Not unexpectedly, my getting a day job and Brian going off on vacation (or "gallivanting", as I prefer to call it) has slowed down development of TahoeLAFS. v1.5 is late! I decided today that I really need to punt certain things, starting with the Macintosh packaging that I've been working on (tickets #762 and #763). Looking at the rest of The RoadMap (http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.5.0 ), most of these can be bumped from v1.5 once we've properly investigated the ones that *might* be critical dangers or regressions from v1.4.1... (Time passes as Zooko updates various tickets...) Okay, I've bumped #566 (after doing something to improve the situation), #691 (after doing something to improve it), #755. That leaves #705 (thanks, Kevan!), #748 (needs to be investigated a bit -- probably not critical and probably not a regression), #758 (should be easy to fix), #668 (ugh -- shall we punt this one?). Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/566 # host tahoe dependencies http://allmydata.org/trac/tahoe/ticket/668 # easy_install allmydata-tahoe is broken http://allmydata.org/trac/tahoe/ticket/691 # improve WUI directory page according to a new user's first impressions http://allmydata.org/trac/tahoe/ticket/705 # "tahoe mv" unlinks the target even when it is a directory http://allmydata.org/trac/tahoe/ticket/748 # Exception: when trying to create directory on testgrid http://allmydata.org/trac/tahoe/ticket/749 # Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings http://allmydata.org/trac/tahoe/ticket/755 # if there is an unrecoverable subdirectory, the web deep-check report loses other information http://allmydata.org/trac/tahoe/ticket/758 # maximum recursion depth exceeded in Tahoe2PeerSelector http://allmydata.org/trac/tahoe/ticket/762 # build of mac package fails due to pkgutil in zetuptoolz http://allmydata.org/trac/tahoe/ticket/763 # build of mac package fails due to some mysterious reason (not pkgutil) From toby.murray at comlab.ox.ac.uk Wed Jul 15 00:47:33 2009 From: toby.murray at comlab.ox.ac.uk (Toby Murray) Date: Wed, 15 Jul 2009 08:47:33 +0100 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: Message-ID: <1247644053.7331.0.camel@toby-laptop> On Tue, 2009-07-14 at 23:29 -0600, Zooko O'Whielacronx wrote: > That leaves #705 (thanks, Kevan!), #748 (needs to be investigated a > bit -- probably not critical and probably not a regression), I don't understand. How is not being able to create mutable files or directories on the testgrid not a regression nor critical? Cheers Toby From zookog at gmail.com Wed Jul 15 06:27:34 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 15 Jul 2009 07:27:34 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: <1247644053.7331.0.camel@toby-laptop> References: <1247644053.7331.0.camel@toby-laptop> Message-ID: Well, I guess it's the difference between bugs in the source code itself that we are about to ship as "v1.5" versus operational issues that effect TestGrid specifically versus operational issues that might typically effect other users. If it is not a new bug in the source code since the v1.4.1 then it wouldn't be a "regression" (people who are happily using v1.4.1 would not have a reason to refrain from upgrading to v1.5). If it is an operational issue that strikes rarely, doesn't cause great harm, and is easy to work around then it isn't "critical". (This part is, of course, a judgment call.) The investigation isn't complete, but so far it looks like the situation on the test grid where you can't create new directories is due to some combination of: 1. Limitations in the code that were already present in the v1.4.1 release (therefore not a regression): ticket #540 2. The limitation is that the testgrid web gateway (http://testgrid.allmydata.org:3567 ) is not handling misbehavior by some of the storage servers. That's a bug, but it isn't probably won't affect lots of users. It can be "worked around" by fixing your storage servers. 3. The misbehaving storage servers are running TahoeLAFS-v1.3-r3747, which is older than the current stable v1.4.1 release. It's possible (but again, without a complete investigation I don't know if it is true) that the cause of the MemoryError in the storage server has been fixed since then. So, I think the next step are: 1. Investigate more. Are any other storage servers besides tahoebs5.allmydata.com bs5c2 misbehaving? Do the munin graphs of bs5c2 show any interesting pattern in memory usage or other statistics? 2. Upgrade bs5c2 and reboot it, probably making TestGrid usable again. 3. ? Maybe experiment with adding some sort of kludge to hard-shutdown in case of MemoryError. I'm about to do #2, even though I don't want doing so to interfere with #1, and then I need to go to work. :-) Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/540 # inappropriate "uncoordinated write error" after handling a server failure From zookog at gmail.com Wed Jul 15 06:40:11 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 15 Jul 2009 07:40:11 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: <1247644053.7331.0.camel@toby-laptop> Message-ID: Aha! And it turns out that upgrading and rebooting the storage servers on tahoebs5 don't change the failure. See ticket #748. Regards, Zooko tickets mentioned in this mail: http://allmydata.org/trac/tahoe/ticket/748 # Exception: when trying to create directory on testgrid From trac at allmydata.org Tue Jul 14 21:46:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:46:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #761: "tahoe cp $DIRCAP/$PATH $LOCAL" raises AttributeError Message-ID: <037.ffdb4fcafac540e99a10cc22625d14b0@allmydata.org> #761: "tahoe cp $DIRCAP/$PATH $LOCAL" raises AttributeError -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: eventually Component: code-frontend-cli | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- {{{tahoe cp --help}}} suggests that I try: {{{ You can also use a dircap as either FROM or TO target: tahoe cp URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html ./ # copy Zooko's wiki page to a local file }}} But when I try it, I get an exception: {{{ $ time ~/playground/allmydata/tahoe/trunk-hashedformat/bin/tahoe cp -r --node-url http://testgrid.allmydata.org:3567 URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html foo Traceback (most recent call last): File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1-r3989', 'console_scripts', 'tahoe')() File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/cli.py", line 429, in cp rc = tahoe_cp.copy(options) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 759, in copy return Copier().do_copy(options) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 444, in do_copy self.try_copy() File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 501, in try_copy return self.copy_to_directory(sources, target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 661, in copy_to_directory self.copy_files_to_target(self.targetmap[target], target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 692, in copy_files_to_target self.copy_file_into(source, name, target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 737, in copy_file_into target.put_file(name, f) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 163, in put_file pathname = os.path.join(self.pathname, name) File "/usr/lib/python2.5/posixpath.py", line 60, in join if b.startswith('/'): AttributeError: 'NoneType' object has no attribute 'startswith' }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 21:49:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:49:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #566: host tahoe dependencies In-Reply-To: <037.69ed23fb905f81e8424d28c6af94c997@allmydata.org> References: <037.69ed23fb905f81e8424d28c6af94c997@allmydata.org> Message-ID: <046.e07ead24aef5c7ec4dce7850ef533a6a@allmydata.org> #566: host tahoe dependencies -------------------------+-------------------------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: assigned Priority: major | Milestone: eventually Component: packaging | Version: 1.2.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Okay I've downloaded the packages from the Tahoe test grid to http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs and [20090715025246 -92b7f-e7058e13ddc5be2d2538f46584b144674fce0045] adds that directory to the {{{find_links}}} in [source:setup.cfg]. Bumping this ticket out of v1.5 Milestone. To close this ticket, find a TahoeLAFS grid that is reliable and fast enough to be a component that builds of TahoeLAFS can rely upon, and upload all those packages to that grid, and change setup.cfg to point to it. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 22:09:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 05:09:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #762: build of mac package fails due to pkgutil in zetuptoolz Message-ID: <037.f1f9893c2d352042b94846d24d7182e1@allmydata.org> #762: build of mac package fails due to pkgutil in zetuptoolz ----------------------------+----------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.4.1 Keywords: mac setuptools | Launchpad_bug: ----------------------------+----------------------------------------------- http://allmydata.org/buildbot/builders/Mac%20OS-X/builds/1039/steps/mac_exe/logs/stdio {{{ File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/modulegraph/modulegraph.py", line 114, in find_module if isinstance(importer, pkg_resources.ImpWrapper): AttributeError: 'module' object has no attribute 'ImpWrapper' }}} This has something to do with the fact that the version of {{{zetuptoolz}}} bundled with TahoeLAFS in [source:misc/dependencies], called 'setuptools-0.6c12dev' replaced pkg_resources with the new pkgutil module, and modulegraph apparently uses pkg_resources. Why modulegraph's use thereof results in this incompatibility I haven't figured out yet. I have tested a version of zetuptoolz which uses the old pkg_resources style instead of the new pkgutil style -- http://allmydata.org/trac/zetuptoolz/changeset/20090703214517-92b7f- 126e534865d9104679dc424aa681cf26a26f2921 -- and shown that this error goes away with that zetuptoolz (this is why this failure doesn't happen on ootles the way it does on the "OS X" (!VirtualZooko) builder). So a good solution to this ticket would be to upgrade our bundled zetuptoolz to that new version of zetuptoolz. Although it would be nice to understand more precisely why it doesn't work this way -- maybe we could submit a patch to some other project to help. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 22:14:42 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 05:14:42 -0000 Subject: [tahoe-dev] [tahoe-lafs] #763: build of mac package fails due to some mysterious reason (not pkgutil) Message-ID: <037.bb3c128de40c25a67f89084a893ec7f6@allmydata.org> #763: build of mac package fails due to some mysterious reason (not pkgutil) -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.4.1 Keywords: mac | Launchpad_bug: -----------------------+---------------------------------------------------- I applied a patch to my local version of zetuptoolz on ootles to get past the problem described in #762. Now the "build a Mac package" step completes on ootles: http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/230/steps/dist/logs/stdio but the (new! improved!) "test the resulting Mac package" steps fails: http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/230/steps/testdmg/logs/stdio The "test the resulting Mac package" step could hopefully be amended to emit more diagnostics in this case... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 23:50:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 06:50:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.642238bd8fa1f70ee9a09e381944f8ef@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings ------------------------+--------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed * milestone: eventually => 1.5.0 Comment: I had some time, so I fixed it, in [3993]. All were cases where we intentionally do something wrong so as to trigger an (assertion) exception and then check the resulting behavior. All were fixed by replacing the !AssertionError with something more specific. The best rule to follow is probably this: * if an exception is worth testing, it's part of the API * !AssertionError should never be part of the API OTOH, it's a bother to create a whole new exception type and add an "if" clause (and then fret about whether the code-coverage stats are going to go down) just to do something that should be highly encouraged like adding preconditions and internal consistency checks. Maybe it means we're being overzealous in our unit tests and exercising things that we shouldn't or don't need to exercise. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:09:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:09:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.4f0bcb9bd72cf377525b9603a578d71e@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): since zooko found a !MemoryError on one of the testgrid servers, this is most likely another instance of #540. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:12:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:12:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #759: bundle newer pycryptopp into SUMO packages In-Reply-To: <037.02df60c5465488587cd253350a674c69@allmydata.org> References: <037.02df60c5465488587cd253350a674c69@allmydata.org> Message-ID: <046.0c5bf437bc93f321e040b9a3b5c7092a@allmydata.org> #759: bundle newer pycryptopp into SUMO packages ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Changes (by warner): * status: assigned => closed * resolution: => fixed Comment: Done. desert-island-test is now green. Note that we haven't actually built tarballs for a month, since one of the "Supported Builders" has gone AWOL, and the buildbot is configured to wait for all supported builders to finish a build before creating tarballs or debs. I plan to remove that coarse dependency and replace it with something smaller.. not sure what yet. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:15:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:15:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maximum recursion depth exceeded in Tahoe2PeerSelector In-Reply-To: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> References: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> Message-ID: <046.39220b9f2ef8f67bdd6861a490c0f427@allmydata.org> #758: maximum recursion depth exceeded in Tahoe2PeerSelector --------------------------------+------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by warner): Huh, yeah, that code !!!is!!! odd.. your analysis feel right, but I'm too jetlagged to understand this code right now either. I want to rewrite the uploader anyways, but that's not going to happen for 1.5. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:21:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:21:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.d7f02322b0f58b099f0017dc5bd781da@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): does anyone know what percentage of home users have functional UPnP firewalls? I seem to remember reading somebody's report on the p2p-hackers mailing list that suggested that a large fraction (half?) of the boxes which claimed to support UPnP were in fact completely broken. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 05:43:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 12:43:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.7939aa57650ee40e6cd52fd9dc9bdea6@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): http://zgp.org/pipermail/p2p-hackers/2006-June/003880.html Alex Pankratov gave that 50% number on p2p-hackers in 2006. Around the same time allmydata.com implemented UPnP in their proprietary product (the one that preceded Tahoe) and measured the success rate among allmydata.com customers. Unfortunately I don't have or remember the numbers. We should ask Jim McCoy and Rob Kinninmont if they have or remember the results. I assume that deployment and robustness of UPnP has improved since then. ;-) Because of foolscap's nice design of using multiple connection hints in parallel and requiring strong SSL authentication, a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 05:49:42 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 12:49:42 -0000 Subject: [tahoe-dev] [tahoe-lafs] #759: bundle newer pycryptopp into SUMO packages In-Reply-To: <037.02df60c5465488587cd253350a674c69@allmydata.org> References: <037.02df60c5465488587cd253350a674c69@allmydata.org> Message-ID: <046.9843dc0db86da8339fbef1209fafeeb9@allmydata.org> #759: bundle newer pycryptopp into SUMO packages ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): The reason it didn't build tarballs all that time was because one of the supported builders was red -- the 'mac-exe' builder on ootles. http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/210/steps/mac_exe/logs/stdio Please don't unliterally change that code -- that's the implementation of our Buildbot Policy http://allmydata.org/trac/tahoe/wiki/BuildbotPolicy . If you think the policy should be different, let's talk about it first. In other words, I intended for it not to build tarballs all that time, because that way people who downloaded a tarball during that month would be getting one that was produced from source code that passed all the supported builders, including the 'mac-exe' builder. If that's too long to go with no new packages, then maybe this means 'mac- exe' should be removed from Supported. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 06:38:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 13:38:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.f768e9abc4867e4cf660ec598ff3c0bb@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): I upgraded and restarted the tahoebs5 storage servers, but I still get an UCWE from testgrid.allmydata.org:3456. Attached are two incident reports: -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Wed Jul 15 08:18:07 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 15 Jul 2009 09:18:07 -0600 Subject: [tahoe-dev] buildbot status update: Green! But where is Dan's ArchLinux? In-Reply-To: <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Message-ID: Folks: All the Supported Buildbots are green! Dan: you warned us when you signed up to run a Supported buildbot that the machine would come and go every day or so. That worked out fine for a while, but now it has stayed gone for about four days. Is this normal? Regards, Zooko From trac at allmydata.org Wed Jul 15 09:49:31 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 16:49:31 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.364ab74109539f1a8bf62c4c4b501393@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings ------------------------+--------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Comment(by swillden): I'm no expert on unit testing, but I think it makes sense that assertions (preconditions, postconditions, etc.) are things that don't need to be tested. I see them not as code which must be validated for correctness, but instead as code that does correctness validation, like the unit tests. Said another way, assertions and other design-by-contract artifacts are actually tests. They're just tests that are normally run all the time rather than only when we choose to explicitly invoke a test run. If you need to write test code to test your assertions, don't you also need to write test code to test your unit tests? And test code to test the test code that tests your tests, and... (try saying that five times fast) :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:04:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:04:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.a42d3cecc5c1a8d9b21b2cc9fbe37d51@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): > I assume that deployment and robustness of UPnP has improved since then. ;-) Thanks for the reference. I admire your confidence :) > a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right? correct. Worst case, the node thinks it is externally visible but is not. Implementation-wise, we probably need some improvements in the overall !IntroducerClient behavior, since the current code assumes that it can get its location hints quickly at startup, and that they never change after that. We need something to handle changes to the location hints at runtime (including the initial determination, which usually requires running /sbin/ifconfig to compute and is thus asynchronous) and then send a new announcement to the Introducer each time. Foolscap is not entirely happy with changing location hints either: FURLs are computed once at startup (specifically at {{{tub.registerReference}}} time), and not updated afterwards, so we'd need some more work to make that behave smoothly. We should probably implement #445 (relay) and #50 (STUNT) eventually, too. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:23:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:23:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.5a8cab15163b603b347d33b610716527@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): > Thanks for the reference. I admire your confidence :) Waaataminute, by "admire" you mean "mock", don't you? As to the testing question that Shawn raised, I think we could write a fake IGD daemon in the test code, for example one which just returns a canned answer in response to the UPnP "are you there" and another canned answer in response to the UPnP "can I have an external port please?", and so on for all the UPnP IGD client functionality that we need. That is slightly complicated (especially if the UPnP IGD client we are using uses irreproducible randomness in its queries and expects the server to echo that randomness), but probably not trickier than the other "network server fakery" that is already in the TahoeLAFS unit tests. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:27:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:27:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.e7b2f61d8ef19a11694e4633420d9d68@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as {{{miniupnp}}}, right? I'm not sure about this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner-tahoe at allmydata.com Wed Jul 15 10:49:06 2009 From: warner-tahoe at allmydata.com (Brian Warner) Date: Wed, 15 Jul 2009 10:49:06 -0700 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: <20090715104906.251b900b@luther> Note that the default server configuration is conservative and does not perform any GC at all. You must explicitly enable anything which threatens data safety like GC. When you enable it, you will provide the expiration timeout, which of course determines how quickly your server's users must renew their leases. Basically, since we have a distributed filesystem (which makes reference counting expensive at best) with least-authority access semantics (which makes it impossible for the servers to count references themselves), the easiest and most reliable approach to deletion is to use leases, timers, and garbage collection. > 3. If a file is listed in a directory then this will lead > automatically to renewal of the relevant leases Nope, not without client involvement. As other folks have pointed out, servers can't read directories (since they don't hold a readcap). So clients (who *do* hold a readcap, or hold one to a parent directory, which gives them transitive access) must be responsible for performing renewals. (if you do a periodic "tahoe deep-check --add-lease ROOT:", then your statement becomes true, because your client is doing the work of finding everything reachable from that directory and renewing the leases, and merely adding/removing a file to any reachable directory is enough to start/stop renewing its lease) Explicitly renewing every single reachable file/directory from the client side is kind of a drag, as it means you (as the client) have a lot of work to do. Part of our designs on Accounting are hoping to reduce this work, but of course there's a funky tradeoff between effort (bandwidth), storage overhead, and safety. The secrets you need to renew (or cancel) a lease are generated in a way that makes them least-authority: there is a short string you could give to someone that would enable them to renew the lease on a single file for you (and not do anything else). Without any format changes (merely new code), we could build something that would let you pass a list of "lease-renewal caps" to a helper service of some sort, and have them renew your leases for you. This was the model we envisioned for the allmydata.com use case, where end users were not expected to have to do work to keep their files alive. The "tahoe manifest" command is part of this. There is also a "master lease renewal secret" which is used to compute the per-share renewal secrets: someone who has that master secret and a file's verifycap can renew the lease on its shares. One idea we've had is to mark all your leases with an account-id of some sort (perhaps just a unique number), and refresh a timer on the account-id instead of on every individual share. Basically you'd be saying "yes, I'm still alive, please keep my files alive too", and sending one message per month instead of one for every file and directory you care about. Lot less bandwidth that way. But that would only reclaim space in large chunks (or never reclaim space at all, if the user sticks around). If you're confident that you can enumerate all the files and directories that you care about, you can periodically compare this manifest against a previous version, and then send out explicit lease-cancel message for the objects that are no longer on the list. (the "cancel lease XYZ" message is the closest we've got to actual server-side deletion). But note that if you get it wrong (perhaps due to race conditions between two machines modifying a shared directory), you could accidentally lose files. Adding one lease per starting point (i.e. per root directory) per walker instance feels like it might avoid the race worries. I suspect that the best approach will be to combine this manifest-delta cancellation thing with the account-id-based timer. If clients are charged/accountable for how much space they're using, then they'll be motivated to make a reasonable-to-them tradeoff between the work done to build/compare these manifests, lease timers, and safety. Another idea (ticket #308) is to change the encryption format of dirnodes to introduce another level into the "writecap -> readcap -> verifycap" hierarchy. The new order would be "writecap -> readcap -> traversalcap -> verifycap", and a traversalcap would give somebody the ability to get the traversalcaps of all children (as well as the verifycap of the dirnode itself). Then, if you gave a hypothetical Lease Renewal Service the traversalcap of your root directory (as well as your master lease-renewing secret), they could renew all of your leases, but couldn't actually read your directories or files. (this requires "semi-private DSA keys", see http://allmydata.org/trac/pycryptopp/ticket/13 for details). You might be willing to reveal the shape of your directory structure to this service, in exchange for letting it take responsibility for your lease-renewal duties. If the service lives network-wise close to the storage servers, it may be considerably faster too. cheers, -Brian From glucnac at gmail.com Wed Jul 15 19:02:17 2009 From: glucnac at gmail.com (dan McNair) Date: Wed, 15 Jul 2009 21:02:17 -0500 (CDT) Subject: [tahoe-dev] buildbot status update: Green! But where is Dan's ArchLinux? In-Reply-To: References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Message-ID: On Wed, 15 Jul 2009, Zooko Wilcox-O'Hearn wrote: > Dan: you warned us when you signed up to run a Supported buildbot that the > machine would come and go every day or so. That worked out fine for a while, > but now it has stayed gone for about four days. Is this normal? I think I have a configuration oopsie. I recently changed how things that run on my machine all the time are started. I'll make sure the buildbut is in functioning order. Sorry! From trac at allmydata.org Wed Jul 15 15:54:52 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 22:54:52 -0000 Subject: [tahoe-dev] [tahoe-lafs] #668: easy_install allmydata-tahoe is broken In-Reply-To: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> References: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> Message-ID: <049.1d9a4ee1d92d3adac2900981580253f5@allmydata.org> #668: easy_install allmydata-tahoe is broken -----------------------+---------------------------------------------------- Reporter: francois | Owner: zooko Type: defect | Status: assigned Priority: major | Milestone: eventually Component: packaging | Version: 1.3.0 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Sigh. I hate to allow this issue to remain unfixed, but there really isn't a good reason to hold up the TahoeLAFS v1.5 release on it. The official install instructions http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know, and all of the "flunkOnFailure" building/packaging buildsteps on all of the SupportedBuilders are passing. Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does ''not'' currently work, I hereby bump this ticket out of the v1.5 Milestone. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 19:22:02 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 02:22:02 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c6ae2c1d39a71b9a8944463eb036d0b9@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Changes (by zooko): * keywords: reliability review => reliability Comment: The comment "# we should probably pick some output that is more informative, and # put that here" can be removed from the test patch. Hm, the implementation works by using {{{urllib.urlopen()}}} to send a query for the metadata ({{{?t=json}}} and then test whether the target exists and if so whether it is a directory. If that test passes then it goes ahead and does the {{{HTTP PUT}}} to overwrite the target. The problem with this is that there is a race condition, also known as a TOCTTOU ("Time Of Check To Time Of Use") issue, where the object under the target name may be non-existent or be a file at the time the check happens, but be a directory when the subsequent PUT happens. A safer implementation would extend the semantics of the PUT to tell the webapi server "except don't do it if the target turns out to be a directory". Hm, I guess 'tahoe mv' really ought to be using {{{POST /uri/$DIRCAP/[SUBDIRS../]?t=rename}}} anyway instead of {{{PUT}}}. Hm, I see that that {{{POST}}} command has an undocumented (in [source:docs/frontends/webapi.txt]) 'replace' option indicating whether it should replace if there is already a child under the target name or abort. [source:src/allmydata/web/directory.py at 20090715025814-92b7f- d4af644430e5daef6d6ad57cc550c8faceaeb2cf#L327] I guess the right implementation of this ticket is to extend that webapi command with a {{{replace=only_files}}} option which will abort if there is a child under the target name and that child is a directory. Unsetting the 'review' keyword. Kevan: what do you think? I don't want to punt this issue out of 1.5 because it is a potentially data-losing ui issue. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Wed Jul 15 21:03:06 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 15 Jul 2009 22:03:06 -0600 Subject: [tahoe-dev] simple authority taxonomy versus a kind of privacy In-Reply-To: <20090715104906.251b900b@luther> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090715104906.251b900b@luther> Message-ID: <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> Dear Brian and tahoe-dev: I was wondering about the word "traversal cap", thinking "Isn't that what one might call a 'deep verify cap'?". Then I thought "Hey wait a second, all of our directory caps are deep.". (I'm glad about this fact -- I think that all directory access rights being transitive is simpler than having both transitive and non-transitive, while being expressive enough for our needs.) Then I thought "Hm, so why don't we just make Directory Verify Caps deep?". The answer has to do with privacy in the sense of traffic-analysis-resistance, but I don't think it is a win. More on that below. Here is a set of four figures which I hope illustrate why I hold a high value for having few types of capabilities and having their meanings be mutually orthogonal and consistent: for *conceptual simplicity*. immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Verify Cap Figure 1: taxonomy of capabilities in TahoeLAFS v1.5 If we added traversal cap to directories, it would become: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Traversal Cap Verify Cap Figure 2: taxonomy of capabilities with Traversal Cap ("Traversal Cap" in this figure could also be named "Deep Verify Cap".) If instead we changed directory verify caps to always be deep, then it would be: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Deep Verify Cap Figure 3: taxonomy of capabilities with all dir caps deep If we were then to add immutable directories (ticket #607), then it would become: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap immutable directories: Deep Read Cap Deep Verify Cap mutable directories: Deep Write Cap Deep Read Cap Deep Verify Cap Figure 4: taxonomy of capabilities with immutable dirs To me, Figures 3 and 4 are conceptually simpler than Figures 1 (the current situation) or 2. Now the reason why it could be useful to have a Shallow Verify Cap -- to give someone the ability to verify the integrity of a directory without also giving them the ability to get the verify-caps of the children -- is for a kind of data-privacy. You might want to give lots of people the ability to verify the integrity of your directories without also giving them the ability to trace your directory structure -- the sizes and link structure of your directories and files. As we've recently been discussing, it might be nice for every storage server to have a verify cap to go with every share that it holds. We generally agree that "verify caps are not secret" -- everyone in the world can see everyone else's verify caps. You might not want everyone to be able to see the shape of your filesystem though! The only problem is: *they can do that anyway*. Anybody who can observe your Tahoe storage service connections (even though they are encrypted) or who operates a storage server can easily detect the exact structure of your filesystem -- which directories are linked to which other directories and files, as well as the precise size of all of the files. To defend against this sort of traffic analysis or pattern detection is somewhere between "hard" and "impossible". Our comrades over at the GNUnet project, the Freenet project, and others have been trying to develop such techniques for years (both Brian and I have contributed to such projects in the past, Brian more recently than I). Whether they're close to succeeding is not clear to me (perhaps some representative of such projects or someone whose expertise is more current than mine could speak up). But it is certain that TahoeLAFS will not offer such privacy in the next couple of releases. So on the one hand, I don't want to mush together the ability to shallow-verify and ability to deep-verify and thus preclude a user from exposing one without the other. That's why we call it "The Least-Authority Filesystem": because we believe that each authority that you can grant ought to be the least authority that you might want to grant. But on the other hand, the reason we call it "Least Authority" instead of the older term "Least Privilege" is because we recognize that the de facto ability to accomplish a goal ("authority") is what is important, not whether you have nominally been granted the official "privilege". That is: people already have the authority to deep-trace the structure of your filesystem, whether or not you officially grant them a capability to do so. There is another maxim in the capability security community (I believe it may be due to Chip Morningstar): "Never prohibit what you can't prevent.". In fact, offering separate Shallow Verify and Deep Verify Caps could endanger people instead of helping them. Imagine that you read the docs for TahoeLAFS and learn the difference between shallow- and deep- directory verify caps. Now you go out to use Tahoe or write code that uses Tahoe, and you have the idea that by choosing not to expose the deep ones you are safe from people learning about your filesystem structure. Your belief is misguided, and if you rely on your filesystem structure being private, you may suffer ill. My final argument is that we shouldn't pay a cost in conceptual complexity today for a potential future feature, if we can instead implement that feature when we later come to need it. If shallow verify caps will be useful in the future, when TahoeLAFS has grown an anonymity/privacy feature (or married a project such as GNUnet or Tor which already has one), then we can at that time implement shallow verify caps, guided by our enhanced understanding of the threat model and the defenses we will then employ. Regards, Zooko tickets mentioned in this message: http://allmydata.org/trac/tahoe/ticket/607 # DIR2:CHK tickets not mentioned in this message: http://allmydata.org/trac/tahoe/ticket/217 # DSA-based mutable files -- small URLs, fast file creation (Search in text for "shallow read-only caps".) From trac at allmydata.org Wed Jul 15 20:36:46 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:36:46 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f369936145f03f890eaab5e1aeefc819@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: undecided => 1.5.0 Comment: Hm. http://testgrid.allmydata.org:3567 currently says: {{{ Connected to 350 of 61 known storage servers }}} And its version numbers are: {{{ allmydata-tahoe: 1.4.1-r3897, foolscap: 0.4.1, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev }}} The [source:NEWS] file says that this ticket -- #653 -- has been fixed. But perhaps it hasn't? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 20:38:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:38:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.e4c0e678144806ce9c1200a1b819382d@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Terrell Russell pointed out that http://testgrid.allmydata.org:3567 currently says: {{{ Connected to 350 of 61 known storage servers }}} Perhaps this issue (ticket #748) is caused by #653. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 20:57:59 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:57:59 -0000 Subject: [tahoe-dev] [tahoe-lafs] #764: webapi/usage calls should be logged - standard reusable format (apache-style access.log) Message-ID: <039.de1dc7a65c8ca72ff48ec1250ed4724b@allmydata.org> #764: webapi/usage calls should be logged - standard reusable format (apache- style access.log) ----------------------------+----------------------------------------------- Reporter: terrell | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- This could/should be optional - but would allow storage server operators to more easily audit their usage. If in a standard, known format - the file would be parseable by a large number of other tools. zooko comments: If you turned on full foolscap logging to a file then you could probably grep out that part... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 21:08:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 04:08:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c9b6d2e22556b803a1f192b4d7bd87b1@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): Thanks for the review, and for seeing that -- I didn't even think about race conditions. The reason I didn't use the rename method you reference is {{{ This operation cannot move the child to a different directory. }}} which I took to mean that, if someone did {{{tahoe mv tahoe:dir/file1 tahoe:file1}}}, using that wouldn't work. Perhaps I'm mistaken, or misinterpreting something. I think I agree with your thoughts on the implementation. I don't suppose that any amount of client-side hacking is going to be enough to address the race condition above. Presumably the webapi would be a better place to address this. Then, what we'd want to do is * Write tests for the extension of the webapi command (in addition to the ones for {{{tahoe mv}}}, which are probably still a good idea even if we don't end up doing a lot of stuff there) * Alter the webapi command to pass the tests * Re-write {{{tahoe_mv.py}}} to use the new functionality Seem okay? If so, I'll start poking around/starting that. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 22:20:31 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 05:20:31 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.3ae3a075324bb5e8160559dd911e9b4a@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): From what I understand of the implementation of {{{POST /uri/$DIRCAP/[SUBDIRS../]?t=rename}}}, the webapi docs are right -- it's calling {{{move_child_to}}} (source:src/allmydata/dirnode.py at 20090713001320-92b7f- fe89f2baaab25c8eb3eb96c146177eb42108aeee#L515) with the {{{new_parent}}} argument set to the parent of the child being moved -- if I understand things correctly, anyway, that means that we can only use that command to move things around within a directory. I'm not sure how easy it'd be to tweak that to behave as we'd want. The {{{PUT}}} method seems like it'd be easy enough to modify. We could check for whatever replace option we wanted to correspond to the behavior we want to implement in {{{tahoe_mv.py}}} ({{{only_files}}} is fine by me), then examine {{{to_file}}} and {{{self.node}}} in the {{{DirectoryNodeHandler}}} to see if they're as they should be -- if so, we upload as we do now, and if not, we can raise an exception. I realize that I probably seem like a cheerleader for {{{PUT}}} at this point, but, given my understanding of the {{{POST}}} method, I'm not aware of any better options. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 05:36:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 12:36:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.5fb3583f0a60f8706bfffe0992d49722@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): You're right about {{{POST ?t=rename}}} not offering this functionality -- sorry I overlooked that. In the future, it might be good to extend {{{POST ?t=rename}}} to do that, but I agree with you that doing so would take longer and be more prone to error than other ways to solve this ticket right now. If you wanted to extend {{{POST ?t=rename}}}, you'd start [source:src/allmydata/web/directory.py at 20090715025814-92b7f- d4af644430e5daef6d6ad57cc550c8faceaeb2cf#L327 here in directory.py], and change it to look up the target node by name before calling {{{move_child_to}}}. Your proposal for how to extend {{{PUT}}} to have {{{replace=only_files}}} sounds right to me. You should push this extended semantics of {{{replace}}} all the way down into [source:src/allmydata/dirnode.py at 20090713001320-92b7f- fe89f2baaab25c8eb3eb96c146177eb42108aeee#L73 the Adder class]. That class currently takes a constructor argument {{{overwrite}}} which is a boolean. You could either change it to a multi-option argument (i.e. {{{overwrite}}} can be one of "yes", "no", "only_files"), or add a second argument named something like {{{preserve_directories}}} which is a boolean and add a precondition assertion that {{{not ((not overwrite) and preserve_directories)}}} (i.e. it is a mistake if someone passed {{{overwrite=False}}} and {{{preserve_directories=True}}} to the constructor of {{{Adder}}}.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 09:25:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 16:25:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.60e0b3962c190c9282ef7ba5f194aa3f@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Grr. My blog broke. [http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html My blog] is hosted on the Test Grid. That was really the last straw! I don't mind if everyone else who uses the Test Grid is inconvenienced by it being impossible to create directories, but I'm not going to leave my blog broken! So I upgraded the web gateway from: {{{ allmydata-tahoe: 1.4.1-r3897, foolscap: 0.4.1, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev }}} {{{ allmydata-tahoe: 1.4.1-r3995, foolscap: 0.4.1, pycryptopp: 0.5.15, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, sqlite: 3.4.2, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev, pysqlite: 2.3.2 }}} Now instead of {{{Connected to 350 of 61 known storage servers}}} it says {{{Connected to 14 of 24 known storage servers}}} and my blog works nicely. Also, people can create directories again. This doesn't mean that this issue is resolved for v1.5 release though. We still need to investigate. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 10:44:21 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 17:44:21 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.ff3f999d3ec653612559b689968b95e3@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by terrell): the testgrid is now behaving properly - but there is still no test case. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 11:38:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 18:38:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.5dc909435caa97f8834cc7173601d2f3@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Per #748#comment:12 -- I upgraded and rebooted the testgrid web gateway server. However, the previous version that was running already had foolscap v0.4.1, so there is apparently ''some'' issue which results in more connections than servers which is still present in that version of foolscap. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 13:49:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 20:49:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.c1c35ad930034ef5e5ff11d84c232419@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:15 zooko]: > Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as {{{miniupnp}}}, right? Makes sense to me. If it were necessary to write test cases to validate the correct functioning of every library used, you'd never have time to write any code. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From david at triendl.name Thu Jul 16 18:24:58 2009 From: david at triendl.name (David Triendl) Date: Fri, 17 Jul 2009 03:24:58 +0200 Subject: [tahoe-dev] calling all VolunteerGrid operators In-Reply-To: References: Message-ID: <20090717012458.GA8321@triendl.name> Good evening everyone, On Sat, Jul 11, 2009 at 03:50:43PM -0600, Zooko O'Whielacronx wrote: > If you are contributing a storage server to The VolunteerGrid, would > you please update the VolunteerGrid page on the wiki and describe your > server a little? For those left wondering: The url is http://allmydata.org/trac/tahoe/wiki/VolunteerGrid > Thanks. After this mail, which is going out to tahoe-dev, I'll > probably send future mails about the VolunteerGrid only to people > whose contact info is listed on that page. It would probably be easier if we had a small mailing list for volunteergrid operators/users, where we can discuss stuff like storage distributions, server outages, public nodes, experiments made, etc. Looking at the number of volunteergrid-related messages on tahoe-dev, I don't think the mailing list would be obstrusive or high-volume in any way. On a related, note, I would like to ask all volunteergrid users about how they use the volunteergrid? What kind of files did you store, what encoding did you use, what are your tahoe/volunteergrid success stories? Oh, and last but not least, I looked at Eugen Leitl's initial call for volunteers for a stable grid ( http://allmydata.org/pipermail/tahoe-dev/2009-February/001248.html ) - There are a lot of people who seemed interested but didn't add a node yet. Why not take a few minutes and join the volunteergrid? After all, everyone benefits from a broader node distribution. Cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090717/8fe30c92/attachment.pgp From zookog at gmail.com Thu Jul 16 20:20:31 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 16 Jul 2009 21:20:31 -0600 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design Message-ID: Folks: I really like the idea of "Semi-Private Keys". I invented this idea! This makes me very proud. I described the idea in lafs.pdf [1] and drew two pretty pictures (Figures 2 and 3) which show how much simpler a system of diminishing crypto-capabilities can be if it is built with semi-private keys. Apropos our recent discussion about capability taxonomy, the use of semi-private keys (or actually of semi-semi-private keys) would greatly simplify the implementation of hierarchies of diminished caps with more levels (such as if Shallow Verify caps were included in addition to Deep Write, Deep Read, and Deep Verify). In addition to being simpler, this construction would allow for an efficient off-line diminish computation. However, I really don't think we should rely on semi-private keys in our next cap design. There is no formal proof of their security, they've never been officially peer reviewed by cryptographers in a journal or conference, they are a new idea and almost all cryptographers on the planet have never heard of them. Therefore, there could be some subtle mathematical flaw in the idea which has gone unnoticed by Brian, Shawn, David-Sarah, myself, and the various other cryptographers who have looked at the idea. Also, even if the construction *is* actually safe, demanding and paranoid users such as governments and cypherpunks may refuse to rely on it out of an abundance of caution. There is another, more technical, reason not to use semi-private keys, which is that elliptic curve keys takes up twice as many bits as their security level. If our read caps contain an elliptic curve with K-bit security, then that curve has to have 2K-bits in it. Concretely, if we had a 96-bit security level (by the way I'm not yet sure that 96-bit security level is strong enough), then a read-key made up a 192-bit ecdsa semi-private key would look something like this: lafs://MR_1IvY9jxDREZmkfVyjM1eFmEqMpP8tGM9B A 128-bit security level (widely accepted) would require a 256-bit elliptic curve key, like this: lafs://MR_NHhjcZrScTqrxT73K7d4zgC4d5BzsjXYOf0JNgt2bFx And a 256-bit security level (super overkill to persuade even the most paranoid person that the crypto is not the weak point): lafs://MR_VVdUttNsfUw8eQYABYV0jGgY6hQGuoMji1LwtezvuZ4m74yhiMqjTC9v7fUOn50XzHXpggKRBpFxH3bL1jcAY3 In contrast, if we could use our current approach which doesn't rely on semi-private keys and somehow compress together the "read key" part of the cap and the "verify hash" part of the cap, then we could have 96-bit security with just one 96-bit value. However, as we recently discussed [2], short bitstrings such as hashes can be targetted by an attacker "at once". In contrast, this paper [3] seems to say that elliptic curve public keys can't be targetted en masse like that. That is, if you are given one 192-bit elliptic curve public key, it will probably cost about 2^96 to crack it. If you are given a trillion 192-bit elliptic curve public keys and you want to crack *any one* of them, it will probably *still* cost about 2^96! his conclusion is counter-intuitive to me, and I didn't really understand the math in that paper, so if anyone else wants to look at [3] and try to help us understand it that would be nice. (Note that this is the cost to crack an elliptic curve public key using the Pollard's rho method. The attack against secure hashes that we were talking about was a "brute force" attack in which you generate random ones yourself and then check if your random one collides with any of the many targets. One could do the same attack on elliptic curve public keys -- generating a random public key and then checking whether it is the same as someone else's public key. However, they are so long -- at least 192 bits -- that the 30 or 40 bits that you can shave off by generating a bilion or a trillion of them doesn't get you into range of having a chance of success.) Anyway, if the content of a cap is a secure hash output then the cap could be short, and so in order to avoid this "attack everyone at once" problem we have to make sure they aren't *too* short. Let's take our desired security level -- 96 bits -- and then add 64 bits to ensure that even if there are trillions upon trillions of targets out there (capabilities in use), the chance of success for the attacker is still negligible: 128 bit (possibly too short?): lafs://MR_4sViRyFdd9gt1WOWhBgEs6 160 bit (nice and safe): lafs://MR_S7msOFJ80xQ1WrDDiEP0M8ULxMT 256 bit (super overkill): lafs://MR_5yuGWCrBICTZEO06e0fhzYDuEqIbfKEpEBOlP239d5b Note: these caps have been formatted for human-usability by prepending them with a URL scheme (spelled here "lafs://", although other people who might be drivers of the standardization process such as Brian and Peter might prefer to spell it "tahoe://") and with "MR_" to mean "mutable file, read cap", and also the crypto bits have been base-62 encoded. Encoded in an 8-bit encoding such as ASCII or UTF-8, they now take up 32 bytes for a 128-bit cap, 37 bytes for a 160-bit cap, or 53 bytes for a 256-bit cap. When caps are not being presented to humans but instead are for computer use, such as when they are embedded inside directories, then they shouldn't be formatted like this, but should instead be basically just the pure 16, 20, 32, or 64 bytes of crypto material. So the trade-off between using semi-private keys embedded directly into the caps versus using secure hashes of traditional public keys in the caps is that the caps can be shorter for the same security level, as shown here, as long as that security level is at least 128 (or perhaps 160) bits. And, the crypto techniques we rely on can be standard. On the other hand, diminishing a capability cannot be performed as an off-line operation. E.g. if you have a read-write cap and you want to generate a read-cap, you have to find a storage server that has the "Extended Write Cap" and fetch it from there before you can generate the read-cap. (I suppose you could also cache the Extended Cap material or even include the necessary parts of it as an optional extension carried around with the cap itself!) Also, using traditional techniques instead of semi-private keys requires the format to be more complex. See lafs.pdf for details about that, but note that since I wrote that I've thought of another way to achieve "only one crypto value" using only standard crypto techniques. Note: I haven't yet published my idea for how to achieve "only one crypto value" by compressing together the read key and the verify hash. Regards, Zooko [1] http://allmydata.org/~zooko/lafs.pdf [2] http://allmydata.org/trac/tahoe/ticket/753#comment:1 [3] http://testgrid.allmydata.org:3567/file/URI:CHK:qxugzb4esi6b6clupozci7x6hu:jscs6fjjtm5jmqh3rchurubi5qmz7pxc2bf54ixt7vn64zmy5abq:3:10:305839/@@named=/multipledlog.pdf From shawn-tahoe at willden.org Thu Jul 16 21:09:04 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Thu, 16 Jul 2009 22:09:04 -0600 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design In-Reply-To: References: Message-ID: <200907162209.04414.shawn-tahoe@willden.org> On Thursday 16 July 2009 09:20:31 pm Zooko O'Whielacronx wrote: > However, I really don't think we should rely on semi-private keys in > our next cap design. There is no formal proof of their security, > they've never been officially peer reviewed by cryptographers in a > journal or conference, they are a new idea and almost all > cryptographers on the planet have never heard of them. I agree. I was only able to find a way to reduce the security by one or two bits, but I'm just a wannabe. The idea *seems* like it should be solid, but... Maybe you should publish it and then think about including it in Tahoe 3.0. > Note: I haven't yet published my idea for how to achieve "only one > crypto value" by compressing together the read key and the verify > hash. Get with it, man! It's like you have a job or something! ;-) Shawn. From trac at allmydata.org Thu Jul 16 21:51:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 04:51:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.82432138d45aceb21077a2ddfcbc56dc@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): FYI, this should have been fixed by [3872], which depended upon a feature added to foolscap-0.4.0 (i.e. foolscap itself does not fix the issue.. it required a tahoe-side change). I guess something may still be broken. We'll probably have to wait for it to become weird again and then look at the introducerclient's internal state. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 22:13:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 05:13:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maximum recursion depth exceeded in Tahoe2PeerSelector In-Reply-To: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> References: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> Message-ID: <046.7bd337269e1a5c324affafe5703df0c7@allmydata.org> #758: maximum recursion depth exceeded in Tahoe2PeerSelector ------------------------------------+--------------------------------------- Reporter: zooko | Owner: Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------------------+--------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed Comment: This should be fixed, by [3997]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 06:07:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 13:07:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.7453660edfe311c06b9254b5bab8a934@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): > I guess something may still be broken. We'll probably have to wait for it to become weird again and then look at the introducerclient's internal state. :-( I guess I shouldn't have rebooted it then. Does that mean we should go ahead and boot this issue out of v1.5 Milestone? I have an idea -- let's add an assertion in the code that the number of connected storage servers <= number of known storage servers. Perhaps even a stronger, more detailed assertion if that's feasible. Here's what evidence I can gather about the problem that was exhibited. I upgraded and restarted testgrid.allmydata.org:3567 at 2009-07-16 16:17:18.835Z (on testgrid.allmydata.org's clock). There was nothing that looked too unusual in the {{{twistd.log}}} that day. There are two incidents reported, attached, from that day: {{{incident-2009-07-16-002829-gc4xv5y.flog.bz2}}} and {{{incident-2009-07-16-002846-pypgfay.flog.bz2}}}. Here is a foolscap log-viewer web service showing each of those logfiles: http://testgrid.allmydata.org:10000/ http://testgrid.allmydata.org:10001 . I have a hard time learning what I want to know from these logfiles. What I want to know (at least at first) is mostly about temporal coincidence. For starters, I'd like to be sure that these incidents occurred before I rebooted and upgraded the server, not after. However, the timestamps, such as "# 19:19:47.489 [23537270]: UNUSUAL excessive reactor delay ({'args': (25.734021902084351,), 'format': 'excessive reactor delay (%ss)', 'incarnation': ('\xf5\x16\x1dl\xb2\xf5\x85\xf9', None), 'num': 23537270, 'time': 1247710787.4896951, 'level': 23}s)" don't tell me what day it was nor what timezone the timestamp is in. Checking the status of http://foolscap.lothar.com/trac/ticket/90 suggests that the timestamps might be in PST, which is UTC-7. If that's the case then ... No, some of the (causally) earliest events in the log are from 18:01. Perhaps they were from 2009-07-15? Argh, I give up. Please tell me how to understand the timing of events in foolscap incident report files. I updated http://foolscap.lothar.com/trac/ticket/90 to plead for fully-qualified, unambiguous timestamps. The triggering incident is "download failure", but scarier to me is that there was a 26 second reactor delay. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 07:11:49 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 14:11:49 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.ae35c1066c76d0185cdddee61ac7cd25@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Oookay, I read the relevant source code and the miscounting of number of connected storage servers was fixed between [3897] (the version that exhibited the problem) and current HEAD. However, I'm not sure if that also means that whatever caused the failures on TestGrid was also fixed. Read on. Here is the path of the code that shows how "Connected to %s of %s known storage servers" was produced at version [3897]: [source:src/allmydata/web/welcome.xhtml at 3897#L55] [source:src/allmydata/web/root.py at 3897#L240] [source:src/allmydata/introducer/client.py at 3897#L277] Here is how it is produced today: [source:src/allmydata/web/welcome.xhtml at 3997#L55] [source:src/allmydata/web/root.py at 3997#L240] [source:src/allmydata/storage_client.py at 3997#L41] The old way could potentially have lots of tuples of {{{(serverid, servicename, rref)}}} for the same serverid and servicename, if new connections were established to the same serverid but the old connection was not lost, i.e. {{{notifyOnDisconnect()}}} wasn't getting called. The new way cannot possibly have an inconsistency between the number of known storage servers and the number of connected storage servers, since both are computed from the same dict -- the known servers are all the items of the dict and the connected storage servers are the ones that have an rref. Brian: what do you think about {{{notifyOnDisconnect()}}} not getting called even while new connections to the same foolscap peer are being established? That's the only explanation I can see for the observed miscounting on the web gateway that was running allmydata-tahoe [3897] and foolscap 0.4.1. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 07:36:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 14:36:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.9fce7b965325df40b8cc3aa4c1230d93@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Hm, note that at least one of the storage servers was in a messed-up state in which it had incurred ~MemoryErrors, so it might have been failing to participate in some sort of foolscap "good-bye" negotiation, thus preventing the older connections to it from being cleaned up. This reminds of these mysterious warnings in the incident reports (I've seen a lot of this kind of message in foolscap logs): {{{ # 13:56:23.900 [867]: WEIRD Tub.connectorFinished: WEIRD, is not in [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 10:27:02 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 17:27:02 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.b49929d6f61175dc524bf69876e70c14@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): (flogtool note: until we get [http://foolscap.lothar.com/trac/ticket/90 foolscap#90] resolved, you can use "flogtool dump --verbose" to get the raw seconds-since-epoch timestamps out of the event logs, which may give you more confidence in comparing them across servers. There is not yet something similar for "flogtool web-viewer", unfortunately. Also note that incident files '''always''' store seconds-since-epoch, so any conversion to a string time format is being done by the "flogtool" program that is processing the incident file) Yeah, one of the improvements in [3931] was to use a single data structure to keep track of storage servers, instead of two separate ones. Seemed more robust. I'd have to look more closely at the old code to figure out how {{{notifyOnDisconnect}}} could be confused. It gets fired when the TCP connection is lost, so there's no requirement that the remote side participate in any shutdown-time protocol, but it does get fired through an eventual-send, so there could conceivably have been a race condition or two. (incidentally, I removed those "Tub.connectorFinished" WEIRD messages in foolscap-0.4.2, the current release.. they were causing more problems than they were worth) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 20:19:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 03:19:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c149ccbb38b9dcbd99912517d902ded5@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): Good idea with implementing the behavior in Adder. I've written some tests for Adder, and then implemented your first suggestion. If you set {{{overwrite}}} to "only_files", then it will refuse to overwrite directories. Instead of changing from {{{True}}} to "yes" and {{{False}}} to "no", I left those as they are now, since it seemed to stand less of a chance of breaking a bunch of stuff. Let me know if you want me to change this. Do we want to add tests for the new {{{overwrite}}} option to the tests for methods that use Adder (e.g.: {{{set_node}}}), or is what I have now good enough? I'll start working on the webapi stuff next, then alter {{{tahoe_mv.py}}} to use the new options. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 06:06:15 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 13:06:15 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.d64e36012f16992e4efd4eca5a89ff81@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by zooko): I would love to have this. A world map with colored pins showing where the VolunteerGrid servers are located would be very cool. Also if there were pins of a different color showing where TahoeLAFS contributors live that would make me happy too. (See [source:CREDITS the CREDITS file] for who I consider to be contributors.) However, I'm not going to prioritize setting this up, so if somebody else wants to click the "accept" button and take responsibility for this task then it is a lot more likely to get done. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 18 06:31:54 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 07:31:54 -0600 Subject: [tahoe-dev] using the volunteer grid (dogfood tasting report, continued) In-Reply-To: <1A90DE83-A23E-4682-B8A9-67CA414AE82C@zooko.com> References: <535BF38B-1D08-44A5-833D-AD7E3E113C07@zooko.com> <1A90DE83-A23E-4682-B8A9-67CA414AE82C@zooko.com> Message-ID: Folks: The VolunteerGrid seems to be working! For a while some of the music files I had stored on the VolunteerGrid were inaccessible. Today I tried again and they are back. This implies that some storage server which had some of the shares was unavailable last time, but I didn't keep a copy of the list of connected servers so I can't tell which it was. Here is the result of my deep check and repair (also attached in HTML format). Oh -- I guess the "Summary" of the directories should *not* say "Unhealthy: some versions are unrecoverable". That was the status *before* the repair. After the repair, the directories are healthy. That's a bug -- I've opened ticket #766. Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/766 # repair results Summary field says "Unhealthy" even though it is healthy after the repair, if it was unhealthy before Deep-Check-And-Repair Results for root SI=tsxqypdes5ysfme6cuqebinmdy Counters: * Objects Checked: 23 * Objects Healthy (before repair): 18 * Objects Unhealthy (before repair): 5 * Corrupt Shares (before repair): 0 * Repairs Attempted: 5 * Repairs Successful: 5 * Repairs Unsuccessful: 0 * Objects Healthy (after repair): 23 * Objects Unhealthy (after repair): 0 * Corrupt Shares (after repair): 0 Files/Directories That Had Problems: * : Unhealthy: some versions are unrecoverable * Metallica-Master of Puppets-13-of-16-fec: Unhealthy: some versions are unrecoverable * Metallica-Master of Puppets-13-of-26-fec: Unhealthy: some versions are unrecoverable * vidz: Unhealthy: some versions are unrecoverable * vidz/home_movies: Unhealthy: some versions are unrecoverable Relative Path Healthy Pre-Repair Recoverable Pre-Repair Healthy Post-Repair Storage Index Summary False True True tsxqypdes5ysfme6cuqebinmdy Unhealthy: some versions are unrecoverable 02 Master Of Puppets.flac True True True inal62aoisoa4huma5fmzq3req Healthy 03 The Thing That Should Not Be.flac True True True 4ub2wplyahagoner4npnzbshle Healthy 04 Welcome Home (Sanitarium).flac True True True jjrflgi4bpkodj5q7ygz6wz6oy Healthy 05 Disposable Heroes.flac True True True tdsby2jfikwsy67vfgoomax26e Healthy 06 Leper Messiah.flac True True True figj73pp7e7swislhr7bo3ol2u Healthy 07 Orion.flac True True True crcijgwqs7wmzd4juhcu4xpy5u Healthy 08 Damage Inc..flac True True True tydsprhi2bkdnqhzxsdzpgbjdq Healthy 2008-08-08_suarez-lnf-16x9-7142_download.mp4 True True True dwdkzbw5pl6xoos4dpqz5andti Healthy Metallica-Master of Puppets-13-of-16-fec False True True xzazzqtgytts6mp3iiv6tjrxdy Unhealthy: some versions are unrecoverable Metallica-Master of Puppets-13-of-16-fec/01 Battery.flac True True True wdbbrh2y4p77y7jpflkbucuzvy Healthy Metallica-Master of Puppets-13-of-26-fec False True True igmsz64hn6egnqu5c6e4bjmtsq Unhealthy: some versions are unrecoverable Metallica-Master of Puppets-13-of-26-fec/01 Battery.flac True True True uwgvqn372sycxncb6p3ct6rd7i Healthy Metallica-Master of Puppets-13-of-26-fec/02 Master Of Puppets.flac True True True uwqr6zia2qn2sgd2mxbwzui6ei Healthy Metallica-Master of Puppets-13-of-26-fec/03 The Thing That Should Not Be.flac True True True xollsyamtkwcg3nxhxwl4qffyq Healthy Metallica-Master of Puppets-13-of-26-fec/04 Welcome Home (Sanitarium).flac True True True 2izwujkldbas3z2oryunctseoi Healthy Metallica-Master of Puppets-13-of-26-fec/05 Disposable Heroes.flac True True True ohmxbqe2vefvltpk4x7phzgqn4 Healthy Metallica-Master of Puppets-13-of-26-fec/06 Leper Messiah.flac True True True fkt6xw5akqo4yzlo2joht7vovy Healthy Metallica-Master of Puppets-13-of-26-fec/07 Orion.flac True True True nztp537mjar6zgzkxqu5ph5yim Healthy Metallica-Master of Puppets-13-of-26-fec/08 Damage Inc..flac True True True ihe5aqjqhl3mjxhtfzxjky4f5u Healthy Metallica-Master of Puppets-4-of-6-fec True True True efqq2yffg36yvclyd5wff5wwrm Healthy vidz False True True 5mshviisrtykianham57n3gbye Unhealthy: some versions are unrecoverable vidz/home_movies False True True ggodl3dahhmp4n2d634ttzs4ie Unhealthy: some versions are unrecoverable runtime: 0.0529370307922 seconds -------------- next part -------------- An HTML attachment was scrubbed... URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20090718/62c3ce03/attachment.html From zookog at gmail.com Sat Jul 18 06:34:02 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 07:34:02 -0600 Subject: [tahoe-dev] private mailing list for VolunteerGrid Message-ID: Calling all Volunteers! I've set up a mailing list for the users and operators of the VolunteerGrid: http://allmydata.org/cgi-bin/mailman/listinfo/volunteergrid-l This is a *private* mailing list -- the list archives are available only to subscribers and my approval is required to become a subscriber. However, you are all welcome. Regards, Zooko From trac at allmydata.org Sat Jul 18 07:50:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 14:50:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.71b1ee568d8a89759069c02545916c9a@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): As I said, it's simple creating an embedded map with OSM. The only point is to get people's (and probably server's) geo-positions. I think I'll write an email to the list and ask for the data. If I get some reasonable reply, we can use that. (Putting that into the OSM database would make manegement and changing easier, though.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 08:11:32 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 15:11:32 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.4f45038beb9768140a85fa60c5259143@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by zooko): Per my comment:1 I created a user account on http://OpenStreetMap.org named "zooko" and clicked on my neighborhood of South Boulder, CO, USA. Is that sufficient to give you the information you need for my location? How can I also enter in the coordinates of things other than myself, such as my server, which is in a different location? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 09:01:47 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 16:01:47 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.891becc287eb5aa25462cb77dd64576d@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): Are you sure you ceated an account? http://www.openstreetmap.org/user/zooko tells me that "zooko" is not known... OSM is first of all a hugh database containint "interesting" stuff (roads, post boxes, houses, benches, ...). Thus you take one of the editors (eg. JOSM, Potlatch, ...) and enter known information about objects (first of all their location) into it. This is done in key/value pairs. For Tahoe, I'd probably do something like: tahoe_user=yes or tahoe_server=yes Actually, it's quite hard getting a good tag design really correct (wrt. extensibility, eg. get ir right for attaching supplied storage size afterwards etc.) For a start, it would probably nice to simply collect longitude/latitude of persons and servers and keep these in a simple CSV file or the like. OpenLayers can easily parse that. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 09:51:48 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 16:51:48 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.1f2710dc055bc6376edff1457970a49b@allmydata.org> #748: Exception: when trying to create directory on testgrid ------------------------------+--------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------------+--------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Okay, we investigated (details posted on #653), and I'm pretty that this was caused by #653, and I'm pretty sure that #653 has been fixed. Closing this as "fixed". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 10:40:58 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 17:40:58 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.c7fac9aac7160b4d26eaab73d4ff202b@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): I've now implemented a small hack to play with pin features :) http://webcontent.osm.lab.rfc822.org/tahoe/ I've used GeoRSS as a base, that is, we can have a Heading, Link and a describing text for each pin, easily. I've split it up into three groups (devs, users and servers) and would like to get your comments on it. (And I think I need help with the map extend. How can I place proper CSS there to let it extend to the rest of the screen?) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 10:44:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 17:44:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.4a8655f6db692aafee6039c9065d763f@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): If I get positive responses here, I'd announce it to the mailing list. Though I'd strongly underline that publishing your *exact* location (and that of the tahoe servers) might or might not be a good idea. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 18 13:23:09 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 18 Jul 2009 22:23:09 +0200 Subject: [tahoe-dev] private mailing list for VolunteerGrid References: Message-ID: <874ot9n3ki.fsf@gnu.org> Hello, "Zooko O'Whielacronx" writes: > This is a *private* mailing list -- the list archives are available > only to subscribers and my approval is required to become a > subscriber. Hmm, the list is private but the grid is currently public. You might want to make the grid private, then? Thanks, Ludo'. From trac at allmydata.org Sat Jul 18 13:16:47 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 20:16:47 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f9d2090af9d4bfbc743ded4a9ea7e66c@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => 1.6.0 Comment: I spoke with Brian on IRC this morning before his gallivanting schedule, and he said that it was unlikely that this problem will recur with the current versions of foolscap and of TahoeLAFS. I agree and am going to bump this ticket out of the v1.5 Milestone. Note that we can't actually close this ticket yet because there isn't a test for the original issue: duplicate entries in {{{IntroducerClient._connections}}} because ~RemoteReferences don't compare equal when they denote the same underlying TCP connection to the same peer, much less the newly discussed issue: duplicate ~RemoteReferences because (somehow) the peer is setting up new ~RemoteReferences while {{{notifyOnDisconnect()}}} is not getting called for the old ones. However, I'm fairly confident that this is (a) not a regression versus TahoeLAFS v1.4.1, and (b) not a critical bug. (In fact, it might actually be already fixed in current trunk.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Sat Jul 18 14:27:09 2009 From: warner at lothar.com (Brian Warner) Date: Sat, 18 Jul 2009 16:27:09 -0500 Subject: [tahoe-dev] simple authority taxonomy versus a kind of privacy In-Reply-To: <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090715104906.251b900b@luther> <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> Message-ID: <4A623E2D.4020805@lothar.com> Zooko Wilcox-O'Hearn wrote: > I was wondering about the word "traversal cap", thinking "Isn't that > what one might call a 'deep verify cap'?". Yeah, "deep-verify" is a reasonable term. Depending upon how we implement lease-renewal-caps and repair-caps, the "traversal cap" might be combineable with some other secret to generate those other caps for all reachable files and directories, so we might also create a "deep-repair" or "deep-renew" cap. But unless we go crazy and expand the dirnode structure to half a dozen columns, these other deep- caps will be based upon the deep-verify cap. Since directories are really just specialized ways of interpreting the contents of a file (mutable for now, we plan to add immutable in the future), every directory will also have the same caps you'd get on the underlying file object. There will always be a "shallow verify cap" for each directory (i.e. the regular verifycap for the underlying file), regardless of whether there is a good use for it or not. So I don't think it makes much sense to try to get rid of them. If you're concerned about people misusing them, just don't expose them in the interface, or don't give them a provocative name. > Now the reason why it could be useful to have a Shallow Verify Cap -- > to give someone the ability to verify the integrity of a directory > without also giving them the ability to get the verify-caps of the > children -- is for a kind of data-privacy. Lease management and file-repair are the two things I can imagine wanting to use lists of shallow verify caps for. A "manifest" is a list of shallow caps (using as weak a cap as you can get.. perhaps just a storage-index), with which you might: * compare against a previous manifest, to find out which files/directories have been added or removed, to cancel leases on the removals or add leases to the additions. * transform into a list of lease-renewal caps, to be given to a service that will keep your files alive while you and your computer are on vacation * transform into a list of repaircaps, for a similar repairer service The reason for having a deep-verify-cap is to provide (for those folks who are willing to give up this information) an easier way to delegate these repair and lease-maintenance tasks. Instead of building a whole manifest and giving thousands of repaircaps to your hired repair service (and rebuilding the list each time you change anything), you give them a single deep-repair-cap instead (which remains valid for a long time). > The only problem is: *they can do that anyway*. Anybody who can > observe your Tahoe storage service connections (even though they are > encrypted) or who operates a storage server can easily detect the > exact structure of your filesystem -- which directories are linked to > which other directories and files, as well as the precise size of all > of the files. I disagree with the first claim. A passive observer will scarcely find it "easy" to determine the shape of your filesystem. The foolscap messages and storage-index arguments are hidden by the encrypted links, so the only information available to the adversary is the number of bytes sent in each direction. If the client is only doing one thing at a time, they can probably match requests to responses, and distinguish between dirnode reads and file reads. Then their only way to distinguish directories is by the size of their packed representation, and they can only learn the size of files that you actually download completely (and they can't distinguish between multiple files of the same size). You'd have to do a recursive download of all your files, one at a time, to let them build up the shape of your directory structure this way. The adversary who is able to mount this attack must be able to observe all of your traffic to nearly all storage servers. A more active adversary (who runs a storage server) must still run several servers (something like numservers/k) to have a good chance of seeing most of your requests, and still only gets to learn anything about the files that you actually download. In addition, the information you get from a deep-verify-cap is significantly greater than what you can get from traffic analysis. With the verifycap, you get the exact file identities (even without downloading anything), which you can compare against other people's manifests, or against information you retrieve from unrelated storage servers. If convergence is enabled, knowing the storage-index lets you mount partial-information-guessing attacks against those files. Also, there are many Tahoe use-cases which involve trust relationships between clients and server operators: the servers might be more trusted than the outside world, so it may be ok that the client is more vulnerable to server behavior than to third party observers. My general feeling is this: the existence of traffic-analysis attacks (which allow adversaries in certain positions to learn "X") is not a strong argument to simply publish "X" to the entire world. The traffic-analysis attacks we've identified are not trivial to mount: why make the bad-guy's job easier? In addition, other people may find reasonable ways to address these problems in the future (someone might use PIR to retrieve shares): we'd be thwarting their noble efforts by publishing the X that they're trying to hide. That feeling may or may not be appropriate here, but in general I'm suspicious of arguments to explicitly abandon privacy properties in response to the threat of an attack. Anyways, I don't understand what sort of code change you're advocating: how could you "get rid of" a dirnode's shallow-verify cap? Let's just keep calling it "the verifycap of the underlying mutable file" instead of "shallow-verify" and not encourage the user to use it or believe they can get any special privacy-preserving properties from it. cheers, -Brian From trac at allmydata.org Sat Jul 18 14:35:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 21:35:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.835da7e0a16abc799bea6113d3b46f3c@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): hm, I do hope we'll be able to close this ticket eventually, without reverting back to the old code, analyzing the problem, fixing it, then discarding the fix to go back to the new code :-). Let's consider closing this as INVALID after a couple of months of correct operation of the testgrid. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 16:57:10 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 23:57:10 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f253ac18fa3ddbb529fcd58c9767abeb@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): The only part that I'm not sure about is the way that multiple {{{RemoteReferences}}} piled up. I.e., I'm confident that there is not a bug in TahoeLAFS with regard to this issue, but I'm not confident that there isn't a bug in foolscap about this issue. Note that we didn't change the version of foolscap on the testgrid webapi (foolscap 0.4.1). Maybe if I write down what I know about it in this ticket that will help me understand it better. Could you summarize what was changed which made me think (in comment:4) that foolscap v0.4.0 fixed the ''original'' issue of having multiple {{{RemoteReferences}}} over the same TCP connection to the same peer? Whatever it was, it didn't fix the problem that was demonstrated using TahoeLAFS [3897] and foolscap 0.4.1 in which there were many more tuples of {{{(serverid, servicetype, rref)}}} than there were servers. This suggests that foolscap 0.4.1 must have been creating new {{{RemoteReferences}}} and not calling back the {{{notifyOnDisconnect()}}} method to clean up old {{{RemoteReferences}}} to the same peers. Now, TahoeLAFS in version [3997] no longer keeps tuples of {{{(peerid, servicetype, rref)}}}, instead it keeps a dict mapping from {{{peerid}}} to {{{NativeStorageClientDescriptor}}}, where a {{{NativeStorageClientDescriptor}}} has at most one {{{RemoteReference}}}. When foolscap calls {{{NativeStorageClientDescriptor._got_connection()}}}, then the {{{NativeStorageClientDescriptor}}} eventually sets {{{self.rref}}} to reference the new {{{RemoteReference}}} (see [source:src/allmydata/storage_client.py at 20090717050709-66853-80eda37caf0df376be79f45cbed728999b68a843#L218 storage_client.py line 218].) If its {{{self.rref}}} already had a reference to a {{{RemoteReference}}} then it would be overwritten by the new one, so the problem of lots of redundant {{{RemoteReference}}}s piling up certainly can't happen in TahoeLAFS anymore. But, if foolscap is generating redundant {{{RemoteReference}}}s passing them to TahoeLAFS, could this cause other problems? For example, I see that {{{NativeStorageClientDescriptor}}} calls {{{rref.notifyOnDisconnect(self._lost)}}}. If foolscap is generating redundant {{{rref}}}s, then maybe the {{{self._lost}}} will eventually be called by one of the old ones that is no longer referenced by the {{{NativeStorageClientDescriptor}}}, and if so then the current {{{rref}}} will get spuriously removed? Maybe we should add some sanity checking to {{{NativeStorageClientDescriptor}}} such as asserting that when its {{{_got_connection()}}} is called that its {{{self.rref}}} is required to be currently {{{None}}}, or asserting that when its {{{self._lost()}}} method is called that the current value of {{{self.rref}}} is the same as it was when the {{{notifyOnDisconnect()}}} was called? In the interests of a stable v1.5, maybe we could make it so that if these checks fail it does something "safe", such as if {{{self._lost()}}} is called when {{{self.rref}}} is not equal to the same thing that it was when the {{{notifyOnDisconnect()}}} was called, that it leaves the {{{self.rref}}} alone (as well as logging a high-scariness incident. What do you think? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 18 20:27:26 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 21:27:26 -0600 Subject: [tahoe-dev] new logo? Message-ID: Hey folks: What do you think? http://allmydata.org/~zooko/TahoeLAFS-logo.png http://allmydata.org/~zooko/TahoeLAFS-logo.svg Like in http://allmydata.org/~zooko/network-and-reliance-topology.png , the red node represents the things that have access to the crypto keys and the black nodes represent things that don't. Regards, Zooko From ludo at gnu.org Sun Jul 19 05:57:07 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sun, 19 Jul 2009 14:57:07 +0200 Subject: [tahoe-dev] new logo? References: Message-ID: <87eiscsue4.fsf@gnu.org> Hi Zooko, "Zooko O'Whielacronx" writes: > http://allmydata.org/~zooko/TahoeLAFS-logo.png > http://allmydata.org/~zooko/TahoeLAFS-logo.svg Do you really want to make "LAFS" part of the name? IMO it clutters the logo and the name. (Other than that, while I agree it's important to emphasize that Tahoe follows the POLA, I'm a bit skeptical about the phrase "LAFS": it's a hint to those who know what POLA is, but it doesn't tell much to those who don't, I think.) Thanks, Ludo'. From kpreid at mac.com Sun Jul 19 06:17:46 2009 From: kpreid at mac.com (Kevin Reid) Date: Sun, 19 Jul 2009 09:17:46 -0400 Subject: [tahoe-dev] new logo? In-Reply-To: References: Message-ID: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> On Jul 18, 2009, at 23:27, Zooko O'Whielacronx wrote: > Hey folks: > > What do you think? > > http://allmydata.org/~zooko/TahoeLAFS-logo.png > http://allmydata.org/~zooko/TahoeLAFS-logo.svg > > Like in http://allmydata.org/~zooko/network-and-reliance-topology.png > , the red node represents the things that have access to the crypto > keys and the black nodes represent things that don't. I think it is lacking in visual coherence: the text has nothing to do with the image, and the image has no particular geometry. I also agree that "LAFS" perhaps shouldn't be part of the prominent name. How about this (made in a hurry, to be refined later): http://switchb.org/kpreid/2009/tahoe/logo-1-o.png http://switchb.org/kpreid/2009/tahoe/logo-1-o.graffle (I'll provide a SVG version if desired; as I don't have a SVG editor this was faster.) -- Kevin Reid From michael at michael.ie Sun Jul 19 08:06:21 2009 From: michael at michael.ie (Michael Walsh) Date: Sun, 19 Jul 2009 16:06:21 +0100 Subject: [tahoe-dev] new logo? In-Reply-To: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: On Sun, Jul 19, 2009 at 2:17 PM, Kevin Reid wrote: > How about this (made in a hurry, to be refined later): > > http://switchb.org/kpreid/2009/tahoe/logo-1-o.png > http://switchb.org/kpreid/2009/tahoe/logo-1-o.graffle Ah, now that actually looks like a logo and is pretty much there IMO. From zooko at zooko.com Sun Jul 19 13:04:10 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 19 Jul 2009 14:04:10 -0600 Subject: [tahoe-dev] discussion of TahoeLAFS crypto on the cryptography mailing list Message-ID: On the cryptography mailing list there was some discussion about the fact at 112-bit elliptic curve key was recently brute-forced, so I described some of how TahoeLAFS is about to switch to (probably maybe?) 192-bit elliptic curve from RSA: http://www.mail-archive.com/cryptography at metzdowd.com/msg10652.html http://www.mail-archive.com/cryptography at metzdowd.com/msg10653.html Regards, Zooko From zooko at zooko.com Sun Jul 19 13:33:57 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 19 Jul 2009 14:33:57 -0600 Subject: [tahoe-dev] new logo? In-Reply-To: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: I like your design! I'd like to mess with it and post a variant -- could I please have an .svg version? As for naming, we have a major problem that "Tahoe" is not searchable/ unique enough to serve as the official name. Try your favorite search engine and search for "Tahoe". I just browsed through the first eighteen pages of the google search results for "Tahoe" and I finally found something that wasn't the lake in California/Nevada: it was this open source software project: http://tahoe.ca.sandia.gov . Anyway, while I'm happy for people to contribute their creative ideas about the logo, I'm not quite as happy to hear new creative ideas about the name. Anyway, I like your logo. --Zooko From sam at samason.me.uk Sun Jul 19 13:49:33 2009 From: sam at samason.me.uk (Sam Mason) Date: Sun, 19 Jul 2009 21:49:33 +0100 Subject: [tahoe-dev] new logo? In-Reply-To: References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: <20090719204933.GJ5407@samason.me.uk> On Sun, Jul 19, 2009 at 02:33:57PM -0600, Zooko Wilcox-O'Hearn wrote: > As for naming, we have a major problem that "Tahoe" is not searchable/ > unique enough to serve as the official name. Try your favorite > search engine and search for "Tahoe". I just browsed through the > first eighteen pages of the google search results for "Tahoe" and I > finally found something that wasn't the lake in California/Nevada: it > was this open source software project: http://tahoe.ca.sandia.gov . Really--for me, it comes up ninth from google.com and sixth from google.co.uk! I'm somewhat surprised that it's coming up as high as it does as the root page doesn't have any text for google to index. -- Sam http://samason.me.uk/ From kpreid at mac.com Sun Jul 19 14:32:47 2009 From: kpreid at mac.com (Kevin Reid) Date: Sun, 19 Jul 2009 17:32:47 -0400 Subject: [tahoe-dev] new logo? In-Reply-To: References: