From trac at allmydata.org Wed Jul 1 00:27:46 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 07:27:46 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.2c3e7ffd8fa97ec059773ceadf77e3d8@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I've missed the actual instruction from my previous post: {{{ vex x86->IR: unhandled instruction bytes: 0x66 0xF 0x6F 0x1 }}} Which maps to: {{{ MOVDQA 66 0F 6F /r xmm1, xmm2/m128 Move aligned double quadword from xmm2/m128 to xmm1 }}} Looks like valgrind doesnt like SSE2 stuff? -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 00:56:17 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 07:56:17 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.c68eb9b80c2e9eec637a2a7ff23c0f8c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Aha! Now i built it without SSE2 support (adding CRYPTOPP_DISABLE_SSE2) and the test passes (and valgrind too): {{{ ==26122== Memcheck, a memory error detector. ==26122== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al. ==26122== Using LibVEX rev 1884, a library for dynamic binary translation. ==26122== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP. ==26122== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation framework. ==26122== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al. ==26122== For more details, rerun with: -v ==26122== running darcsver running test running egg_info writing requirements to pycryptopp.egg-info/requires.txt writing pycryptopp.egg-info/PKG-INFO writing top-level names to pycryptopp.egg-info/top_level.txt writing dependency_links to pycryptopp.egg-info/dependency_links.txt writing manifest file 'pycryptopp.egg-info/SOURCES.txt' running build_ext copying build/lib.linux-i686-2.5/pycryptopp/_pycryptopp.so -> pycryptopp test_chunksize (pycryptopp.test.test_sha256.SHA256) ... ok ---------------------------------------------------------------------- Ran 1 test in 10.929s OK ==26122== ==26122== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 3736 from 9) ==26122== malloc/free: in use at exit: 4,983,817 bytes in 906 blocks. ==26122== malloc/free: 325,282 allocs, 324,376 frees, 71,999,912 bytes allocated. ==26122== For counts of detected errors, rerun with: -v ==26122== searching for pointers to 906 not-freed blocks. ==26122== checked 5,115,828 bytes. ==26122== ==26122== LEAK SUMMARY: ==26122== definitely lost: 0 bytes in 0 blocks. ==26122== possibly lost: 66,612 bytes in 181 blocks. ==26122== still reachable: 4,917,205 bytes in 725 blocks. ==26122== suppressed: 0 bytes in 0 blocks. ==26122== Rerun with --leak-check=full to see details of leaked memory. }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 01:43:12 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 08:43:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.83adce40d919b1bd21d2bcd7d39dbcfb@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): And even more fun: Using an SSE2-less build of pycryptopp makes tahoe pass all the tests without crashes/hangs (http://allmydata.org/trac/tahoe/ticket/737): {{{ Ran 599 tests in 395.987s PASSED (skips=3, expectedFailures=2, unexpectedSuccesses=7, successes=587) }}} So i guess there is a bug in either Crypto++ SSE code or my CPU SSE implementation that somehow also makes python's time() return NaNs or segfault. FWIW this system CPU is: {{{ processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 10 model name : AMD Athlon(tm) XP 2600+ stepping : 0 cpu MHz : 1916.490 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow up }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 02:10:58 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 09:10:58 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.d597553ffabd32659fd2cf5e85e284b5@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Now it hit me - AMD AthlonXP doesn't even support SSE2, only the base SSE - why is MOVDQA even being used? and why doesn't it outright crash with an invalid opcode when not in valgrind? Also for completeness tested with --disable-embedded-cryptopp - I'm getting the same erros from valgring/failed test/hanging tahoe test. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 06:26:41 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 13:26:41 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.ae22458ad79e8c269ed7223b90ee0d77@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): When a standard Crypto++ source tree is built (i.e., not the special embedded build in pycryptopp) then it produces a file named {{{cryptest.exe}}} (even on Unix). If you run {{{./cryptest.exe v}}} then it will execute the Crypto++ self-tests. (If you run {{{./cryptest.exe b}}} then it will execute the benchmarks.) I don't know yet whether the self-tests exercise the same paths of hashing, as Brian described in the original issue report: "two updates, in which the total number of bytes is a multiple of 128, and in which the first write is an odd number less than 64, will cause this bug." -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 06:57:46 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 13:57:46 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): Wei Dai added optimized assembly for SHA-256 for amd64 in the Crypto++ v5.6.0 release. Perhaps a bug was added in that change. Isn't MOVDQA an SSE instruction (not requiring SSE2)? -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 08:49:47 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 15:49:47 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.f82eb11c4ef240ed6af4440c79217d62@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): According to http://en.wikipedia.org/wiki/X86_instruction_listings - MOVDQA belongs to SSE2. However, according to http://mubench.sourceforge.net/results.html - "this cpu does not support SSE2, but appears to run some SSE2 opcodes that have corresponding MMX opcodes without error". -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 16:53:29 2009 From: trac at allmydata.org (pycryptopp) Date: Wed, 01 Jul 2009 23:53:29 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.657e5e78c888f4a820ba065eba370833@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I've downloaded and built crypto++ from source, all the tests succeed in cryptest.exe -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Wed Jul 1 19:39:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 02:39:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.14a3ab86fd863103692555ef7455a11a@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): So, summarizing discussions with zooko and warner... * We first need to handle testing for backwards compatibility. This means that we'll want to capture a base32-encoded representation of the output of _pack_contents as it sits now when run against some small, known directory structure. We'll then use that to test _unpack_contents (to make sure that the unpacked directory structure is what we're expecting), and _pack contents (to make sure that packing the same known directory structure results in the same packed representation). * Then we can do the optimizations that this ticket suggests. (that didn't really need to be a list, huh.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 19:51:12 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 02:51:12 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.f24683f987bf118a2b8b59219fb8d3a1@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): I've started work on the first point by writing a little program to try to get the base32 representation of the output of _pack_contents. Unfortunately, I run into path issues when I try to run it. At the top of my script, I have {{{ import sys base = "/path-to-tahoe/tahoe/support/lib/python2.6/site-packages" sys.path.append(base) }}} to tell the Python interpreter about tahoe's support directory. The goal was to have a somewhat less robust version of the detection that the tahoe executable does when it first starts (since I know where the directory is on my system, and don't imagine that the script is going to be run elsewhere, since it's kind of a one-time deal to make tests work). When I run the script, and try to import something from the tests directory, I get {{{ Traceback (most recent call last): File "get_dirnode_strings.py", line 7, in from test.test_dirnode import FakeClient File "/path-to-tahoe/tahoe/src/allmydata/test/__init__.py", line 2, in from foolscap.logging.incident import IncidentQualifier ImportError: No module named foolscap.logging.incident }}} which would seem to suggest that my path hacking doesn't do what I want it to. Is there something easy that I'm missing that would make this work? (also, my approach for getting this information is based on what of tahoe's code I've read, which isn't much. if there's a more obvious way to get the strings that we need, or if my program is flawed in some other way, please let me know) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 20:15:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 03:15:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.08e98101386c9d89e22d34678edded96@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): The way to do that is to set the {{{PYTHONPATH}}} environment variable before invoking the {{{python}}} interpreter. There are some hacks (mostly due to setuptools, I think) that need to run at interpreter startup time and learn about the available Python packages, so adding to your {{{sys.path}}} after interpreter startup won't work. It was a good idea though. If you can't or don't want to set your {{{PYTHONPATH}}}, then you'll need to find the {{{foolscap}}} package, probably in {{{/path-to- tahoe/tahoe/support/lib/python2.6/site- packages/foolscap-0.4.2-py2.6.egg}}}, and add that whole path (including the {{{thing.egg}}} part) to your {{{sys.path}}}, and then do likewise with the other packages. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 21:28:27 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 04:28:27 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.967b100497482f651da290da3f3d08b5@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): That fixed the issue -- thanks. Just so I'm clear on the next part: Currently, {{{_unpack_contents}}} returns a dictionary mapping the name of each child node to a pair {{{(child node instance, metadata)}}}, where metadata is a dictionary of metadata. {{{_pack_contents}}} gets the {{{rocap}}} and {{{rwcap}}} (if present) from the child node instance. What we want to do is * Modify {{{_unpack_contents}}} to return a dict mapping the name of each child node to a triple {{{(child node instance, metadata, serialized)}}}, where child node instance and metadata are as they are now, and serialized is the serialized from of the other data. * Modify {{{_pack_contents}}} to deal with this new representation; specifically, to use {{{serialized}}} if present instead of serializing the other data, which we think is expensive. * Modify {{{Adder}}}, {{{MetadataSetter}}} and {{{Deleter}}} to work with these changes; specifically, they should, when modifying a node, set the corresponding {{{serialized}}} value to None, which will tell {{{_pack_contents}}} that it needs to re-serialize that node. * Write tests to make sure that all of this works (probably before doing the first 3 items). In particular, * If we pass contents packed with an unmodified {{{_pack_contents}}} to the new {{{_unpack_contents}}}, we should get the same underlying directory structure back (i.e.: all expected child node names are present, and metadata values are as they should be. Is there anything else we'd want to check?) * The output of the new {{{_pack_contents}}} should be the same as the output from the old one, given the same directory structure. * We should be able to interact with {{{Adder}}}, {{{Deleter}}}, and {{{MetadataSetter}}} in the same way that we do now. Are there already tests to adequately verify this, or should I plan on writing those, too? (apologies if this is pedantic, but I'm still getting the hang of tahoe, and don't want to go off on an irrelevant tangent because I've made a bad assumption somewhere) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 1 21:39:17 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 04:39:17 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.0e129bcc467803e21b66338dd266559d@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): That sounds good to me! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Thu Jul 2 07:20:06 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 2 Jul 2009 08:20:06 -0600 Subject: [tahoe-dev] v1.5 status Message-ID: Folks: To track the imminent v1.5 release, see The Roadmap: http://allmydata.org/trac/tahoe/roadmap and particularly the list of open tickets in the 1.5 Milestone: http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.5.0 If you can help with some of these, please do! :-) I have a 3-day weekend coming up (tomorrow, Friday, off for USA Independence Day), so I hope to make some progress or even completely release v1.5 this weekend. Regards, Zooko From trac at allmydata.org Thu Jul 2 11:27:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 18:27:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.da82c521f420a10c6e91b67acab7777f@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): I usually write code like this by adding a new unit test. You can run a specific unit test with a minimum of fuss (and startup time and extraneous noise) by doing e.g. {{{make quicktest TEST=allmydata.test.test_dirnode.DeepStats.test_stats}}} . That will get all the PYTHONPATH stuff set up for you. Anything you {{{print}}} from the unit test will get displayed, so I use this for one- off tools all the time. Also, you can use {{{misc/run-with-pythonpath.py}}}, which will set up the right environment and then run the command of your choice. For example, if you wrote a {{{foo.py}}} to do the stuff you just described, then you could invoke {{{python misc/run-with-pythonpath.py python foo.py}}}. I'm disappointed that the sys.path technique you tried didn't work.. it used to. Your approach to {{{_pack_contents}}} sounds great! Note that the {{{serialized}}} form should contain the whole {{{name+rocap+encrwcap+metadata}}} string (i.e. be sure to include the name). That way, if {{{_pack_contents}}} sees that it has a pre-serialized string available, it can just append that to the list of entries that it's building, and doesn't need to re-serialize the childname either. At the end of that loop, it should do a single {{{"".join(entries)}}} to perform the final assembly (rather than doing incremental {{{+=}}} operations, which would be a lot slower). > If we pass contents packed with an unmodified _pack_contents to the new _unpack_contents, we should get the same underlying directory structure back (i.e.: all expected child node names are present, and metadata values are as they should be. Is there anything else we'd want to check?) We should make sure that the rocap/rwcap is the same too. Doing child.get_uri() and comparing it against a constant is plenty. > We should be able to interact with Adder, Deleter, and MetadataSetter in the same way that we do now. Are there already tests to adequately verify this, or should I plan on writing those, too? test_dirnode.py should already have test coverage for the modifiers. It exercises all the {{{NewDirectoryNode}}} methods, like {{{set_uri}}} and {{{list}}}. If they pass, then you've updated the modifier classes successfully. Also, if it works on your platform (it's somewhat touchy right now), use our test-coverage tools: {{{make quicktest-figleaf figleaf-output}}}, and check to see that all the code you've touched is actually being run. The buildbot has a link to the current coverage data (generated under py2.4; it seems that py2.5 gives slightly different answers). I think it should be pretty easy to make these changes and achieve the same or better coverage than before. Also, I don't know if it'd be worth it, but you could get fancy and instead of changing the data structure from a list of two-tuples to a list of three-tuples, you could: * create a subclass of {{{list}}} * have it store an auxilliary value (the unmodified serialized string) for each key * override {{{__setitem__}}} to remove that auxilliary value * add a {{{get_both_items}}} method, which takes a key and returns a two-tuple of (aux-serialized, value) (which is really (aux-serialized, (child,metadata))) * add a {{{set_both_items}}} method which takes (key, aux-serialized, (child,metadata)) * then the Modifier classes like {{{Adder}}} and {{{Deleter}}} don't have to change: they'll set the entries that have changed and leave the rest alone * now {{{_pack_contents}}} uses {{{get_both_items}}} and prefers the pre- serialized form if it's still there Since {{{NewDirectoryNode.list}}} returns whatever {{{_unpack_contents}}} returns, you might manage to write less code if you use this technique. Otherwise you'll need to find all the callers of {{{_read}}} and update them to tolerate the new three-tuple (and add a wrapper to {{{list}}} to convert it into two-tuple form for external callers, so that it continues to honor the same interface defined in {{{allmydata.interfaces.IDirectoryNode}}}). Oh, and of course, it would be a good idea to actually measure the time that serialization/deserialization takes before doing any of this work. Create 10/100/1k/10k/100k entries at random, call {{{_pack_contents}}} under {{{timeit.py}}} or some other sort of how-long-does-it-take loop, and figure out an {{{A+Bx}}} curve (I'd expect the serialization time to be some constant A plus some per-entry time B, let's figure out what A and B are). Let's make sure that serialization is the culprit, it might be that call to {{{_create_node}}} in unpack that we should focus on. And let's set a reasonable goal.. maybe we should be able to unpack+modify+pack 10k entries in less than a second, or something. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 11:34:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 18:34:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.d82e373ae33bdcb541d7287ea61c05e6@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): The measuring part is my job -- #327 (performance measurement of directories). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 13:45:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 20:45:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.f9180504d3048799a3abc0223ffca6fb@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: [4-tea-2] Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by zooko): Oh, when I wrote comment:7, I was thinking that Python >= 2.6 comes with sqlite, but it is actually Python >= 2.5. [4-tea-2], would you please run the following and post the results? {{{ PYTHONPATH=./support/lib/python2.5/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' }}} ? For example, here are the results on my machine: {{{ $ PYTHONPATH=./support/lib/python2.4/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' [allmydata-tahoe 1.4.1-r3914 (/home/zooko/playground/allmydata/tahoe/trunk/src), pysqlite 2.5.5 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/pysqlite-2.5.5-py2.4-solaris-2.11-i86pc.egg), Nevow 0.9.32 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/Nevow-0.9.32-py2.4.egg), foolscap 0.4.1 (/usr/lib/python2.4/site- packages/foolscap-0.4.1-py2.4.egg), Twisted 8.2.0 (/usr/lib/python2.4 /site-packages/Twisted-8.2.0-py2.4-solaris-2.11-i86pc.egg), zope.interface 3.5.1 (/usr/lib/python2.4/site- packages/zope.interface-3.5.1-py2.4-solaris-2.11-i86pc.egg), simplejson 2.0.7 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4 /site-packages/simplejson-2.0.7-py2.4-solaris-2.11-i86pc.egg), zfec 1.4.4 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/zfec-1.4.4-py2.4-solaris-2.11-i86pc.egg), pycryptopp 0.5.4 (/usr/lib/python2.4/site- packages/pycryptopp-0.5.4-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), pyOpenSSL 0.8 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/pyOpenSSL-0.8-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), pyutil 1.3.26 (/usr/lib/python2.4 /site-packages/pyutil-1.3.26-py2.4.egg), argparse 0.8.0 (/usr/lib/python2.4/site-packages/argparse-0.8.0-py2.4.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site- packages/setuptools-0.6c12dev.egg), zbase32 1.1.0 (/usr/lib/python2.4 /site-packages/zbase32-1.1.0-py2.4.egg)] }}} Then, please look at the {{{_auto_deps.py}}} file which is in the {{{allmydata-tahoe}}} package. For example, on my machine it is: {{{ $ grep -A1 -B1 -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata/_auto_deps.py # Sqlite comes built into Python >= 2.5, and is provided by the "pysqlite" # distribution for Python 2.4. -- if sys.version_info < (2, 5): # pysqlite v2.0.5 was shipped in Ubuntu 6.06 LTS "dapper" and Nexenta NCP 1. install_requires.append("pysqlite >= 2.0.5") }}} The {{{_auto_deps.py}}} ought to be the same as the current trunk version: [source:_auto_deps.py]. Finally, please look at the {{{requires.txt}}} file in the {{{.egg- info}}}, like this: {{{ $ grep -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata_tahoe.egg- info/requires.txt pysqlite >= 2.0.5 }}} This shows that on my machine {{{pysqlite}}} appears in the {{{requires.txt}}}, because the .egg-info was built using Python 2.4. If you built with Python 2.5 then {{{pysqlite}}} should not appear in your {{{.egg-info}}}. You could try rm'rf the whole {{{allmydata_tahoe.egg- info}}} directory and rebuilding. Thanks! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 14:38:33 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 21:38:33 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.cbeaf0fd2b138d4931ca04faf753b360@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Changes (by zooko): * owner: [4-tea-2] => warner Comment: Ooh, I get it. [4-tea-2] didn't build the {{{.egg-info}}} himself, it came with the .deb that we built. So, we have to decide whether we want our .deb to depend on {{{python-pysqlite}}}, or whether we want it to not completely support Python 2.4 (or we could even build different .deb's for each Python version). I'm okay with either of those two options. (Not so much with the Python-version-specific debs option.) I guess that the build machine where the Debian packages are being created is using Python 2.4, so {{{pysqlite}}} is being added to the {{{.egg- info}}}'s {{{requires.txt}}}. Assigning to Brian to tell me what our policy is about .debs and pysqlite. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 14:53:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 02 Jul 2009 21:53:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.2cd9af12ac0e38a7a8349689d1958fd2@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): Sigh. So, building separate .debs for py2.4/py2.5/py2.6 is painful (nobody does it anymore, now that debian-python-policy generates .pycs at install time). And I don't believe the .deb control-file "Depends:" format isn't rich enough to express (py>=2.5 or pysqlite). On the other hand, the "standard" version of python is well-known per- platform (it's 2.4 in the older ones, 2.5 in the newer ones, and I guess Ubuntu Karmic will probably have 2.6). So we could say that when we build a .deb on a platform with /usr/bin/python=2.4, we declare the dependency upon pysqlite, and when we build it on a py2.5-using platform, we don't. To implement this, we should update the misc/*/debian/control files to declare the pysqlite dependency on the older platforms, and leave the newer ones alone (not declaring it). This also means that we should probably stop using the feisty rules for all platforms.. we at least need to identify the release for which python switched to 2.5 and use a newer set of rules for the 2.5-using set. I'll try to look at this today, but I'm not sure I'll get the time. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 18:25:39 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 01:25:39 -0000 Subject: [tahoe-dev] [tahoe-lafs] #683: handle arbitrary URIs in directories In-Reply-To: <038.2582f5dad5cb978edfed1f357bfc6e52@allmydata.org> References: <038.2582f5dad5cb978edfed1f357bfc6e52@allmydata.org> Message-ID: <047.1ccc7c8ba20c048bd9fa86981af1db8e@allmydata.org> #683: handle arbitrary URIs in directories ---------------------------+------------------------------------------------ Reporter: kpreid | Type: defect Status: new | Priority: major Milestone: undecided | Component: code-dirnodes Version: 1.3.0 | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): [3960] changes the way dirnodes are processed to tolerate unrecognized URIs. This should make tahoe-1.5 able to survive new formats that come from the future (i.e. if a 1.5 client tries to read or modify a directory which has new-format entries which were placed there by some 1.6-or-beyond version). It's at least a start. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 19:23:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 02:23:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.1ed7db4171fc132c4f72f7b0021bc38b@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): Ok, I think I understand this better now. * debian/etch ships with python2.4 as the standard /usr/bin/python (http://packages.debian.org/etch/python). * our most recent tahoe .deb for etch (http://allmydata.org/debian/dists/etch/tahoe/binary-i386/allmydata- tahoe_1.4.1-r3916_all.deb) was built on an Etch system with python2.4 * therefore its {{{_auto_deps.py}}} correctly declares a requirement on pysqlite, enforced at runtime, via the call to {{{require_auto_deps()}}} in {{{src/allmydata/__init__.py}}} * the .deb's .egg-info/requires.txt correctly includes a line which requires {{{pysqlite >= 2.0.5}}}, which will probably be enforced by anything that goes poking around .egg-info . In particular, the code that was added to src/allmydata/scripts/runner.py (the main CLI entry point) to call {{{pkg_requires.require('allmydata-tahoe')}}} will read the .egg-info data and enforce its requirements. * however, the packaging (i.e. debian/control) does not declare this requirement. This is a bug. Installing just this .deb (and no extra non-Depends-required packages) and then running the supplied /usr/bin/tahoe with python2.4 should fail with the same sort of exception displayed, because {{{_auto_deps.py}}} insists on having pysqlite, but APT did not know that python-pysqlite2 should have been installed. * But, 4-tea-2 appears to be using python2.5 to start the Tahoe node on this etch system, because the traceback listed above shows {{{/usr/lib/python2.5/site-packages}}}. Etch includes a python2.5 package (http://packages.debian.org/etch/python2.5), but it does not touch /usr/bin/python, so either 4-tea-2 has set up a symlink or installed python2.5 outside the awareness of APT (perhaps directly to /usr/local/bin). This is what was confusing me. * the .deb provides an allmydata_tahoe-1.4.1_r3916-py2.4.egg-info/ directory, and does not provide any .egg-info data for python2.5 . I don't know how setuptools reacts to this: would it refuse to believe that tahoe was actually installed when run under py2.5? If so, then I'm still confused as to where 4-tea-2's tahoe was finding the .egg-info data. * when run under python2.5, the code in {{{_auto_deps.py}}} should *not* require pysqlite * however the .egg-info line (generated on our Etch buildslave) still requires it, and thus the scripts/runner.py call will demand it. Why is that {{{pkg_resources.require('allmydata-tahoe')}}} line there? If it weren't, then the correct runtime-computed dependencies in _auto_deps.py would be enforced, and the incorrect package-build-time dependencies in .egg-info/requires.txt would not. Now, it happens that the .deb packaging for this is incorrect, and the Etch .deb should declare a dependency on python-sqlite2 because that's needed by the default version of python on an Etch system, and it happens that fixing the packaging would also happen to make this problem go away. But it's still wrong: this is a dependency that changes depending upon the version of Python used to run bin/tahoe, and recording the value that was computed at package-build time seems inappropriate. It should be possible to run tahoe under py2.5 on this system without having pysqlite installed. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 19:29:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 02:29:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.049fe36022d8d9bfa0710dcb477daff3@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by zooko): I guess a quicker solution is just to have the {{{allmydata-tahoe}}} .deb always depend on {{{python-sqlite}}}. While we're at it, I can swap the order of attempted-import in [source:src/allmydata/__init__.py] and [source:src/allmydata/scripts/backupdb.py], so that whatever {{{python- sqlite}}} package that is installed is the one that actually gets used (in preference to the {{{sqlite}}} module that comes in the Python standard library). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 20:29:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 03:29:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.2fbe75c17f968a8f126390e1d12f00d0@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by zooko): Brian has implemented much of this, but it isn't commented-in, documented, etc. in the current trunk. The next step (after the Tahoe v1.5.0 release) is to experiment with use cases and UI. Anybody out there want to actually use this for something? Shawn Willden has mentioned that he wants to make sure his mom's PC gets {{{K}}} shares. Jake Appelbaum has mentioned that he wants to make sure non-Tor-hidden-server nodes get {{{N-K}}} shares. Some corporations have mentioned that they would like to send approximately the same number of shares to each of {{{Q}}} different geographically distributed data centers. I think we need some brave user like Shawn or Jake to try this feature out and complain about how it doesn't do what he wants. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 21:26:03 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 04:26:03 -0000 Subject: [tahoe-dev] [tahoe-lafs] #708: graceful handling of capabilities in a format from the future that you can't understand In-Reply-To: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> References: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> Message-ID: <046.4ebceb7e4d279aa03cb1adae2dcce551@allmydata.org> #708: graceful handling of capabilities in a format from the future that you can't understand -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Resolution: fixed | Keywords: forward-compatibility Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Over on [comment:5:ticket:683 ticket #683, comment 5], Brian wrote: > [3960] changes the way dirnodes are processed to tolerate unrecognized URIs. This should make tahoe-1.5 able to survive new formats that come from the future (i.e. if a 1.5 client tries to read or modify a directory which has new-format entries which were placed there by some 1.6-or-beyond version). It's at least a start. As I understand it, the fact that we can't add unknown caps into a directory means that we can't copy directories which contain caps from the future. (If we do copy such a directory then the entries in it which had new-style caps will be omitted from the newly created copy of the directory). In theory it should be possible to do that safely just by copying the write-cap field from the entry in the source dir into the write-cap field of the newly created entry in the target dir, and likewise copying the read-cap. Brian: would there be any danger in that approach? I don't know how important it would be for clients from the past to be able to copy your new-style caps. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 21:57:21 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 04:57:21 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.cfd527d59088cddae35e9cfbd3c11aa6@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by swillden): Even before the ability to ensure that specific servers get {{{k}}} shares, I want to be able to specify that the local node get ''no'' shares. For backup purposes, you're probably going to use a local (often on the machine being backed up!) Tahoe node as the interface to the grid, and it makes little sense to store a share there, at least if the primary purpose of your backups is disaster recovery. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 22:07:23 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 05:07:23 -0000 Subject: [tahoe-dev] [tahoe-lafs] #422: have buildslaves automatically build debian packages of foolscap, zfec, pycryptopp, etc. In-Reply-To: <037.be7b09b76050f4a640d70974371002b3@allmydata.org> References: <037.be7b09b76050f4a640d70974371002b3@allmydata.org> Message-ID: <046.d7cf98978ad8b1234ea381769b5fc3f7@allmydata.org> #422: have buildslaves automatically build debian packages of foolscap, zfec, pycryptopp, etc. -------------------------+-------------------------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.0.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by zooko): [http://allmydata.org/buildbot-pycryptopp/waterfall pycryptopp buildbot] and [http://allmydata.org/buildbot-zfec/waterfall zfec buildbot] are now automatically building .deb's. Those .deb's are not currently being uploaded to anywhere and integrated into an apt repository, though. Also zfec .deb's depend on other .deb's (pyutil and argparse) which are not yet available. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 2 22:13:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 05:13:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.5f7fcaf3af9ecba367de10baa4913267@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Changes (by zooko): * owner: => zooko * status: new => assigned * milestone: undecided => 1.5.0 Comment: I've implemented most of this, but not yet the part about: * Add a hyperlink with anchor text "What is this?" next to the "mutable" check-box which explains what mutable files are and how you never want to use them. Hm, maybe this also means hide the "mutable" check-box behind some sort of expert-user-mode? :-/ I haven't committed it to trunk yet, but probably will before the v1.5 release. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Thu Jul 2 23:41:39 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Fri, 3 Jul 2009 00:41:39 -0600 Subject: [tahoe-dev] GridBackup functional (barely) In-Reply-To: <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> <063.632f971041d0c508e5991f3404ad7cb2@allmydata.org> Message-ID: <200907030041.39696.shawn-tahoe@willden.org> After far too long, GridBackup works, for a suitable definition of "works". Specifically, it scans the file system, produces and stores the backup logs (full and incremental) containing file metadata, and generates a queue of upload jobs to back up the actual file content. It also backs up the file content, and produces the read cap index entries that links the backup logs to the content. When doing backups, it packs small files together to help speed up the upload process. Very small files are actually stored in the read cap index directly, and tiny files (including empty files) are stored in the backup log. It also prioritizes the uploads, favoring recently-modified files, small files and user files. The favoritism for recently-modified files in particular tends to mean that the stuff you're actually working on gets uploaded very early. Both the backup scanning and upload processes are completely interruptable and restartable. The initial full scan is slow (gotta hash your whole drive, basically), but subsequent incremental scans are pretty fast. Uploading is slow, but that's pretty much unavoidable (though I do have some plans to improve it a little via parallelism). I am actually going to begin using GridBackup for my daily backups tomorrow, after I get an allmydata.com account and figure out how to point my local Tahoe node at it. In a couple of months I plan to set up a friendnet and switch to using that. What doesn't work yet is: 1. Difference-based backups. All of the infrastructure is in place to do them, and adding that functionality should be nearly trivial, but I want to focus on testing the basic backup functionality for a while. 2. Restores. Yeah, this is a rather important bit :-). It's my next major task. It also shouldn't be tremendously difficult, though, since it will use all of the code used to create the logs, read cap index, etc., to parse them. 3. Cross-platform support. All development and testing has been done on Linux, and while there is some stuff in place to handle Windows and OS X issues, testing, debugging and, I'm sure, fixing is required. 4. GUI. Right now the UI consists of two scripts (GridBackup and GridUpload) and a config file (~/.GridBackup/config.ini). The scripts don't even have any command line options; you have to configure everything through the config file. Somewhere in there I'm also going to do a major code restructuring. I've learned a lot about how Python code should be put together so the current codebase is an inconsistent mishmash of approaches. Other items that are even further away are: slick platform-appropriate installers that install and configure both GridBackup and Tahoe; integration of GridBackup and GridUpload into a single, daemonable (twistd) process that uses twisted to achieve some level of upload parallelism to improve performance; and other file system scanners, including one for OS X that uses fsevents, and a cross-platform scanner that operates remotely, to enable a 'backup server'. Anyway, I have lots more work to do, but the code actually successfully makes backups now. Anyone who is interested can grab the current code from my github-hosted repo, at: http://github.com/divegeek/GridBackup If you decide to play with it, you'll almost certainly run into problems. Feel free to ask questions here, or mail me directly. I'll try to actually pay a little attention to my IRC client which hangs out on #tahoe, too. Shawn. From trac at allmydata.org Thu Jul 2 23:32:55 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 06:32:55 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a647a6dd114542370e41c70cd1e727ac@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): It would be interesting if it were flakey hardware, since the same problem happens pretty uniformly across otherwise stable and independent machines that have similarities: they are x86, pretty old CPUs, etc. (Like, one is crazy rock-solid stable up for 600+ super busy days machinery, so solid I run a strata 2 ntp on it.) I downloaded and built crypto++ manually, and running the tests as bdew did I found 100% success so long as I kept the /dev/random entropy pool fed. (Else it would fail because it would wait too long for random data to appear.) I will attempt these other things that bdew seems so facile with. :) -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From shawn-tahoe at willden.org Fri Jul 3 07:18:49 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Fri, 3 Jul 2009 08:18:49 -0600 Subject: [tahoe-dev] Client/helper RAM usage Message-ID: <200907030818.49575.shawn-tahoe@willden.org> Are files being processed (FEC-encoded, etc.) loaded into RAM? Is it possible to upload a 2 GiB file on a machine with only 512 MiB RAM? Is there any way to estimate the memory consumption that will be required? I'm thinking about putting a client or maybe a helper on a Linux Virtual Server that I have, but it has only 512 MiB of RAM, and that is a hard figure. There is no swap. Shawn. From trac at allmydata.org Fri Jul 3 09:21:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 16:21:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #708: graceful handling of capabilities in a format from the future that you can't understand In-Reply-To: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> References: <037.4039f2b8e6dc0268025ef8a2bbc6e859@allmydata.org> Message-ID: <046.51d70b736d3358965b036e89b856ad5e@allmydata.org> #708: graceful handling of capabilities in a format from the future that you can't understand -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: forward-compatibility Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by warner): * component: code-encoding => code-dirnodes Comment: The internal 'move' method does just that, and the JSON representation of a directory includes all the information we have about the unknown object (i.e. both the writecap and the readcap). What I don't know is how the CLI-side "tahoe cp" works, specifically if the put-lots-of-caps-at-once dirnode webapi operation will accept the same "unknown cap" structure that the JSON representation hands down. Also, I wanted to discourage people from adding new unknown caps to a directory (because they might just be adding complete junk, or a typo, or a blank string, and it'd be nice to detect that early), so the current code is liberal in what it accepts from the backend, but strict in what it accepts from the frontend, and this might prevent the frontend-based tools from doing this sort of copy. So yes, I think that approach would be safe, and it might already work. (of course we have no way to tell if the unknown-cap is a file or a directory, or something even more exotic, so we might be creating a hardlink to a mutable directoryish-thing when the rest of the copy operation was trying to make a deepcopy of individual files). The test would need to go in test_cli.py where it tests the "tahoe cp" operation. grep around the test suite for !UnknownNode, you have to be a bit sneaky to get the cap-from-the-future into a directory to start with. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 09:24:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 16:24:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #745: Debian package: Missing dependency on Debian/etch In-Reply-To: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> References: <041.10c6073ec9b58684f2e1a9b1d45f0639@allmydata.org> Message-ID: <050.4656b18e8fb136c0a29393b4676f4337@allmydata.org> #745: Debian package: Missing dependency on Debian/etch ------------------------------+--------------------------------------------- Reporter: [4-tea-2] | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian,packaging | Launchpad_bug: ------------------------------+--------------------------------------------- Comment(by warner): I updated the debian packaging rules last night, and now the py2.4-based distributions (etch and edgy, I believe) add a python-sqlite2 dependency, and the more recent ones do not. I'd prefer to not force py2.5-based platforms to install an extra package. And I'd prefer that we use the stdlib {{{sqlite3}}} module in preference to an external {{{pysqlite2}}} module. Think about it this way: when py2.4 is long dead, do you still want us to be requiring people to install a third-party module to duplicate stdlib functionality? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 12:44:30 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 19:44:30 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.616b702ff82b2832be0c56e601e07d01@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): I have written a test program for crypto++ which demonstrates the same problem for a SHA256: the results differ based on whether the input is split up on an odd boundary or not. I have compiled my crypto++ library with and (I believe) without the optimized assembly routines. I will add my program to this ticket. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 12:51:28 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 19:51:28 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a31a76be36a536d1731865be6f5ca33c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): Replying to [comment:17 midnightmagic]: > I have written a test program for crypto++ which demonstrates the same problem for a SHA256: > the results differ based on whether the input is split up on an odd boundary or not. I have > compiled my crypto++ library with and (I believe) without the optimized assembly routines. I lied. Manually disabling the optimized assembly makes my sample program output the correct results. Here are the details: {{{ g++ -DCRYPTOPP_DISABLE_X86ASM -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3 -pipe -I. -c mytest2.cpp && g++ -o mytest2 mytest2.o -L. -lcryptopp && ./mytest2 }}} will work and results in: {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E }}} Just build crypto++ and then plunk mytest2.cpp in the same directory as the libcryptopp.a was built in, and run the above. {{{ g++ -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3 -pipe -I. -c mytest2.cpp && g++ -o mytest2 mytest2.o -L. -lcryptopp && ./mytest2 }}} will NOT work, and results in: {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: D159F4747C64249633829F8355DC84678A67CCCE92ABC8F770D33111AFB43E42 STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (etc) SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 13:15:12 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 20:15:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.bf5403de5324d1d822b8664ec82a8488@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): Way to go, MM (and BD)! I will report this to the Crypto++ maintainers. In the meantime, we should probably think about disabling optimized assembly (at least on some platforms?) so that we can go ahead with the Tahoe-1.5 release. Do you have a platform were the optimized assembly *does* work so you can see how much different it makes in performance? Thanks. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 13:22:33 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 20:22:33 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.933e95cb7bf3195ccba198a692cecebe@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by zooko): I posted to Crypto++ list: http://groups.google.com/group/cryptopp- users/browse_thread/thread/d3fb86b84a4793b7 I wouldn't be surprised if Wei Dai (author of Crypto++) can come up with a fix for this fast enough to go into Tahoe-LAFS v1.5.0. But in case he doesn't, Midnight Magic is writing a patch for pycryptopp's setup.py to turn off optimized assembly on some platforms. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 15:07:00 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 22:07:00 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.e2bfc308d52604665222aa1fbec77e6c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): I posted there too. Anyway, I've got a diff that allows pycryptopp to succeed with the chunktests Werner built. It's relatively specific, and it seems unfair. The older machinery is that which would most benefit from assembly optimizations. Anyway.. I'll attach the diff in just a moment. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 15:18:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 22:18:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #573: Allow client to control which storage servers receive shares In-Reply-To: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> References: <040.5f645d96d4e373e8e06fc47997621b6e@allmydata.org> Message-ID: <049.0d9e82740575998bee30bcf1a601413c@allmydata.org> #573: Allow client to control which storage servers receive shares --------------------------------+------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-peerselection | Version: 1.2.0 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by midnightmagic): My use-case is to specify which servers receive the lion's share of the blocks, for two reasons: 1. To ensure that known-reliable machines receive the most blocks. 2. To ensure that the space is used in a way I want to control manually. For example, even though one of my machines in my home storage grid has multiple TB of relatively reliable storage space on it, I would prefer to use that as sparingly as possible and instead dump most of them on machines that are allocated specifically to the task of '''being''' a Tahoe storage node. Also, out of those Tahoe nodes that are participating, I happen to know that machines X, Y, and Z are extremely reliable, whereas others have had flakey drive bus issues in the past. I want X, Y, and Z to definitely get blocks.. and the rest of them, maybe. So I'll be using this feature for sure. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:01:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 23:01:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.98a2f326e3b37c8fc702c77c78211217@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): SUCCESS! Rebuilding pycryptopp without ASM optimizations makes it pass the chunked SHA256 test, and setting PYTHONPATH to that top-level directory makes "tahoe ls" Just Work on an existing grid, and the failing command NOW WORKS PERFECTLY. So there is a patch in pycryptopp/ticket/24 for setup.py which detects the platform involved and turns off assembly optimizations in just that specific platform and bit-width (32bit). I would say that, if bdew could do the same and it works, we can put a platform detection for his also and likely close all these tickets out until crypto++ fixes their CPU features detection. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:06:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 03 Jul 2009 23:06:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.4035bb4464404d4550053918341b6ad1@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Update: since the fix mentioned in #738 (disabling crypto++ assembly optimizations) was implemented in my local copy here, python2.5 setup.py test NOW COMPLETES THROUGH THE SPOT WHERE IT FROZE BEFORE!! So this specific issue I think, if those other ones can be closed, can be closed also once we verify that the same fix might work for bdew. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 16:43:49 2009 From: trac at allmydata.org (pycryptopp) Date: Fri, 03 Jul 2009 23:43:49 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.86ee9d6f867e6aa499cc2044a3b151b4@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): It looks like this might be specific to AMD XP, which I don't have access to. I can't reproduce it on my AMD 64 machine. Can someone give me a shell account on an AMD XP machine so I can debug this? If so, please use http://www.ibiblio.org/weidai/mailme.php to email the login info to me. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 17:12:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 00:12:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.8c78605340b609db28c9977e1c06421b@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): Okay, I've got tests in place -- I'll attach them in a bit. One minor nit: I was working on the assumption that {{{_pack_contents}}} and {{{_unpack_contents}}} were inverses -- i.e., if I do {{{_pack_contents(_unpack_contents(string))}}}, I should get {{{string}}} back. I made this a condition in my test function, but it failed -- {{{_pack_contents}}} would return a different string, even when fed with the unmodified dict returned from {{{_unpack_contents}}}. But all of the data inside the dict seem the same -- at least the rocaps, rwcaps, and metadata. Is this normal? Also, the hardcoding of the base32-encoded representation of that directory tree is kind of horrific -- any ideas for a better way of doing that? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 17:12:56 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 00:12:56 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.c011d5435c295a51ff105e9dd2762105@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): Never mind, I reproduced it on my non-SSE2 Celeron processor. I'm working on a fix. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 18:25:12 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 01:25:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.d9295b0d82bbb508950e84028aa805e6@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by midnightmagic): Replying to [comment:23 weidai]: > Never mind, I reproduced it on my non-SSE2 Celeron processor. I'm working on a fix. Perhaps not so coincidentally, a non-SSE2 Celeron is running on one of the other machines where this problem is also visible. (Amazing how fast you got on top of this.. thanks eh! I appreciate you helping me breathe more life into what might otherwise be my computer graveyard..) -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 20:05:58 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 03:05:58 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.9cbbb85fb8fd3c9adb82238742ba8677@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by weidai): I've attached a fix. Please try it out. I'll explain the bug on the Crypto++ list later if anyone is curious. I'll also add some additional testing code to the next release of Crypto++ to catch this type of bugs. -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Fri Jul 3 20:14:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 03:14:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.22e809842c35e2b8b2e0476e82904e6c@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): [source:src/allmydata/dirnode.py#L201] The reason is that the encryption of the write-cap uses a random IV. There isn't an easy way to make that deterministic and still secure right now, so we can't rely on it for testing. (Hm, for future reference -- i.e. after Tahoe v1.5.0 -- maybe we ''could'' generate the random IV as the secure hash of the write cap itself. That would be make it deterministic without, as far as I can currently see, losing security.) As to the base32-encoded directory tree, that looks okay to me. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 20:58:46 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 03:58:46 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.81bb933202d9647205c57f74085366ad@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): Okay. Given that, do you think that the other tests in {{{test_unpack_and_pack_behavior()}}} are going to be enough? I've finished doing a few things. * I've changed the tests for {{{_pack_contents}}} and {{{_unpack_contents}}} in {{{test_dirnode.py}}} to not check the output of {{{_pack_contents}}} for equality, since that will always fail at the moment. I've also moved a misplaced comment. * I've added tests for {{{CachingDict}}}, the data structure described by warner above (cool idea, by the way). * I've added an implementation of {{{CachingDict}}} to {{{dirnode.py}}}. It basically does what warner suggested that it do. * I've changed {{{_pack_contents}}}, {{{_unpack_contents}}}, and {{{_create}}} to use {{{CachingDict}}}s instead of {{{dict}}}s. With these done (tests.txt, dict.txt, optimizations.txt applied), {{{python setup.py test}}} passes on my machine. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 3 22:53:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 05:53:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.393d376365e151302ae966d6e669a513@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): It turns out that the logic for the first point was already in {{{tahoe_mv.py}}} -- just not the behavior itself. From a cursory look at the webapi docs, appending something to a directory is as simple as sticking the name of the child after the name of the directory in the URL -- so if we want to add {{{tahoe:test.pdf}}} to {{{tahoe:testdir/}}}, we can make {{{ http://127.0.0.1:3456/uri//testdir/test.pdf }}} unless I'm misunderstanding something. That's basically what I did in this patch. I haven't looked into implementing the second part yet. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 00:37:41 2009 From: trac at allmydata.org (pycryptopp) Date: Sat, 04 Jul 2009 07:37:41 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.a7c615311fd882385ffa9af95e3fc0f2@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): I'm getting a compile error with this patch: (gcc 4.3.3-12, binutils 2.19.1) {{{ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict- prototypes -fPIC -I. -I/usr/include/python2.5 -c cryptopp/sha.cpp -o build/temp.linux-i686-2.5/cryptopp/sha.o -w cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cryptopp/sha.cpp: Assembler messages: cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' cryptopp/sha.cpp:435: Error: ambiguous operand size for `dec' error: command 'gcc' failed with exit status 1 }}} -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Sat Jul 4 08:24:03 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 15:24:03 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d193d555dd3973788582d1e3af4a265e@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): I looked at your patch and it looks right to me. I think I would change the {{{if path.endswith("/"):}}} to {{{if to_url.endswith("/"):}}}. Also the patch description is not "Make mv move files into directories, instead of overrwriting directories.", because this patch doesn't (yet) achieve that in the case that the target directory name doesn't have a trailing slash. We need a test to go along with this patch -- perhaps just add some statements to {{{src/allmydata/test/test_system.py}}} to move a file into a directory-with-trailing-slash and assert that it works. Err, wait a second -- there is already a test of this in [source:src/allmydata/test/test_system.py#L973]. It says: {{{ d1.addCallback(self.log, "mv 'P/sekret data' P/personal/") d1.addCallback(lambda res: home.move_child_to(u"sekrit data", personal)) }}} This test currently passes, even without your patch that looks for a trailing slash and appends the filename to the target URL. Why is that? We should have a test that fails before we apply a patch to make the test pass. For the next part of the issue, we should add a test that tries to mv a file to a directory, without trailing slash. According to Kevin Reid's suggestion, Tahoe should refuse to do anything in that case, e.g. if 'sekret data' is a file and 'personal' is a directory, then {{{tahoe mv "sekret data" personal}}} should emit an error message and make no change. The test should make sure that tahoe emits a suitable error message, in addition to making sure that it doesn't move or remove the "sekret data" file. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 4 11:09:57 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 04 Jul 2009 20:09:57 +0200 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? Message-ID: <87hbxs7216.fsf@gnu.org> Hello! I noticed that http://allmydata.org/source/tahoe/releases/allmydata-tahoe-1.4.1.zip contains a file `allmydata-tahoe-1.4.1/src/allmydata/web/tahoe.css' that's only 7-line long, whereas the Debian package for Etch has `/usr/share/pycentral/allmydata-tahoe/site-packages/allmydata/web/tahoe.css', which is 136-line long. Am I missing something? Thanks, Ludo'. From zooko at zooko.com Sat Jul 4 14:16:29 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sat, 4 Jul 2009 15:16:29 -0600 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? In-Reply-To: <87hbxs7216.fsf@gnu.org> References: <87hbxs7216.fsf@gnu.org> Message-ID: <129A1BD4-0439-4E49-AD1A-5F83A2378343@zooko.com> Hello, Ludovic Court?s! Kevin Reid contributed an improved tahoe.css since the Tahoe v1.4.1 release. If the .deb for Etch that you are using is more recent than v1.4.1 then perhaps this explains it -- you have Kevin's new CSS file. Here's how to tell: is the web interface incredibly ugly and klunky? Then you're using the old CSS file! ;-) Oh, actually here is a better way to tell: if the version you are running is equal to or newer than v1.4.1-r3881 then you have the new CSS. Regards, Zooko From trac at allmydata.org Sat Jul 4 14:05:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:05:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #668: easy_install allmydata-tahoe is broken In-Reply-To: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> References: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> Message-ID: <049.b40fe6076cbee785fb62ca9701741d18@allmydata.org> #668: easy_install allmydata-tahoe is broken -----------------------+---------------------------------------------------- Reporter: francois | Owner: zooko Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.3.0 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by ndurner): From IRC: {{{ (20:58:05) ndurner: Hmpf, my build instructions for Windows don't work anymore (22:42:48) zooko: I think your report should go into ticket #668. }}} {{{ E:\allmydata-tahoe-1.4.0>python setup.py build Not found: tahoe-deps Not found: ../tahoe-deps Traceback (most recent call last): File "setup.py", line 346, in zip_safe=False, # We prefer unzipped for easier access. File "C:\Python25\lib\distutils\core.py", line 112, in setup _setup_distribution = dist = klass(attrs) File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 219, in __init__ File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 243, in fetch_build_eggs File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 522, in resolve File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 758, in best_match File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin e 770, in obtain File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l ine 286, in fetch_build_egg File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 452, in easy_install File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 482, in install_item File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 661, in install_eggs File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 936, in build_and_install File "E:\allmydata- tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas y_install.py", line 927, in run_setup distutils.errors.DistutilsError: Setup script exited with error: Python was buil t with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 4 14:25:01 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 04 Jul 2009 23:25:01 +0200 Subject: [tahoe-dev] Truncated `tahoe.css' in source distribution? References: <87hbxs7216.fsf@gnu.org> <129A1BD4-0439-4E49-AD1A-5F83A2378343@zooko.com> Message-ID: <873a9cnnte.fsf@gnu.org> Zooko Wilcox-O'Hearn writes: > Hello, Ludovic Court?s! Hi! > Kevin Reid contributed an improved tahoe.css since the Tahoe v1.4.1 > release. If the .deb for Etch that you are using is more recent than > v1.4.1 then perhaps this explains it -- you have Kevin's new CSS file. OK. > Oh, actually here is a better way to tell: if the version you are > running is equal to or newer than v1.4.1-r3881 then you have the new > CSS. Debian's package version is 1.4.1-r3916, which explains the difference. Thanks, Ludo'. From trac at allmydata.org Sat Jul 4 14:31:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:31:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #747: Files uploaded to FTP server never appear Message-ID: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> #747: Files uploaded to FTP server never appear ---------------------------+------------------------------------------------ Reporter: ndurner | Owner: zooko Type: defect | Status: new Priority: major | Milestone: undecided Component: code-frontend | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ While connected to Tahoe's builtin FTP server: {{{ ncftp / > ls personal-bzr/ ncftp / > put kv.tar kv.tar: 1,24 MB 983,36 kB/s ncftp / > ls personal-bzr/ }}} This is with Tahoe 1.3, Twisted 8.2.0 on Windows XP. Even with the patch from FTP-and-SFTP.txt applied, the assertion at ftpd.py:293 still failed, so I uncommented that line. The last hunk in the patch didn't apply because the file is missing (nothing unusual, see Ticket #688). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 14:44:07 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:44:07 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid Message-ID: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> #748: Exception: when trying to create directory on testgrid -------------------------+-------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- In a random attempt to see what the new CSS wui looks like, I connected to testgrid.allmydata.com:3567 just now. I clicked the "Create a directory" button and was presented with the exception listed above. I'm afraid I see no further information to provide but the error is reproducable. The actual HTML source returned is the following: {{{ Exception

<class 'allmydata.mutable.common.UncoordinatedWriteError'>:

<class 'allmydata.mutable.common.UncoordinatedWriteError'>:

}}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 4 14:54:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 04 Jul 2009 21:54:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.fc62614d726b451f3523311e61d71392@allmydata.org> #748: Exception: when trying to create directory on testgrid -------------------------+-------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by zooko): Please look in your {{{~/.tahoe/logs/incidents}}} directory and attach any incident report files that occurred around this time. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Sat Jul 4 23:34:15 2009 From: warner at lothar.com (Brian Warner) Date: Sat, 4 Jul 2009 23:34:15 -0700 Subject: [tahoe-dev] Client/helper RAM usage In-Reply-To: <200907030818.49575.shawn-tahoe@willden.org> References: <200907030818.49575.shawn-tahoe@willden.org> Message-ID: <20090704233415.7f0ffab3@Cookies.lan> On Fri, 3 Jul 2009 08:18:49 -0600 Shawn Willden wrote: > Are files being processed (FEC-encoded, etc.) loaded into RAM? Is it > possible to upload a 2 GiB file on a machine with only 512 MiB RAM? > Is there any way to estimate the memory consumption that will be > required? Only in small pieces, yes, and yes, respectively :) We've carefully designed both the mutable/immutable encoding format and the immutable encode/decode algorithms to minimize the memory footprint they need while operating. They only ever work on one segment at a time, which (by default) is 128KiB. In general, the RAM footprint is a small multple of the segment size (maybe 3 or 4). We read in a segment's worth of plaintext from disk, encrypt it (then throw out the plaintext), encode it (then throw out the ciphertext), push it (then throw out the shares). It works the same way on the download side. The mutable publish/retrieve code does not yet do this: because we're still operating under "SDMF" rules, we only process one segment per mutable file. The maximum size limit for mutable files was recently removed, which means the maximum segment size limit was removed (segsize=filesize). So if you actually upload a 1GB *mutable* file, then you'll need 3 or 4 GB of ram. When we implement "MDMF", we'll change the mutable code to stream segments out the same way that immutable files do, and then switch back to a 128MiB segsize. There's a considerable static footprint, of course (this is python, after all). The buildbot performance page has a graph of this footprint over time. On a 32-bit machine, it's something like 40MB. > I'm thinking about putting a client or maybe a helper on a Linux > Virtual Server that I have, but it has only 512 MiB of RAM, and that > is a hard figure. There is no swap. I'd expect that to work just fine. cheers, -Brian From trac at allmydata.org Sat Jul 4 23:51:01 2009 From: trac at allmydata.org (pycryptopp) Date: Sun, 05 Jul 2009 06:51:01 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.05ad58ea6da4185fcca68b7f792dfc29@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments ----------------------+----------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: new Priority: critical | Version: 0.5.1 Keywords: integrity | Launchpad_bug: ----------------------+----------------------------------------------------- Comment(by bdew): Ok, the second patch works fine for me and fixes both this and http://allmydata.org/trac/tahoe/ticket/737 -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Sun Jul 5 07:00:15 2009 From: trac at allmydata.org (pycryptopp) Date: Sun, 05 Jul 2009 14:00:15 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.586311b89c1daa68da05fdd2edde260c@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments --------------------------+------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: closed Priority: critical | Version: 0.5.1 Resolution: fixed | Keywords: integrity Launchpad_bug: | --------------------------+------------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Fixed by [20090705135508-92b7f-c3c3ad54fdfc52fa22c1c2fbd70479ba27802952]. Thanks, MM, BD, Wei Dai! -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From zookog at gmail.com Sun Jul 5 07:59:16 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 08:59:16 -0600 Subject: [tahoe-dev] Client/helper RAM usage In-Reply-To: <20090704233415.7f0ffab3@Cookies.lan> References: <200907030818.49575.shawn-tahoe@willden.org> <20090704233415.7f0ffab3@Cookies.lan> Message-ID: Those graphs are linked from and described on this page: http://allmydata.org/trac/tahoe/wiki/Performance I don't think those graphs are reliable, though, as described in #227. However, I do think that those graphs, which are measuring vmsize, should show an upper bound on the number that you care about, so since the vmsize on that 32-bit machine is shown as being around 40 MB on our Performance page graphs, your virtual machine with 512 MiB should work juuuust fine. :-) I used Tahoe-LAFS on Zandr's 266 MHz ARM NAS box yesterday. It has 128 MiB of RAM. It seemed to work okay, but I had to stop experimenting with it since both my ISP and Zandr's router seem to be flaky at the moment (so I could no longer open TCP connections to that box). Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/227 # our automated memory measurements might be measuring the wrong thing From zookog at gmail.com Sun Jul 5 08:14:49 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 09:14:49 -0600 Subject: [tahoe-dev] Tahoe on little ARM CPU (Re: Client/helper RAM usage) Message-ID: On Sun, Jul 5, 2009 at 8:59 AM, Zooko O'Whielacronx wrote: > I used Tahoe-LAFS on Zandr's 266 MHz ARM NAS box yesterday. ?It has > 128 MiB of RAM. ?It seemed to work okay, but I had to stop > experimenting with it since both my ISP and Zandr's router seem to be > flaky at the moment (so I could no longer open TCP connections to that > box). Here is the statistics output from uploading a file from my home computer, using Zandr's NAS box as the Tahoe-LAFS gateway. It shows that there was only one storage server reachable to the NAS box and accepting shares at that time (I wonder why). It also shows that the speed of CPU-intensive parts like erasure coding and encryption was faster than the speed of pushing shares over the network to the storage servers. Regards, Zooko Uploading File... done! Upload Results: * URI: URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 * Download link: /uri/URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 * Sharemap: o 0 -> placed on [6fyx5u4z] o 1 -> placed on [6fyx5u4z] o 2 -> placed on [6fyx5u4z] o 3 -> placed on [6fyx5u4z] o 4 -> placed on [6fyx5u4z] o 5 -> placed on [6fyx5u4z] o 6 -> placed on [6fyx5u4z] o 7 -> placed on [6fyx5u4z] o 8 -> placed on [6fyx5u4z] o 9 -> placed on [6fyx5u4z] * Servermap: o [6fyx5u4z] got shares: #0,#1,#2,#3,#4,#5,#6,#7,#8,#9 * Timings: o File Size: 371474 bytes o Total: 32 seconds (11.4kBps) + Storage Index: 1.51s (246.1kBps) + [Contacting Helper]: # [Helper Already-In-Grid Check]: + [Upload Ciphertext To Helper]: () + Peer Selection: 3.97s + Encode And Push: 27 seconds (19.3kBps) # Cumulative Encoding: 6.51s (57.1kBps) # Cumulative Pushing: 12 seconds (29.2kBps) # Send Hashes And Close: 6.75s + [Helper Total]: Return to the Welcome Page From zookog at gmail.com Sun Jul 5 09:03:32 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sun, 5 Jul 2009 10:03:32 -0600 Subject: [tahoe-dev] Tahoe on little ARM CPU (Re: Client/helper RAM usage) In-Reply-To: References: Message-ID: I just did it again and this time it uploaded different shares to different servers (on the Test Grid). Again, the encoding is faster than the pushing, which is encouraging -- it means that this little ARM 266 MHz NAS box is more limited by the speed of its network connection to the storage servers than by its CPU power. (Although it is being limited by both, and this is a very light load -- only one client.) Regards, Zooko # URI: URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 # Download link: /uri/URI:CHK:a5l34bmeviutszz55b5dav3xhu:ze4pmszhqoxa5ypitvhoiobghozcbmf5nlala2kk4mujbn5424pq:3:10:371474 # Sharemap: * 0 -> placed on [fcmlx6em] * 1 -> placed on [xiktf6ok] * 2 -> placed on [jfdpabh3] * 3 -> placed on [qvqv7jmm] * 4 -> placed on [7tlov7eg] * 5 -> placed on [lwkv6cji] * 6 -> placed on [wfninubk] * 7 -> placed on [trjdor3o] # Servermap: * [fcmlx6em] got share: #0 * [jfdpabh3] got share: #2 * [lwkv6cji] got share: #5 * [qvqv7jmm] got share: #3 * [trjdor3o] got share: #7 * [wfninubk] got share: #6 * [xiktf6ok] got share: #1 * [7tlov7eg] got share: #4 # Timings: * File Size: 371474 bytes * Total: 16 seconds (22.1kBps) o Storage Index: 509ms (729.3kBps) o [Contacting Helper]: + [Helper Already-In-Grid Check]: o [Upload Ciphertext To Helper]: () o Peer Selection: 994ms o Encode And Push: 15 seconds (34.2kBps) + Cumulative Encoding: 3.41s (109.0kBps) + Cumulative Pushing: 7.46s (49.8kBps) + Send Hashes And Close: 3.79s From shawn-tahoe at willden.org Sun Jul 5 11:34:17 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 5 Jul 2009 12:34:17 -0600 Subject: [tahoe-dev] Verifying file contents Message-ID: <200907051234.17821.shawn-tahoe@willden.org> Is there an easy/inexpensive way to check that a local file matches a remote (immutable) file? I could extract the Tahoe code that calculates the key from the local file, given the current convergence secret, and then compare that with the key in the URI. Or perhaps modify Tahoe to expose a "key calculation" service through the web API. Another thing I could do is upload the local file as an immutable file. If the file is already in the grid, this shouldn't take very long, and the returned URI should match. But if the local file doesn't match the remote file, the local file will be uploaded to the grid, which I don't really want. Is there a way (through the web API) to "fake" a PUT? That is, do everything except actually communicate with storage servers? A problem with all of these methods is that they break if the convergence key doesn't match, but I suppose that's a feature, not a bug. Thanks, Shawn From zooko at zooko.com Sun Jul 5 12:07:56 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 5 Jul 2009 13:07:56 -0600 Subject: [tahoe-dev] Verifying file contents In-Reply-To: <200907051234.17821.shawn-tahoe@willden.org> References: <200907051234.17821.shawn-tahoe@willden.org> Message-ID: <72EC76A2-59D8-4559-BEA7-ED1439D86C2E@zooko.com> On Jul 5, 2009, at 12:34 PM, Shawn Willden wrote: > Is there an easy/inexpensive way to check that a local file matches > a remote (immutable) file? I think this is pretty much the same thing as #713. If we had #713 then you could run that and then compare it to the cap to the remote file. Perhaps you should paste your new/related use case from this thread into that ticket. Regards, Zooko Tickets mentions in this e-mail: http://allmydata.org/trac/tahoe/ticket/713 # tahoe make-verify-cap From trac at allmydata.org Sun Jul 5 13:25:58 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 20:25:58 -0000 Subject: [tahoe-dev] [tahoe-lafs] #713: tahoe make-verify-cap In-Reply-To: <037.8ce6933cfa6fcba192d994ce27bd7c31@allmydata.org> References: <037.8ce6933cfa6fcba192d994ce27bd7c31@allmydata.org> Message-ID: <046.40e60990f6ab2a08db001a259b1e7366@allmydata.org> #713: tahoe make-verify-cap -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-frontend-cli | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by swillden): Replying to [comment:6 swillden]: > The need to have read cap index files has driven me to find other uses for them as well, uses that I'd have to find some other way to implement if I didn't have them, so between the performance issue and that one, I'm not sure I'd use a "dry run" option, even if it existed. As zooko pointed out in response to a recent mailing list question, this would be a good solution for a need I have to verify that a grid-based immutable file matches a local file. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 15:33:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 22:33:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.186de45a6287ad75e32b1076a66e53a8@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by kevan): I've run the benchmarks, and am attaching the results. To get the {{{unmodifed_results}}} file, I simply ran {{{bench_dirnode.py}}} against an unmodified checkout from trunk. To get the {{{modified_results}}} file, I needed to change a call to {{{dict}}} in {{{bench_dirnode.py}}} to refer to {{{dirnode.CachingDict}}} (otherwise {{{_pack_contents}}} raises an {{{AssertionError}}}). I'm attaching a patch that duplicates that change. I then applied {{{dict.txt}}}, {{{optimizations.txt}}}, and {{{tests.txt}}} to an otherwise unmodified source tree, and ran {{{bench_dirnode.py}}} against it. The results seem to favor the optimized implementation. I'm also updating dict.txt again -- hopefully for the last time -- to fix another *args related issue that I found. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 15:39:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 05 Jul 2009 22:39:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #327: performance measurement of directories In-Reply-To: <037.99b82637bbb101b1ebc214b8c8630a60@allmydata.org> References: <037.99b82637bbb101b1ebc214b8c8630a60@allmydata.org> Message-ID: <046.060503d43e096893edacfcbbf4b9145d@allmydata.org> #327: performance measurement of directories ------------------------------------------+--------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: assigned Priority: major | Milestone: eventually Component: code-dirnodes | Version: 0.8.0 Keywords: test performance scalability | Launchpad_bug: ------------------------------------------+--------------------------------- Comment(by kevan): (this is in response to the first iteration of the benchmarking script; the issue is addressed in r3965) Cool! I've run it on my machine, and noticed that it actually shows slower results for the optimized code. I think that's a matter of methodology, though. From what I understand (having never used benchutil), to test pack_contents you're building a list of (name, child) tuples, feeding that into a dict, and then feeding the dict to pack_contents, and you're testing how long that takes for increasing numbers of tuples. To test unpack_contents, you're doing that, but saving the result of pack_contents in a global variable, then feeding that to unpack_contents to see how long it takes. If I'm right, we aren't seeing any speed improvements because the benchmark isn't actually testing the optimizations. In order to do that, we need to feed pack_contents a dictionary that was actually output from unpack_contents (or else built with set_both_items instead of __setitem__) -- that way, the set_both_items method of the dict wrapper will have been used to set the serialized value, and pack_contents will find and use that value, thus (ideally) making it faster. One way to do this might be to stick the logic for child creation into a more general setup method -- something which, when called, would generate + pack the list of children, and return the results of pack_contents on that list, for example. Then the init method for the unpack test could store that value in the global packstr variable, and work as it does now, while the init method for the pack test could unpack that, then feed the resulting dictionary into pack_contents again, where the optimizations would start working. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 5 22:07:00 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 05:07:00 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.6a3e6c4fbc8a28f92529d8660378cbc0@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Wow, Wei Dai is fast. Check it out, he's fixed the problem already in SVN: [http://cryptopp.svn.sourceforge.net/viewvc/cryptopp?view=rev&sortby=date&revision=470 Crypto++ SVN Rev 470] Impressive. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Mon Jul 6 19:31:14 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 6 Jul 2009 20:31:14 -0600 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> Message-ID: <51864C9C-C63D-4BA1-A6ED-21C203113440@zooko.com> On Jul 6, 2009, at 17:15 PM, tahoe-lafs wrote: > #737: python2.5 setup.py test runs CPU to 100% on 32-bit single- > core NetBSD "4" ... > Comment(by midnightmagic): ... > Other things fail, but.. looks like it's just because it's trunk and > certainly not the same problem this ticket references. Do tell! :-) Regards, Zooko From trac at allmydata.org Mon Jul 6 13:55:12 2009 From: trac at allmydata.org (pycryptopp) Date: Mon, 06 Jul 2009 20:55:12 -0000 Subject: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD with multiple segments In-Reply-To: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> References: <054.f01281bb4081e8def8f5665c62abf540@allmydata.org> Message-ID: <063.b7ec91724c4574716b5e330c06005a99@allmydata.org> #24: SHA256 failure on NetBSD with multiple segments --------------------------+------------------------------------------------- Reporter: warner | Owner: bdew Type: defect | Status: closed Priority: critical | Version: 0.5.1 Resolution: fixed | Keywords: integrity Launchpad_bug: | --------------------------+------------------------------------------------- Comment(by midnightmagic): Just wanted to chime in and state that trunk as of the patch committed, above, works perfectly on all machines of mine (NetBSD all, all Athlon XP, Celeron, Pentium III) and the problem pycryptopp test now passes perfectly: {{{ test_chunksize (pycryptopp.test.test_sha256.SHA256) ... ok }}} The following machines now report success, and this apparently with full ASM optimizations: __quickie__ NetBSD quickie 4.99.7 NetBSD 4.99.7 (quickie) #0: Tue Jan 2 14:47:23 PST 2007 root at shoggoth:/v/src/sys/arch/i386/compile/quickie i386 CPU: AMD Athlon XP 2500+ (686-class), 1837.62 MHz, id 0x6a0 __warp__ NetBSD warp 4.0_STABLE NetBSD 4.0_STABLE (warp) #0: Sat Jun 20 23:42:00 PDT 2009 root at shoggoth:/v/src/sys/arch/i386/compile/obj/warp i386 CPU: Intel Celeron (686-class), 1196.44 MHz, id 0x6b1 __draco__ NetBSD draco 3.99.23 NetBSD 3.99.23 (GENERIC) #0: Thu Jul 27 08:22:27 PDT 2006 root at shoggoth:/usr/netbsd- build/netbsd-3.99.23/i386/OBJ/usr/v/src/sys/arch/i386/compile/GENERIC i386 CPU: Intel Pentium III (686-class), 731.29 MHz, id 0x683 -- Ticket URL: pycryptopp Python bindings for the Crypto++ library From trac at allmydata.org Mon Jul 6 16:15:39 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:15:39 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.32310311d43ffe71a1070b1e2e058e30@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): Using trunk from pycryptopp and trunk tahoe (via PYTHONPATH and deleting tahoe/support/lib/site-packages/pycrypto*) the test no longer fails and pushes through the spot where it failed before. .. with full crypto optimizations. Other things fail, but.. looks like it's just because it's trunk and certainly not the same problem this ticket references. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 16:18:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:18:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #738: failure in block hash tree In-Reply-To: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> References: <045.4eea0d1907b3107618e096c249d606ba@allmydata.org> Message-ID: <054.e6f675ea1334573718e1bc6b68dc3311@allmydata.org> #738: failure in block hash tree ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: bdew Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-encoding | Version: 1.4.1 Keywords: integrity | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by midnightmagic): pycryptopp in trunk now works perfectly (well the tests don't fail anyway) on all three machines, as listed in [http://allmydata.org/trac/pycryptopp/ticket/24 pycryptopp ticket 24]. Using pycryptopp trunk, I now have apparently perfectly-working tahoe nodes where before they were only remotely usable. Therefore, I believe this ticket can be closed, from my perspective. If 1.5.0 is going to include these fixes, then all's well! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 16:43:00 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 06 Jul 2009 23:43:00 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.5261c72d223550638298d700da75f064@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" ---------------------------+------------------------------------------------ Reporter: midnightmagic | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by bdew): Same here, with fixed pycryptopp all the tests pass without hangs/rashes: http://allmydata.org/buildbot/builders/BlackDew%20debian- unstable-i386/builds/40/steps/test/logs/stdio -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 19:39:49 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 02:39:49 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings Message-ID: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings --------------------+------------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------+------------------------------------------------------- These six unit tests fail if optimization is turned on: {{{ $ PYTHONOPTIMIZE=2 PYTHONPATH=./support/lib/python2.6/site-packages/ trial allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte allmydata.test.test_uri.Constraint.test_constraint allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte allmydata.test.test_uri.Constraint.test_constraint allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named allmydata.test.test_web.Web.test_GET_unhandled_URI allmydata.test.test_web.Web.test_GET_unhandled_URI_named }}} ... {{{ allmydata.test.test_uri Constraint test_constraint ... [FAIL] allmydata.test.test_mutable Roundtrip test_corrupt_all_verbyte ... [FAIL] allmydata.test.test_web Web test_GET_unhandled_URI ... [FAIL] [ERROR] test_GET_unhandled_URI_named ... [FAIL] [ERROR] =============================================================================== [FAIL]: allmydata.test.test_uri.Constraint.test_constraint Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_uri.py", line 189, in test_constraint self.failUnlessRaises(AssertionError, uri.NewDirectoryURI.init_from_string, good) twisted.trial.unittest.FailTest: raised instead of AssertionError: Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 752, in _run self.getSuppress(), method) File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/utils.py", line 191, in runWithWarningsSuppressed result = f(*a, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_uri.py", line 189, in test_constraint self.failUnlessRaises(AssertionError, uri.NewDirectoryURI.init_from_string, good) --- --- File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 235, in failUnlessRaises result = f(*args, **kwargs) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/uri.py", line 337, in init_from_string bits = uri[mo.end():] exceptions.AttributeError: 'NoneType' object has no attribute 'end' =============================================================================== [FAIL]: allmydata.test.test_mutable.Roundtrip.test_corrupt_all_verbyte Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_mutable.py", line 1033, in _do_retrieve self.failUnless(substring in "".join(allproblems)) twisted.trial.unittest.FailTest: None =============================================================================== [FAIL]: allmydata.test.test_web.Web.test_GET_unhandled_URI Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_web.py", line 341, in done % (which, substring, str(res))) twisted.trial.unittest.FailTest: test_GET_unhandled_URI: substring '400 Bad Request' not in '[Failure instance: Traceback (failure with no frames): : 500 Internal Server Error ]' =============================================================================== [FAIL]: allmydata.test.test_web.Web.test_GET_unhandled_URI_named Traceback (most recent call last): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/test_web.py", line 341, in done % (which, substring, str(res))) twisted.trial.unittest.FailTest: GET_unhandled_URI_named: substring '400 Bad Request' not in '[Failure instance: Traceback (failure with no frames): : 500 Internal Server Error ]' =============================================================================== [ERROR]: allmydata.test.test_web.Web.test_GET_unhandled_URI Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/common.py", line 220, in renderHTTP return m(ctx) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/filenode.py", line 181, in render_GET if self.node.is_mutable(): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/common.py", line 183, in is_mutable return self.my_uri.is_mutable() exceptions.AttributeError: CHKFileVerifierURI instance has no attribute 'is_mutable' =============================================================================== [ERROR]: allmydata.test.test_web.Web.test_GET_unhandled_URI_named Traceback (most recent call last): File "/usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 106, in maybeDeferred result = f(*args, **kw) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/common.py", line 220, in renderHTTP return m(ctx) File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/web/filenode.py", line 181, in render_GET if self.node.is_mutable(): File "/home/zooko/playground/allmydata/tahoe/trunk/optimize_directories/src/allmydata/test/common.py", line 183, in is_mutable return self.my_uri.is_mutable() exceptions.AttributeError: CHKFileVerifierURI instance has no attribute 'is_mutable' ------------------------------------------------------------------------------- Ran 4 tests in 0.586s FAILED (failures=4, errors=2) }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 6 23:08:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 06:08:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.ddd46174d6a278a2e034761bf7a49644@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): The test passes because it's testing the filesystem layer, while {{{tahoe_mv.py}}} uses the webapi. Basically, we want {{{tahoe mv}}} to do the following: * If we're doing {{{tahoe mv file1 file2}}}, {{{file1}}} should replace {{{file2}}}. * If we're doing {{{tahoe mv file1 folder1/}}}, {{{file1}}} should be inserted into {{{folder1}}}. * If we're doing {{{tahoe mv file1 folder1}}}, {{{tahoe mv}}} should print an error. To the user, this looks more or less the same regardless of how we do it on the backend -- {{{tahoe mv}}} should output "OK" in the first two cases, and an error message in the second. I'm attaching some tests that make sure that happens. I guess there are a few ways to implement this sort of functionality. * We could rely on the CLI (i.e., {{{tahoe mv}}}) to be smart enough to distinguish between files and folders, and structure HTTP requests appropriately. By using the methods described in {{{=== Get Information About A File Or Directory (as JSON) ===}}}, we can easily (and rather laboriously) determine whether the target is a file or a directory, and then act accordingly. * Alternatively, we could maybe bake some sort of functionality into the webapi that covers this. I'm not sure of the best way to do this, though. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 02:09:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 09:09:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.338ef5136a06df0ae201cc50d41d20cd@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): could you summarize the benchmark numbers in terms of A+Bx factors? Also, it's not clear to me what those numbers point to as being the expensive part: is it the netstring parsing? or the encryption? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 07:38:33 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 14:38:33 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.68c41b34520b39dc0adc967c3ee833b9@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed ---------------------------+------------------------------------------------ Reporter: warner | Owner: kevan Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Keywords: dirnode | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Summarizing into {{{A+Bx}}} is not the way I do it. Instead I look at a handful of data points, like this: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.09, worst: 0.10 (of 5), reps/s: 11, ave rate: 732 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.36, 2th- worst: 0.35, worst: 0.39 (of 5), reps/s: 2, ave rate: 714 N: 1024, time: best: 1.51, 2th-best: 1.51, ave: 1.55, 2th- worst: 1.51, worst: 1.72 (of 5), reps/s: 0, ave rate: 659 N: 4096, time: best: 10.48, 2th-best: 10.49, ave: 10.66, 2th- worst: 10.53, worst: 11.29 (of 5), reps/s: 0, ave rate: 384 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 747 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.35, 2th- worst: 0.35, worst: 0.35 (of 5), reps/s: 2, ave rate: 728 N: 1024, time: best: 1.52, 2th-best: 1.53, ave: 1.53, 2th- worst: 1.53, worst: 1.53 (of 5), reps/s: 0, ave rate: 670 N: 4096, time: best: 10.48, 2th-best: 10.49, ave: 10.50, 2th- worst: 10.52, worst: 10.53 (of 5), reps/s: 0, ave rate: 390 benchmarking N: 64, time: best: 0.08, 2th-best: 0.09, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 744 N: 256, time: best: 0.35, 2th-best: 0.35, ave: 0.35, 2th- worst: 0.35, worst: 0.35 (of 5), reps/s: 2, ave rate: 727 N: 1024, time: best: 1.52, 2th-best: 1.52, ave: 1.54, 2th- worst: 1.53, worst: 1.60 (of 5), reps/s: 0, ave rate: 665 N: 4096, time: best: 10.49, 2th-best: 10.50, ave: 10.51, 2th- worst: 10.51, worst: 10.52 (of 5), reps/s: 0, ave rate: 390 }}} This is a good example of why I do it this way: because it isn't linear! So any {{{A+Bx}}} summary would be off. I already have a patch which fixes the observable non-linearity there. I guess I might as well commit that one so that everyone can see it. There -- [3969], entitled "directories: keep track of your position as you decode netstring after netstring from an input buffer instead of copying the trailing part | This makes decoding linear in the number of netstrings instead of O(N^2^).". After this patch, the same benchmark on the same machine says: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.08, worst: 0.10 (of 5), reps/s: 11, ave rate: 750 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.34, 2th- worst: 0.33, worst: 0.37 (of 5), reps/s: 2, ave rate: 755 N: 1024, time: best: 1.28, 2th-best: 1.28, ave: 1.32, 2th- worst: 1.28, worst: 1.48 (of 5), reps/s: 0, ave rate: 776 N: 4096, time: best: 4.83, 2th-best: 4.83, ave: 4.97, 2th- worst: 4.83, worst: 5.51 (of 5), reps/s: 0, ave rate: 824 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 760 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.33, 2th- worst: 0.33, worst: 0.33 (of 5), reps/s: 2, ave rate: 767 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.29, 2th- worst: 1.29, worst: 1.29 (of 5), reps/s: 0, ave rate: 794 N: 4096, time: best: 4.83, 2th-best: 4.83, ave: 4.83, 2th- worst: 4.83, worst: 4.84 (of 5), reps/s: 0, ave rate: 847 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 759 N: 256, time: best: 0.33, 2th-best: 0.33, ave: 0.33, 2th- worst: 0.33, worst: 0.33 (of 5), reps/s: 2, ave rate: 767 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.30, 2th- worst: 1.29, worst: 1.36 (of 5), reps/s: 0, ave rate: 786 N: 4096, time: best: 4.82, 2th-best: 4.83, ave: 4.85, 2th- worst: 4.87, worst: 4.90 (of 5), reps/s: 0, ave rate: 844 }}} As to your question about what's the expensive part, after the optimization patch from Kevan plus a few that I have here in my sandbox, benchmarking shows: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 849 N: 256, time: best: 0.29, 2th-best: 0.30, ave: 0.31, 2th- worst: 0.32, worst: 0.32 (of 5), reps/s: 3, ave rate: 832 N: 1024, time: best: 1.24, 2th-best: 1.24, ave: 1.30, 2th- worst: 1.27, worst: 1.52 (of 5), reps/s: 0, ave rate: 787 N: 4096, time: best: 4.28, 2th-best: 4.29, ave: 4.40, 2th- worst: 4.31, worst: 4.82 (of 5), reps/s: 0, ave rate: 931 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.08, worst: 0.08 (of 5), reps/s: 13, ave rate: 843 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.31, worst: 0.32 (of 5), reps/s: 3, ave rate: 840 N: 1024, time: best: 1.14, 2th-best: 1.14, ave: 1.16, 2th- worst: 1.17, worst: 1.20 (of 5), reps/s: 0, ave rate: 883 N: 4096, time: best: 4.30, 2th-best: 4.31, ave: 4.53, 2th- worst: 4.64, worst: 4.99 (of 5), reps/s: 0, ave rate: 904 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 860 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 880 N: 1024, time: best: 1.14, 2th-best: 1.14, ave: 1.15, 2th- worst: 1.14, worst: 1.21 (of 5), reps/s: 0, ave rate: 889 N: 4096, time: best: 4.28, 2th-best: 4.28, ave: 4.29, 2th- worst: 4.29, worst: 4.30 (of 5), reps/s: 0, ave rate: 955 }}} and profiling shows: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.09, 2th- worst: 0.08, worst: 0.10 (of 5), reps/s: 11, ave rate: 738 N: 256, time: best: 0.34, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.38 (of 5), reps/s: 2, ave rate: 744 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.48, 2th- worst: 1.46, worst: 2.04 (of 5), reps/s: 0, ave rate: 693 N: 4096, time: best: 4.84, 2th-best: 4.85, ave: 4.97, 2th- worst: 4.87, worst: 5.45 (of 5), reps/s: 0, ave rate: 824 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.10, 2th- worst: 0.09, worst: 0.18 (of 5), reps/s: 9, ave rate: 611 N: 256, time: best: 0.33, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.34 (of 5), reps/s: 2, ave rate: 764 N: 1024, time: best: 1.29, 2th-best: 1.29, ave: 1.30, 2th- worst: 1.30, worst: 1.30 (of 5), reps/s: 0, ave rate: 790 N: 4096, time: best: 4.85, 2th-best: 4.85, ave: 4.88, 2th- worst: 4.87, worst: 4.96 (of 5), reps/s: 0, ave rate: 840 benchmarking N: 64, time: best: 0.08, 2th-best: 0.08, ave: 0.08, 2th- worst: 0.08, worst: 0.09 (of 5), reps/s: 11, ave rate: 754 N: 256, time: best: 0.33, 2th-best: 0.34, ave: 0.34, 2th- worst: 0.34, worst: 0.34 (of 5), reps/s: 2, ave rate: 764 N: 1024, time: best: 1.30, 2th-best: 1.30, ave: 1.31, 2th- worst: 1.31, worst: 1.37 (of 5), reps/s: 0, ave rate: 779 N: 4096, time: best: 4.85, 2th-best: 4.87, ave: 4.89, 2th- worst: 4.91, worst: 4.96 (of 5), reps/s: 0, ave rate: 837 674783 function calls in 4.909 CPU seconds Ordered by: internal time List reduced from 75 to 32 due to restriction <32> ncalls tottime percall cumtime percall filename:lineno(function) 36830 2.422 0.000 2.591 0.000 base32.py:60(b2a_l) 7366 0.527 0.000 1.606 0.000 base32.py:208(a2b_l) 54026 0.154 0.000 0.154 0.000 netstring.py:3(netstring) 20879 0.149 0.000 0.149 0.000 hashutil.py:26(digest) 7366 0.111 0.000 0.111 0.000 hashutil.py:163(_xor) 36830 0.093 0.000 0.093 0.000 string.py:306(join) 3683 0.088 0.000 0.088 0.000 encoder.py:205(iterencode) 58928 0.080 0.000 0.080 0.000 string.py:480(translate) 1 0.076 0.076 1.905 1.905 dirnode.py:248(_pack_contents) 7366 0.066 0.000 0.066 0.000 netstring.py:7(split_netstring) 29464 0.060 0.000 2.141 0.000 base32.py:48(b2a) 49124 0.058 0.000 0.058 0.000 hashutil.py:23(update) 3683 0.056 0.000 0.386 0.000 dirnode.py:196(_encrypt_rwcap) 7366 0.055 0.000 0.199 0.000 hashutil.py:48(tagged_pair_hash) 13513 0.054 0.000 0.134 0.000 hashutil.py:38(tagged_hasher) 3683 0.050 0.000 0.050 0.000 decoder.py:341(raw_decode) 1 0.046 0.046 2.999 2.999 dirnode.py:218(_unpack_contents) 3683 0.042 0.000 0.155 0.000 dirnode.py:207(_decrypt_rwcapdata) 7366 0.040 0.000 1.697 0.000 base32.py:199(a2b) 125222 0.040 0.000 0.040 0.000 assertutil.py:30(precondition) 3683 0.040 0.000 0.151 0.000 hashutil.py:166(hmac) 13513 0.037 0.000 0.286 0.000 hashutil.py:43(tagged_hash) 20879 0.035 0.000 0.035 0.000 hashutil.py:19(__init__) }}} Whoops, time for me to go to work! I hope to measure and commit the rest of my optimization patches today after work. There is one that I really want other people's input on before I commit it... That one generates the per-entry IV with a secure hash of the writecap instead of with {{{os.urandom(16)}}}. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 08:01:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 15:01:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries Message-ID: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> #750: deterministic IV for writecaps for dir entries -----------------------------+---------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Keywords: confidentiality | Launchpad_bug: -----------------------------+---------------------------------------------- Okay, here is a patch which replaces {{{os.urandom(16)}}} with a (tagged) secure hash of the write cap. This protects the directory entries against the "VM rollback problem", which otherwise might expose the writecaps of entries to someone who has only the readcap to the dir in the case that the writer of the dir suffered a vm rollback. In addition, this makes packing deterministic, so that pack(unpack(packeddir)) == packeddir. (Kevan's original unit tests for this ticket assumed that would be the case.) Is there any cryptographic problem that making this change would raise? It means that encryption of the writecaps doesn't have "semantic security" (which I think corresponds to IND-CPA?), but they actually didn't anyway since each one is accompanied by its readcap. Perhaps surprisingly, this patch appears to reduce the CPU usage a little bit for packing directories (at least on Mac OS X). Please examine this patch, all cryptographers and security experts! Before this patch: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.09 (of 5), reps/s: 13, ave rate: 837 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.30, worst: 0.32 (of 5), reps/s: 3, ave rate: 851 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.18, 2th- worst: 1.15, worst: 1.31 (of 5), reps/s: 0, ave rate: 866 N: 4096, time: best: 4.37, 2th-best: 4.37, ave: 4.49, 2th- worst: 4.41, worst: 4.93 (of 5), reps/s: 0, ave rate: 912 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 862 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 872 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.15, 2th- worst: 1.15, worst: 1.15 (of 5), reps/s: 0, ave rate: 889 N: 4096, time: best: 4.36, 2th-best: 4.38, ave: 4.39, 2th- worst: 4.38, worst: 4.45 (of 5), reps/s: 0, ave rate: 933 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 857 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.30 (of 5), reps/s: 3, ave rate: 870 N: 1024, time: best: 1.15, 2th-best: 1.15, ave: 1.15, 2th- worst: 1.15, worst: 1.15 (of 5), reps/s: 0, ave rate: 888 N: 4096, time: best: 4.37, 2th-best: 4.38, ave: 4.38, 2th- worst: 4.38, worst: 4.39 (of 5), reps/s: 0, ave rate: 935 }}} after this patch: {{{ benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.08, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 844 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.30, 2th- worst: 0.29, worst: 0.32 (of 5), reps/s: 3, ave rate: 858 N: 1024, time: best: 1.11, 2th-best: 1.11, ave: 1.15, 2th- worst: 1.11, worst: 1.27 (of 5), reps/s: 0, ave rate: 894 N: 4096, time: best: 4.25, 2th-best: 4.26, ave: 4.37, 2th- worst: 4.27, worst: 4.79 (of 5), reps/s: 0, ave rate: 938 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 859 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 874 N: 1024, time: best: 1.11, 2th-best: 1.12, ave: 1.12, 2th- worst: 1.12, worst: 1.12 (of 5), reps/s: 0, ave rate: 917 N: 4096, time: best: 4.26, 2th-best: 4.26, ave: 4.27, 2th- worst: 4.28, worst: 4.28 (of 5), reps/s: 0, ave rate: 959 benchmarking N: 64, time: best: 0.07, 2th-best: 0.07, ave: 0.07, 2th- worst: 0.07, worst: 0.08 (of 5), reps/s: 13, ave rate: 856 N: 256, time: best: 0.29, 2th-best: 0.29, ave: 0.29, 2th- worst: 0.29, worst: 0.29 (of 5), reps/s: 3, ave rate: 871 N: 1024, time: best: 1.11, 2th-best: 1.11, ave: 1.13, 2th- worst: 1.12, worst: 1.18 (of 5), reps/s: 0, ave rate: 908 N: 4096, time: best: 4.25, 2th-best: 4.25, ave: 4.25, 2th- worst: 4.26, worst: 4.26 (of 5), reps/s: 0, ave rate: 963 }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 12:09:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 19:09:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" In-Reply-To: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> References: <045.b936813cbea3a15734e0ecd22cfb0fc5@allmydata.org> Message-ID: <054.7f2155e9763af9cb279f6bf7c648e2ba@allmydata.org> #737: python2.5 setup.py test runs CPU to 100% on 32-bit single-core NetBSD "4" -------------------------------+-------------------------------------------- Reporter: midnightmagic | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Fixed by [3966] which increases the requirement on pycryptopp to >= 0.5.15. Note that if you are building pycryptopp against an external libcryptopp, however, then you may still have this bug if your libcryptopp has it. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 7 14:06:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 07 Jul 2009 21:06:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #747: Files uploaded to FTP server never appear In-Reply-To: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> References: <039.6fbc96e50c158c710b04fadd601c5397@allmydata.org> Message-ID: <048.92c84d26d442f40aa1dd413fd4dcf068@allmydata.org> #747: Files uploaded to FTP server never appear -------------------------------+-------------------------------------------- Reporter: ndurner | Owner: zooko Type: defect | Status: closed Priority: major | Milestone: undecided Component: code-frontend | Version: 1.3.0 Resolution: invalid | Keywords: Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by ndurner): * status: new => closed * resolution: => invalid Comment: False alarm, sorry. I patched in the wrong directory, Tahoe's complaints were correct. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Tue Jul 7 15:59:05 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Wed, 08 Jul 2009 00:59:05 +0200 Subject: [tahoe-dev] Tahoe packaged for NixOS Message-ID: <87k52krtfq.fsf@gnu.org> Hello! I packaged Tahoe [0] and its dependencies [1] for Nixpkgs/NixOS. NixOS (http://nixos.org/) is a GNU/Linux distribution, which could well be dubbed "the least-authority GNU/Linux distro" given that it follows POLA for build processes. The build farm is currently in a bad shape, but one can normally see builds at, e.g., [2] and [3]. For zfec, pycryptopp, pyutil, etc., tests are run and all pass. However, trying to run Tahoe's tests currently fails: --8<---------------cut here---------------start------------->8--- running develop Checking .pth file support in support/lib/python2.5/site-packages error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: 'support/lib/python2.5/site-packages/test-easy-install-20210.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: support/lib/python2.5/site-packages This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). --8<---------------cut here---------------end--------------->8--- I haven't yet investigated, but I wonder where the offending path comes from. Thanks, Ludo'. [0] https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/tools/networking/p2p/tahoe/default.nix [1] https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/top-level/python-packages.nix [2] http://hydra.nixos.org/job/nixpkgs/trunk/tahoe [4] http://hydra.nixos.org/job/nixpkgs/trunk/pythonPackages.zfec From zooko at zooko.com Tue Jul 7 16:06:13 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 7 Jul 2009 17:06:13 -0600 Subject: [tahoe-dev] Tahoe packaged for NixOS In-Reply-To: <87k52krtfq.fsf@gnu.org> References: <87k52krtfq.fsf@gnu.org> Message-ID: On Jul 7, 2009, at 16:59 PM, Ludovic Court?s wrote: > I packaged Tahoe [0] and its dependencies [1] for Nixpkgs/NixOS. > NixOS (http://nixos.org/) is a GNU/Linux distribution, which could > well be dubbed "the least-authority GNU/Linux distro" given that it > follows POLA for build processes. Cool! :-) > Checking .pth file support in support/lib/python2.5/site-packages > error: can't create or remove files in install directory Oh, this should be fixed when I finally fix ticket #668. For what it is worth, this is also setuptools issue 54 (be more like distutils with regard to --prefix=), but it doesn't look like it is going to be fixed in setuptools anytime soon, so I intend to patch our version of setuptools to do this. Regards, Zooko tickets mentioned in this email: http://allmydata.org/trac/tahoe/ticket/668 # easy_install allmydata- tahoe is broken From zooko at zooko.com Wed Jul 8 06:00:46 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 8 Jul 2009 07:00:46 -0600 Subject: [tahoe-dev] buildbot news: Shawn's jaunty-amd64 demoted, BlackDew's debian-i386 and Zandr's lenny-armel promoted, Arthur's lenny-c7 added. Zooko's mac-powerpc dropped In-Reply-To: References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> Message-ID: <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Folks: Shawn's jaunty-amd64 machine had mysterious failures and has been demoted from "Supported" to "Unsupported" status. BlackDew's debian-i386 and Zandr's lenny-armel are green and have been promoted to "Supported". This makes for the first "embedded" CPU in the Supported category! :-) Arthur Lutz has contributed a C7 box running Debian, which as been added in the Unsupported category. Finally, we must say a sad farewell, at least temporarily, to my PowerPC G4 867 MHz laptop "draco". My four-year old son dropped draco and now it won't power up. http://allmydata.org/buildbot/ Regards, Zooko From trac at allmydata.org Wed Jul 8 21:28:44 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 04:28:44 -0000 Subject: [tahoe-dev] [tahoe-lafs] #752: speed up directories more Message-ID: <037.22675d737680f7eb9a51c0982f43738e@allmydata.org> #752: speed up directories more ---------------------------+------------------------------------------------ Reporter: zooko | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-dirnodes | Version: 1.4.1 Keywords: performance | Launchpad_bug: ---------------------------+------------------------------------------------ Even after [3969], [3971], [3972], ''and'' the not-yet-committed patch to generate IVs with hashes of writecaps (#750) ''and'' a not-yet-committed patch to run with {{{-OO}}} optimization mode, directories are still too slow. This set of five patches speeds directories up significantly: on my amd64 workstation yukyuk: before these five optimization patches: unpack and repack a 64-entry directory: 0.10 seconds, a 1024-entry directory: 1.75s, a 4096-entry directory: 7.75s, a 65536-entry directory: 441.65s. after these five optimization patches: unpack and repack a 64-entry directory: 0.08s, a 1024-entry directory: 1.3s, a 4096-entry directory: 4.95s, a 65536-entry directory: 115.4s on Zandr's 266 MHz armel box: before these five optimization patches: unpack and repack a 64-entry directory: 2.19 seconds, a 1024-entry directory: 35.48s, a 4096-entry directory: 183.05s (65536-entry directory not measured). after these five optimization patches: unpack and repack a 64-entry directory: 2.16s, a 1024-entry directory: 32.55s, a 4096-entry directory: 124.28s (a 65536-entry directory not measured). So this is a significant speed-up. Except wait -- 5s to open a directory with 4096 entries on an amd64 workstation, or 2 minutes to open it on a 266 MHz armel NAS? We can do better than that! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 8 21:43:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 04:43:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.a7e6b0a188b493d34e90d5419d82b1c8@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Okay, I went ahead and committed this in [3976]. Brian: I hope you sneak away from your vacation and log into the internet on your cell phone and review this patch before I release v1.5... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 09:40:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 16:40:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.e25adef058d655bd447afe4035145ebc@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): I think this is ok.. we're always using the same data with the same key, so no worries there.. we'd be enabling readers to mount a dictionary attack if we weren't already doing so by putting the readcap in the dirnode contents, so there's no increase in exposure there either. I think it would be more accurate to call this a "salt" instead of an "IV" though, since 1) we're using CTR mode, and 2) we're hashing this value to generate and AES key, not an IV. I don't know what I was thinking when I named that value "IV" in the original code. Also, looking at the code, we're using two hashes and not using the intermediate value anywhere. So how about using just one hash? Instead of: {{{ IV = hashutil.mutable_rwcap_iv_hash(self._node.get_writekey()) key = hashutil.mutable_rwcap_key_hash(IV, self._node.get_writekey()) cryptor = AES(key) }}} we could use something like: {{{ wk = self._node.get_writekey() key = hashutil.mutable_rwcap_key_hash2(wk) cryptor = AES(key) }}} where {{{mutable_rwcap_key_hash2}}} is a one-argument function that hashes a fixed prefix with the writekey. I too am surprised by the apparent speedup. Maybe the kernel context- switching time needed to read from /dev/urandom (or the refill-/dev/urandom code that gets invoked when you drain few kb from the entropy pool) is larger than the SHA256 code that gets invoked by the extra hash. In any case, using just one hash ought to be faster than either the original code or the attached patch. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 11:08:41 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 09 Jul 2009 18:08:41 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.4e2f0a24ed98123f013138428a75740f@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Whoops, I had forgotten that the name "IV" was wrong, and was misled by it into thinking it ought to be separate from the key. Your suggested patch sounds better to me. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 18:59:41 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 01:59:41 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.528f3fb8d5c2d5b706aeb8737cb177d0@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by kevan): Note that r3971 didn't actually commit the optimization code -- just the dictionary that it uses. If you want that, you should also apply the optimizations.txt patch and the tests.txt patch (which also contains tests for CachingDict). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 19:21:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 02:21:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.e12a05813c4a679c54f783cde7161200@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Oh how embarassing that I thought that was the optimization patch. I ''was'' actually going to ask you what you thought about the mysterious fact that [3971] seems to moderately speed up pack and unpack but not so much unpack-and-repack! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 9 20:24:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 03:24:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #329: dirnodes could cache encrypted/serialized entries for speed In-Reply-To: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> References: <038.d61ce46e4f33192f3f5ee311f9f6e832@allmydata.org> Message-ID: <047.ee27b173168b26831182b358912c67f7@allmydata.org> #329: dirnodes could cache encrypted/serialized entries for speed -------------------------------+-------------------------------------------- Reporter: warner | Owner: kevan Type: enhancement | Status: closed Priority: minor | Milestone: undecided Component: code-dirnodes | Version: 0.8.0 Resolution: fixed | Keywords: dirnode Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Now that I've applied your actual optimization patch, time to unpack and repack a 4096-entry directory dropped substantially. However, time to pack and time to unpack also changed, which suggests that my benchmark script is accidentally using your cache, or that extraneous factors on my system are screwing up the results or something. Before {{{optimizations.txt}}}: {{{ benchmarking N: 64, time: best: 0.08, 2th-best: 0.09, ave: 0.10, 2th- worst: 0.10, worst: 0.15 (of 5), reps/s: 9, ave rate: 632 N: 256, time: best: 0.35, 2th-best: 0.37, ave: 0.38, 2th- worst: 0.39, worst: 0.40 (of 5), reps/s: 2, ave rate: 675 N: 1024, time: best: 1.40, 2th-best: 1.42, ave: 1.63, 2th- worst: 1.74, worst: 1.88 (of 5), reps/s: 0, ave rate: 629 N: 4096, time: best: 5.29, 2th-best: 5.90, ave: 6.16, 2th- worst: 6.36, worst: 7.15 (of 5), reps/s: 0, ave rate: 665 benchmarking N: 64, time: best: 0.09, 2th-best: 0.09, ave: 0.09, 2th- worst: 0.09, worst: 0.09 (of 5), reps/s: 11, ave rate: 705 N: 256, time: best: 0.36, 2th-best: 0.36, ave: 0.39, 2th- worst: 0.37, worst: 0.51 (of 5), reps/s: 2, ave rate: 650 N: 1024, time: best: 1.38, 2th-best: 1.43, ave: 1.51, 2th- worst: 1.51, worst: 1.80 (of 5), reps/s: 0, ave rate: 676 N: 4096, time: best: 5.22, 2th-best: 5.77, ave: 6.08, 2th- worst: 6.33, worst: 6.79 (of 5), reps/s: 0, ave rate: 674 benchmarking N: 64, time: best: 0.10, 2th-best: 0.10, ave: 0.14, 2th- worst: 0.18, worst: 0.19 (of 5), reps/s: 7, ave rate: 473 N: 256, time: best: 0.36, 2th-best: 0.37, ave: 0.39, 2th- worst: 0.40, worst: 0.42 (of 5), reps/s: 2, ave rate: 664 N: 1024, time: best: 1.41, 2th-best: 1.42, ave: 1.46, 2th- worst: 1.48, worst: 1.52 (of 5), reps/s: 0, ave rate: 702 N: 4096, time: best: 5.22, 2th-best: 5.58, ave: 5.81, 2th- worst: 6.09, worst: 6.20 (of 5), reps/s: 0, ave rate: 705 }}} after {{{optimizations.txt}}}: {{{ benchmarking N: 64, time: best: 0.05, 2th-best: 0.05, ave: 0.06, 2th- worst: 0.06, worst: 0.07 (of 5), reps/s: 17, ave rate: 1091 N: 256, time: best: 0.21, 2th-best: 0.21, ave: 0.23, 2th- worst: 0.24, worst: 0.27 (of 5), reps/s: 4, ave rate: 1093 N: 1024, time: best: 0.83, 2th-best: 0.84, ave: 0.88, 2th- worst: 0.87, worst: 1.00 (of 5), reps/s: 1, ave rate: 1170 N: 4096, time: best: 3.29, 2th-best: 3.35, ave: 3.54, 2th- worst: 3.58, worst: 4.09 (of 5), reps/s: 0, ave rate: 1156 benchmarking N: 64, time: best: 0.06, 2th-best: 0.06, ave: 0.06, 2th- worst: 0.06, worst: 0.06 (of 5), reps/s: 17, ave rate: 1106 N: 256, time: best: 0.22, 2th-best: 0.22, ave: 0.22, 2th- worst: 0.22, worst: 0.23 (of 5), reps/s: 4, ave rate: 1158 N: 1024, time: best: 0.85, 2th-best: 0.85, ave: 0.86, 2th- worst: 0.87, worst: 0.88 (of 5), reps/s: 1, ave rate: 1188 N: 4096, time: best: 3.27, 2th-best: 3.28, ave: 3.36, 2th- worst: 3.38, worst: 3.60 (of 5), reps/s: 0, ave rate: 1218 benchmarking N: 64, time: best: 0.06, 2th-best: 0.06, ave: 0.07, 2th- worst: 0.09, worst: 0.10 (of 5), reps/s: 13, ave rate: 881 N: 256, time: best: 0.22, 2th-best: 0.22, ave: 0.28, 2th- worst: 0.26, worst: 0.46 (of 5), reps/s: 3, ave rate: 929 N: 1024, time: best: 0.85, 2th-best: 0.90, ave: 1.00, 2th- worst: 1.02, worst: 1.34 (of 5), reps/s: 0, ave rate: 1022 N: 4096, time: best: 3.25, 2th-best: 3.33, ave: 3.49, 2th- worst: 3.67, worst: 3.74 (of 5), reps/s: 0, ave rate: 1172 }}} I've applied your patches plus one from me to fix a conflict between one of my optimizations and one of yours. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From kpreid at mac.com Thu Jul 9 20:40:37 2009 From: kpreid at mac.com (Kevin Reid) Date: Thu, 9 Jul 2009 23:40:37 -0400 Subject: [tahoe-dev] Mac users: Publish your iCal calendars over Tahoe Message-ID: <282AB964-F9F1-4B01-AF20-C78866C53073@mac.com> How to publish iCal calendars over Tahoe: 1. Create a directory and get its http-URL cap, e.g. http://localhost:9797/uri/URI%3ADIR2%3Ablahblahblah/ 2. In the iCal publishing settings choose "Publish on: a Private Server". Paste the the aforementioned directory cap into the Base URL field. Enter anything in the Login and Password fields, because iCal insists on them. (Note also that if you reenter the publish dialog ("Change Location...") the port number will be missing. This is a bug in iCal.) 3. Get the read-only directory cap by going to the directory in a browser and clicking "Read-Only Version", e.g. http://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/ Append to it whatever you entered in the "Publish calendar as:" field, followed by ".ics". This is the URL to subscribe to. http://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/foo.ics (Or, if you aren't terribly annoyed by URL-scheme-abuse-as-disposition- instructions, that's webcal://localhost:9797/uri/URI%3ADIR2-RO%3Ablihblihblih/foo.ics ) (Note: I'm using a directory instead of a mutable file because iCal insists on having a url of the format /.ics, and I didn't immediately find anything in the tahoe web interface that would be in that format and also PUTable.) -- Kevin Reid From trac at allmydata.org Fri Jul 10 02:41:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 09:41:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance Message-ID: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ As we work on new encoding schemes (like DSA-based mutable files), I'm thinking that we want to put a lower bound on the cap/SI length to maintain reasonable margin against collisions. 256 bits would be more than enough. 128 is ok, but a bit tight. 92 bits would make me nervous. robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation before it) because, for something that is meant as a backup system, even the slightest possibility of the CHK-based indexing scheme mapping two documents to the same storage index was too high for him. (I believe he would be more satisfied with a scheme that used centrally-allocated guaranteed- unique storage indices, which we could do but would require more coordination and longer caps, since we could no longer use a randomly-generated readkey to derive the storage index. In exchange for a controllable but non-zero probability of collision, we get to avoid central coordination and use smaller caps). The specific places where collisions could occur are: * mapping from file contents to CHK-derived readkey * mapping from readkey (CHK-derived or randomly generated) to storage index * mapping from randomly-generated mutable writekey to storage index The "birthday paradox" determines the chance of collision. If I'm doing my math right, if you want less than '{{{p}}}' chance of getting any collisions when selecting items out of a namespace of size '{{{N}}}', then you can't select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" (where "surprise" would be a better term) because that square root causes C to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to C=19. In the Tahoe context, {{{C}}} is the number of files you can add to the grid. In the current case, our 128-bit storage index (N=2**128) means that p=0.5 gets us a nice large 2**64 number of files, except that p=0.5 is insufficient margin: we'd much prefer a vanishingly small chance of collision, like p=2**-64. Fortunately we get two bits of margin for every one bit we reduce from C. The table looks like: ||N||numfiles||prob(collision)|| ||96||2**48||2**-1 (0.5)|| ||96||2**40||2**-16|| ||96||2**32||2**-32|| ||96||2**24||2**-48|| ||128||2**64||2**-1 (0.5)|| ||128||2**56||2**-16|| ||128||2**48||2**-32|| ||128||2**32||2**-64|| ||256||2**128||2**-1 (0.5)|| ||256||2**96||2**-64|| Note that our {{{N}}} is the minimum of the storage-index size and the top-most cap value (i.e. the readkey for immutable files, or the writekey for mutable files). So a DSA-based mutable file with a 92-bit writecap gives us an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 or 256 bits. Also note that the allmydata.com grid currently has something like 10M objects in it, about C=2**23. So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap would be nice, it's too short to provide enough collision margin. I want to be able to put trillions of files into a grid, and I want a the chance of collision to be so small that I don't ever need to worry about it, and 96 bits isn't really there. 128 bits is probably good enough, but doesn't have enough margin to be obviously and unquestionably safe (C=2**32 is a lot of files but you can imagine people wanting more, p=2**-64 is a tiny probability but you can imagine people wanting a bit better). 256 would be plenty (but of course I want my filecaps to be shorter than that). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 07:14:43 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 14:14:43 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.c8e38efecda693263f3db9c04b64358b@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Thanks for this analysis. I like your comments at the end that we want a bit of "overkill" in number of files and chance of collision. People who don't want to rely on the collision-resistance of secure hash functions at ''all'' are probably not part of our market, but people who are willing to do so in principle, but would feel better with a nice fat margin of safety are definitely in our market. Note that if you generate a new write cap (private key) and then check and it turns out that the same write cap has been generated by someone else then now you have gained the ability to write to their mutable file or directory! That's why I have been thinking that 96-bits was too few for write caps. Originally I had been thinking something like "It would probably not be worthwhile for any attacker to spend 2^96^ computer power trying to forge a write cap.". But this way of thinking discounts two important factors: chance of success and number of targets. If there are 2^40^ writecaps in use, then if an attacker does a mere 2^36^ work (excluding the cost of checking whether each writecap that they generate is already out there), then they have a 2^-20^ chance of forging ''someone's'' writecap. (Thanks to Daniel J. Bernstein's papers and mailing list postings for helping me understand this important point. http://cr.yp.to/snuffle/bruteforce-20050425.pdf ) However, a storage-index collision doesn't sound nearly as serious to me. No integrity or confidentiality is necessarily lost due to storage-index collision, right? Well, it could complicate the question of "which servers are handling which shares of this mutable file" -- an issue that is already not well managed. Anyway, nowadays I think of storage-indexes as a separate layer built on top of the crypto layer. People can define their storage indexes as secure hashes of some pieces of the capabilities if they want, or choose random storage indices, or hierarchical ones based on the DNS names, or just not have storage indices at all and require every downloader to query every server. It shouldn't impact the security of the crypto layer, if the crypto layer includes integrity checking using the verifycap itself on the storage-server side. I think we should write a document called something like "crypto failure modes (What could possibly go wrong?)" that explains what the consequences are of each different type of failure. (As requested by Nathan: http://allmydata.org/pipermail/tahoe-dev/2009-April/001661.html .) The one discussed above is caused by "two people choose the same write cap (signing key) seed (possibly due to malice)". That one leads to an integrity failure, where one of the people thinks that they are the only one with a write-cap to a specific file or directory, but actually someone else also has the same write-cap. So I think that is the worst one (because I value integrity highly). Another similar integrity failure could come about from a failure of the digital signature algorithm -- i.e. if someone were able to forge digital signatures even without the writecap. (Note that a collision in the hash function used by the digital signature algorithm could cause this. People who don't want to rely on collision-resistance of secure hash functions at ''all'' can't even rely on rsa, dsa, ssh, ssl, or gpg, although I should hasten to add that those algorithms typically include some randomness in the input to their secure hash function, to make it that much harder for attackers to cause collisions.) After those integrity failures, there are confidentiality failures. The obvious one is someone being able to crack AES-128-CTR without having the readkey. Another one is if the content-hash-key encryption where to generate the same readkey for two different immutable files. I suppose that's another reason why using an {{{added convergence secret}}} is safer (although I hasten to add that I see no reason to distrust the collision- resistance of SHA-256d-truncated-to-128-bits at the present). Note that of course in practice the dangers from bugs and from operator error (i.e. misplacing your keys) are a lot greater than these algorithmic crypto risks. So much greater that the former are pretty much guaranteed to happen and the latter will probably never. Nonetheless, I value getting the crypto part right so that it is secure and also so that everyone who is willing to rely on crypto in principle is willing to rely on our crypto, so thanks for you help with this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 07:27:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 10 Jul 2009 14:27:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.2a61e7ade88853f48bbfd3a29371d39e@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): P.S. Oh, what I wrote about people choosing whatever storage index they want is possibly confused because I'm conflating two ideas into the word "storage index". Let's talk about those two uses as "server selector" and "file identifier". The former is how you decide which servers you're going to use (either when uploading a file for the first time or when using a cap to find a file that's already up there). The latter is how you tell a storage server which of the shares that it has are the shares that you are currently interested in. I currently think that the verify cap is a good thing to use for the latter role, but perhaps not for the former. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 10 22:48:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 05:48:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d626ddfe413e7f8ad4b64c6c22b2bd6e@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): I'm attaching a patch that implements + tests for the behavior described above. However, when testing, I came upon another maybe-bug. If, using the trunk build from last night, I do {{{ tahoe mv tahoe:file1 file1 }}} with the expectation that it will copy the remote file to my system, it purports to succeed (i.e.: no error message) but doesn't actually copy anything to my system, and unlinks the remote file. Is this what we want to happen? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 03:25:36 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 10:25:36 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.f3fe64637b23c67b7e4c95e8b79c2ab4@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Old description: > As we work on new encoding schemes (like DSA-based mutable files), I'm > thinking that we want to put a lower bound on the cap/SI length to > maintain > reasonable margin against collisions. 256 bits would be more than enough. > 128 > is ok, but a bit tight. 92 bits would make me nervous. > > robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation > before it) because, for something that is meant as a backup system, even > the > slightest possibility of the CHK-based indexing scheme mapping two > documents > to the same storage index was too high for him. (I believe he would be > more satisfied with a scheme that used centrally-allocated guaranteed- > unique > storage indices, which we could do but would require more coordination > and > longer caps, since we could no longer use a randomly-generated readkey to > derive the storage index. In exchange for a controllable but non-zero > probability of collision, we get to avoid central coordination and use > smaller caps). > > The specific places where collisions could occur are: > > * mapping from file contents to CHK-derived readkey > * mapping from readkey (CHK-derived or randomly generated) to storage > index > * mapping from randomly-generated mutable writekey to storage index > > The "birthday paradox" determines the chance of collision. If I'm doing > my > math right, if you want less than '{{{p}}}' chance of getting any > collisions > when selecting items out of a namespace of size '{{{N}}}', then you can't > select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" > (where "surprise" would be a better term) because that square root causes > C > to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to > C=19. > In the Tahoe context, {{{C}}} is the number of files you can add to the > grid. > > In the current case, our 128-bit storage index (N=2**128) means that > p=0.5 > gets us a nice large 2**64 number of files, except that p=0.5 is > insufficient > margin: we'd much prefer a vanishingly small chance of collision, like > p=2**-64. Fortunately we get two bits of margin for every one bit we > reduce > from C. The table looks like: > > ||N||numfiles||prob(collision)|| > ||96||2**48||2**-1 (0.5)|| > ||96||2**40||2**-16|| > ||96||2**32||2**-32|| > ||96||2**24||2**-48|| > ||128||2**64||2**-1 (0.5)|| > ||128||2**56||2**-16|| > ||128||2**48||2**-32|| > ||128||2**32||2**-64|| > ||256||2**128||2**-1 (0.5)|| > ||256||2**96||2**-64|| > > Note that our {{{N}}} is the minimum of the storage-index size and the > top-most cap value (i.e. the readkey for immutable files, or the writekey > for > mutable files). So a DSA-based mutable file with a 92-bit writecap gives > us > an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 > or > 256 bits. > > Also note that the allmydata.com grid currently has something like 10M > objects in it, about C=2**23. > > So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap > would be nice, it's too short to provide enough collision margin. I want > to > be able to put trillions of files into a grid, and I want a the chance of > collision to be so small that I don't ever need to worry about it, and 96 > bits isn't really there. 128 bits is probably good enough, but doesn't > have > enough margin to be obviously and unquestionably safe (C=2**32 is a lot > of > files but you can imagine people wanting more, p=2**-64 is a tiny > probability > but you can imagine people wanting a bit better). 256 would be plenty > (but of > course I want my filecaps to be shorter than that). New description: As we work on new encoding schemes (like DSA-based mutable files), I'm thinking that we want to put a lower bound on the cap/SI length to maintain reasonable margin against collisions. 256 bits would be more than enough. 128 is ok, but a bit tight. 92 bits would make me nervous. robk's friend Sylvan expressed concerns about Tahoe (and mnet/mojonation before it) because, for something that is meant as a backup system, even the slightest possibility of the CHK-based indexing scheme mapping two documents to the same storage index was too high for him. (I believe he would be more satisfied with a scheme that used centrally-allocated guaranteed- unique storage indices, which we could do but would require more coordination and longer caps, since we could no longer use a randomly-generated readkey to derive the storage index. In exchange for a controllable but non-zero probability of collision, we get to avoid central coordination and use smaller caps). The specific places where collisions could occur are: * mapping from file contents to CHK-derived readkey * mapping from readkey (CHK-derived or randomly generated) to storage index * mapping from randomly-generated mutable writekey to storage index The "birthday paradox" determines the chance of collision. If I'm doing my math right, if you want less than '{{{p}}}' chance of getting any collisions when selecting items out of a namespace of size '{{{N}}}', then you can't select more than {{{C = sqrt(2*N*p)}}} items. This is called a "paradox" (where "surprise" would be a better term) because that square root causes C to be surprisingly low: for birthdays (in which N=365), p=0.5 leads to C=19. In the Tahoe context, {{{C}}} is the number of files you can add to the grid. In the current case, our 128-bit storage index (N=2**128) means that p=0.5 gets us a nice large 2**64 number of files, except that p=0.5 is insufficient margin: we'd much prefer a vanishingly small chance of collision, like p=2**-64. Fortunately we get two bits of margin for every one bit we reduce from C. The table looks like: ||N||numfiles|| ||prob(collision)|| ||96||2**48||->||2**-1 (0.5)|| ||96||2**40||->||2**-17|| ||96||2**32||->||2**-33|| ||96||2**24||->||2**-49|| ||128||2**64||->||2**-1 (0.5)|| ||128||2**56||->||2**-17|| ||128||2**48||->||2**-33|| ||128||2**32||->||2**-65|| ||192||2**96||->||2**-1|| ||192||2**80||->||2**-33|| ||192||2**64||->||2**-65|| ||256||2**128||->||2**-1 (0.5)|| ||256||2**96||->||2**-65|| Note that our {{{N}}} is the minimum of the storage-index size and the top-most cap value (i.e. the readkey for immutable files, or the writekey for mutable files). So a DSA-based mutable file with a 92-bit writecap gives us an {{{N}}} of 2**92, even if it is expanded into a storage-index of 128 or 256 bits. Also note that the allmydata.com grid currently has something like 10M objects in it, about C=2**23. So, I'm thinking that as much as a nice short 96-bit DSA mutable writecap would be nice, it's too short to provide enough collision margin. I want to be able to put trillions of files into a grid, and I want a the chance of collision to be so small that I don't ever need to worry about it, and 96 bits isn't really there. 128 bits is probably good enough, but doesn't have enough margin to be obviously and unquestionably safe (C=2**32 is a lot of files but you can imagine people wanting more, p=2**-64 is a tiny probability but you can imagine people wanting a bit better). 256 would be plenty (but of course I want my filecaps to be shorter than that). -- Comment(by warner): as before, I think I'd like to continue using "storage index" for what you're calling the "file identifier", but yeah split out "server selector" or "peer-selection index" or some similar term for the purpose of determining which servers you're going to be talking to. One way of describing this would be "we used to use the storage-index as the peer-selection index, but these days we put two separate values in the filecap". I am also starting to think of these as separate concepts, but remember that we've yet to actually implement such a split. Sylvan's concern was about availability: he considered a backup system to be broken if its design has a built-in probability of file unrecoverability. It's easier to see the problem if we set the encryption-key and hash lengths to infinity, but restrict the storage index to say four bits. Then upload two files, and try to download one of them.. you've got a 1/16 chance of getting a download failure because your two files had the same storage-index, you downloaded the wrong bits, and now they won't pass the integrity check. Also, when we talk about this, we should be careful to distinguish between the failure modes of mutable files versus immutable files.. they're very distinct. And, collisions at different levels have very different consequences: if the storage index is too small, we'll get availability failures; if the immutable encryption key or mutable writekey is too small, we'll get confidentiality failures. I've been assuming that we'll keep the security parameters sufficiently large.. this ticket was specifically about the availability concerns resulting from a too-small storage index. If we compress the filecap by deriving the storage-index from the writekey, then clearly we're limited by {{{min(len(writekey),len(storage-index))}}}. Mostly I ticketed this issue because it's something I want to keep in mind as we design the next revision of the filecap format. If we don't already have a wiki page for it, I'll add one to organize our ideas.. I think they're currently spread across half a dozen tickets. I updated the table in the description: I think 192-bit caps will let us have an effectively-infinite number of files (2**64) with an effectively-zero chance of collision (2**-65). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:23:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:23:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #674: controlled access to your WUI In-Reply-To: <037.61baae0405033f54a68bcd066354261f@allmydata.org> References: <037.61baae0405033f54a68bcd066354261f@allmydata.org> Message-ID: <046.3a4f3df4508ab893b934394af8ff3988@allmydata.org> #674: controlled access to your WUI -------------------------+-------------------------------------------------- Reporter: zooko | Owner: nobody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.3.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Comment(by warner): good point. I suppose that means making the $WUI_SECRET pages come from a completely different origin, by using a separate port number or something. We'd still need $WUI_SECRET, of course. We'll have to have a rule that says we never emit unescaped external content on any page served below $WUI_SECRET . That means not using this namespace for delivering files or directories, and it also means being careful about escaping node nicknames and log messages and anything else that we might want to serve from that space. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:31:12 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:31:12 -0000 Subject: [tahoe-dev] [tahoe-lafs] #495: zfec crashes and incorrect precondition values In-Reply-To: <036.ff54e5856c0f9cdb10822c30c5bdfb32@allmydata.org> References: <036.ff54e5856c0f9cdb10822c30c5bdfb32@allmydata.org> Message-ID: <045.313f56345acbe2af07d39a3724f7f52d@allmydata.org> #495: zfec crashes and incorrect precondition values ------------------------+--------------------------------------------------- Reporter: greg | Owner: zooko Type: defect | Status: closed Priority: minor | Milestone: undecided Component: code | Version: unknown Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed * component: unknown => code Comment: This appears to be fixed in zfec-1.4.5 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 04:44:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 11:44:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #714: Not possible to renew leases from the WUI. In-Reply-To: <038.6fc6800acc928e5cdcc1648750d676f0@allmydata.org> References: <038.6fc6800acc928e5cdcc1648750d676f0@allmydata.org> Message-ID: <047.18158e3309b645d3981aae535ee269be@allmydata.org> #714: Not possible to renew leases from the WUI. -----------------------------------+---------------------------------------- Reporter: kpreid | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | -----------------------------------+---------------------------------------- Changes (by warner): * status: assigned => closed * resolution: => fixed * milestone: eventually => 1.5.0 Comment: done, in [3937], so it should be in 1.5.0 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 06:11:19 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 13:11:19 -0000 Subject: [tahoe-dev] [tahoe-lafs] #432: writing down filecaps: revise URI scheme In-Reply-To: <038.2bb9b2fab7dfea20676c145eca6fdd40@allmydata.org> References: <038.2bb9b2fab7dfea20676c145eca6fdd40@allmydata.org> Message-ID: <047.255569cba90686b8ef9709c62b2a34b4@allmydata.org> #432: writing down filecaps: revise URI scheme ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: task | Status: new Priority: major | Milestone: eventually Component: code-encoding | Version: 1.0.0 Keywords: caps | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): NewCapDesign is the place to design the new cap format. It contains a detailed list of design criteria. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 11:03:36 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 18:03:36 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.43305e40bb1caae498925ff30b9bd733@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): I was just trying to make my storage servers for [wiki:VolunteerGrid the volunteer grid] more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. So this is just a note to remind myself that I wish someone would implement this. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 11:12:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 18:12:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location Message-ID: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- I'm configuring a laptop which is behind a home NAT to have a specified {{{tub.location}}} which says the IP address of my home network. (As described in [source:docs/configuration.txt].) However, sometimes I take this laptop to other networks, and it would be nice if it would do the process of auto-detecting its current IP address and advertise both that ''and'' the manually-configured IP address that I am putting in, so that if clients can connect to it either way then they win. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 14:00:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 21:00:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information Message-ID: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information -------------------------------+-------------------------------------------- Reporter: zooko | Owner: warner Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- If I do a deep-check on a directory, I start getting results reported on the web page showing the files and subdirectories within that directory. Reloading (or waiting for the automatic self-reload) shows more and more results. Until one of the subdirectories is unrecoverable, in which case the web page containing the deep check results is replaced with a web page saying only this: {{{ UnrecoverableFileError: the directory (or mutable file) could not be retrieved, because there were insufficient good shares. This might indicate that no servers were connected, insufficient servers were connected, the URI was corrupt, or that shares have been lost due to server departure, hard drive failure, or disk corruption. You should perform a filecheck on this object to learn more. }}} To close this ticket, make it so that I can still see all the other result that have already been generated, plus further results about other files and subdirectories that haven't yet been checked, even while there is an unrecoverable subdirectory present. I'm using the current trunk: 1.4.1-r3982. Brian: are you willing to take this ticket? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 11 14:50:43 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 11 Jul 2009 15:50:43 -0600 Subject: [tahoe-dev] calling all VolunteerGrid operators Message-ID: Folks: If you are contributing a storage server to The VolunteerGrid, would you please update the VolunteerGrid page on the wiki and describe your server a little? Thanks. After this mail, which is going out to tahoe-dev, I'll probably send future mails about the VolunteerGrid only to people whose contact info is listed on that page. Regards, Zooko From trac at allmydata.org Sat Jul 11 15:56:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 22:56:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location In-Reply-To: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> References: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> Message-ID: <046.20cbce44f4693fd5521fc7f1a9b06b77@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): maybe we could use a magic value, like {{{tub.location = auto+1.2.3.4:1234}}}, which is interpreted to mean "merge {{{1.2.3.4:1234}}} with the auto-detected location". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 16:20:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 23:20:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #612: build binary packages of pysqlite In-Reply-To: <037.c2244a0bcbee3a7fb6fe0ac856505fe0@allmydata.org> References: <037.c2244a0bcbee3a7fb6fe0ac856505fe0@allmydata.org> Message-ID: <046.951885ef896789f5ce82b5319acf0e5e@allmydata.org> #612: build binary packages of pysqlite ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: undecided Component: packaging | Version: 1.2.0 Resolution: wontfix | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Changes (by zooko): * status: new => closed * resolution: => wontfix Comment: Since Python >= 2.5 comes with sqlite built in and since I'm hoping to drop support for Python 2.4 after the 1.5.0 release, I'm closing this ticket as wontfix. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 11 16:28:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 11 Jul 2009 23:28:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information In-Reply-To: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> References: <037.d0d825b95a13358bba667ecb3ad45769@allmydata.org> Message-ID: <046.f6ea262473f49ac4940020aded70f0a7@allmydata.org> #755: if there is an unrecoverable subdirectory, the web deep-check report loses other information ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Changes (by warner): * status: new => assigned * component: code-frontend-web => code-dirnodes Comment: yeah, I'll work on this. Basically traversal failures during a deep-check or deep-repair operation should increment a counter and move on, instead of throwing an exception and stopping the walker. I don't know if I can finish it in time for 1.5.0 though. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 03:21:43 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 10:21:43 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c078d69a8cda56882185c8bf057d9e49@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by warner): Hm, good question. "tahoe mv" was indeed meant for tahoe-to-tahoe moves, so that an unqualified "file1" really means "tahoe:file1". (the fact that "tahoe mv alias:file1 alias:file1" results in deleting a file is certainly a bug, probably in the dirnode code). I don't know if it's better to add docs/etc to teach people to expect that "tahoe mv" doesn't touch the local disk, or to add code/tests to make "tahoe mv" behave more like people's existing expectations. There's value in having all tahoe CLI commands that happen to overlap regular unix commands (cp, ln, mv, rm) be prepared to handle both local-disk and tahoe- filesystem arguments. But it also adds code, complication, and redundancy (e.g. why would you ever use "tahoe rm ~/.emacs" instead of regular rm?). Overall, I guess 'mv' should handle both local and tahoe-side files, and should behave like 'cp'. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 03:35:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 10:35:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.ac39bf40ae408a50fc2ccda8b205d12a@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): oh, I'm being stupid and the new code is bad. What we're doing is encrypting a list of child writecaps in such a way that the containing dirnode's writecap is required to retrieve them. Obviously each child writecap must be encrypted with a different key. It is reasonable to use the hash of the child writecap as a salt (since there's no requirement that the constant child writecap be encrypted differently each time), but it is not ok to use the hash of the parent dirnode writecap as a salt (because that's the same for all children). My suggested change is wrong too. The code needs to be more like: {{{ salt = hashutil.mutable_rwcap_iv_hash(rwcap) key = hashutil.mutable_rwcap_key_hash(salt, self._node.get_writekey()) crypto = AES(key) }}} I'll make this change soon. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 07:21:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 14:21:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.45dfcecb759e1003826585a2cd0d1b98@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): Whoops. Ouch. I'm sure glad you caught that! I was thinking when I wrote the patch that I was using the writecap of the entry as the input, but obviously I was using the key of the directory! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 07:24:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 14:24:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.4e59327da6bbf1dfd45a03760d3e5f8b@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by zooko): By the way, a more traditional way to do something like this ''would'' be to use the same key (the one for the dir) to encrypt each entry and use a unique IV for each entry. We are in the habit of instead generating a unique key for each thing we want to encrypt and typically just leaving the IV at 0, which seems fine to me, too. Your proposed fix is in the latter tradition. Please hurry up and commit it so that nobody uses trunk to write directories insecurely! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Sun Jul 12 11:31:17 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 12:31:17 -0600 Subject: [tahoe-dev] Is anyone using GridBackup? Message-ID: <200907121231.17312.shawn-tahoe@willden.org> I hope not, because I'm about to rev the read cap index format, and I'd rather not have to maintain backward compatibility, yet. I doubt anyone is, but I thought I'd better toss this out just in case. If you are using it, let me know. I still won't keep support for the previous format in the code, but I'll put together a script to update your read caps -- but only if it's really important to you. Going forward, I have infrastructure in place to allow the index format to evolve, so future changes will be able to maintain backward compatibility. Shawn. From trac at allmydata.org Sun Jul 12 16:09:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 23:09:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.3d3a709ec974f427a0816ba1953dff20@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: reopened Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Comment(by warner): Huh, I wasn't even aware of pycryptopp offering IV!=0. I'll test the patch tomorrow, but I'm not sure I'll be able to to commit it until about 24h from now. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 12 16:52:08 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 12 Jul 2009 23:52:08 -0000 Subject: [tahoe-dev] [tahoe-lafs] #750: deterministic IV for writecaps for dir entries In-Reply-To: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> References: <037.04402dae7d400e66041e033bd0979cef@allmydata.org> Message-ID: <046.543b7669887a86d5f8525534fbf7393d@allmydata.org> #750: deterministic IV for writecaps for dir entries -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: enhancement | Status: closed Priority: major | Milestone: 1.5.0 Component: code-dirnodes | Version: 1.4.1 Resolution: fixed | Keywords: confidentiality Launchpad_bug: | -------------------------------+-------------------------------------------- Changes (by warner): * status: reopened => closed * resolution: => fixed Comment: pushed, in [3984]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Sun Jul 12 17:45:32 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 18:45:32 -0600 Subject: [tahoe-dev] UEB hash size Message-ID: <200907121845.32680.shawn-tahoe@willden.org> Hi guys, What's the rationale for including the full 256-bit UEB hash in the CHK URI? Those URIs could be shortened considerably by truncating it to, say, 128 bits. How difficult would it be to allow Tahoe to operate with either full UEB hashes or abbreviated hashes? I'm thinking I could shorten my read cap index entries significantly if I could store only half of the UEB hash. Along those lines in the context of small, trusted grids, it might even be reasonable to omit the UEB hash entirely (in my case I have a separate content hash, and it should also be possible to hash the retrieved and decrypted file to regenerate the read key and check integrity that way, assuming an unchanged convergence secret). What is the bare minimum data needed to retrieve, reassemble and decrypt an immutable file? Just the AES read key? The web API doesn't like CHK URIs that conform to the full specification, but I'm sure that could be changed easily enough, if it made sense. Shawn. From trac at allmydata.org Sun Jul 12 18:02:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 01:02:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.033e3dad16b4dc934a1420c44e7a83fa@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:7 zooko]: > I was just trying to make my storage servers for [wiki:VolunteerGrid the volunteer grid] more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in. > > So this is just a note to remind myself that I wish someone would implement this. :-) I've fiddled a little bit with miniupnpc, which ghazel already mentioned. It's a C library but includes a Python wrapper. It's pretty simple. The following code was all it took to tell my gateway (a Linux box running linux-igd) to set up a forwarding for my Tahoe node: {{{ import miniupnpc u = miniupnpc.UPnP() u.discoverdelay = 200 u.discover() u.selectigd() external_port = 3456 r = u.getspecificportmapping(external_port, 'TCP') while r != None and external_port < 65536: external_port += 1 r = u.getspecificportmapping(external_port, 'TCP') u.addportmapping(external_port, 'TCP', u.lanaddr, 3456, "Tahoe %u" % external_port, '') }}} Obviously, that needs a lot of error checking, but it's still really straightforward. I ran that script from the command line, and watched the log on my gateway to verify that the mapping was set up. I then verified that I can connect to my local Tahoe server from outside. Works fine. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Sun Jul 12 18:37:12 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 19:37:12 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907121845.32680.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> Message-ID: <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> I really don't like the idea of adding an insecure mode, if we can help it. Could you explain more -- perhaps even quantify -- what impact the size of the cap has on your system? Hopefully there's a solution which is both secure and efficient. Regards, Zooko From trac at allmydata.org Sun Jul 12 19:18:55 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 02:18:55 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.ac182060dc9195fbdaf7c7f1d82efe02@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): Wait, if you you have 4-bit storage indexes, ''and servers verify integrity of shares'', then you don't get an unrecoverable file, you get a failure to upload the second one, right? I really like the idea of servers verifying the integrity of shares, because without that then even if you have a nice fat storage index, the first server that you upload to could quickly turn around and upload garbage to all the other servers, under the same storage index that you just told him. A failure to upload (backup) is a much better problem to have than a failure to download (recover), as long as it can be detected, understood, and fixed by the operators. I.e. "Oh, we need to add servers", or "Oh, we need to start using longer storage indexes". :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Sun Jul 12 19:50:43 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 20:50:43 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907121845.32680.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> Message-ID: <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> And in answer to your questions: On Jul 12, 2009, at 18:45 PM, Shawn Willden wrote: > What's the rationale for including the full 256-bit UEB hash in the > CHK URI? Those URIs could be shortened considerably by truncating > it to, say, 128 bits. It is that the integrity of an immutable file cap is the "exactly one file matches this cap" guarantee. To ensure this requires 2K bits in the immutable cap to guarantee K bits of security, because of a birthday-surprise attack in which an attacker generates two (or more) files with the same immutable file cap so that they have the ability to undetectably swap in the alternate files substituted for the original file, after they've distributed the cap to other people. To generate such files requires only about K bits of work to find multiple matching files for a 2K-bit immutable file cap. > How difficult would it be to allow Tahoe to operate with either > full UEB hashes or abbreviated hashes? It is a neat idea. We've discussed it before, but I can't find the reference. I seem to recall that Brian had a good summary of the risk of publishing a shortened immutable cap. Perhaps he just pointed out that in the future people may come to distrust whether the file that they get by retrieving with that cap is really the only file that could have matched. If your shortened cap is sufficiently, let's say 192-bits, this risk doesn't sound like a big issue as far as brute computer power goes -- even if people in the future have vastly improved computation technology, 2^96 computations will probably still be very, perhaps even prohibitively, expensive. However, the possibility of people uncovering algorithmic weaknesses in the hash algorithm that we are using (currently SHA-256d, hopefully in the future SHA-3) can reduce the effective strength. By the way, I'm sitting on a good idea that I haven't finished writing up yet for how to combine the encryption key and the integrity-checking hash together so that you have only one value (perhaps of size 256 bits) instead of two values -- one for the key and one for the hash. Perhaps that would solve most of your performance issues? As I mentioned in my previous mail, I'd like to understand more about what the performance implications are in GridBackup. > What is the bare minimum data needed to retrieve, reassemble and > decrypt an immutable file? Just the AES read key? That, and some way to find the shares, which we currently call the "storage index". That would omit not only the integrity check on the ciphertext (to guarantee that the immutable cap you started with could match only one file) but also the integrity check on the shares (to identify which servers are responsible for serving up corrupted shares, in the case that the resulting file was corrupted). Regards, Zooko From shawn-tahoe at willden.org Sun Jul 12 19:56:31 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 20:56:31 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> References: <200907121845.32680.shawn-tahoe@willden.org> <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> Message-ID: <200907122056.31646.shawn-tahoe@willden.org> On Sunday 12 July 2009 07:37:12 pm Zooko Wilcox-O'Hearn wrote: > I really don't like the idea of adding an insecure mode, if we can > help it. Could you explain more -- perhaps even quantify -- what > impact the size of the cap has on your system? Hopefully there's a > solution which is both secure and efficient. I'm not suggesting an "insecure" mode; I don't think reducing the UEB hash size from 32 bytes to 16 bytes (or even smaller) affects security at all. In fact, as far as I can see the UEB hash is entirely redundant -- it's a sort of an integrity optimization that allows shares to be checked more quickly, but as far as ensuring the integrity of the file as a whole, it's redundant, because the read key is derived from the content hash, so you can use it to verify integrity. And including the whole 32 bytes is further redundant, when a truncated version would do just as well. As for the impact of the size, the smaller the URI data, the more read cap index entries I can pack into an index file in the grid-based burst trie (without making the files so big they're slow to upload and retrieve), and since those files have to get uploaded and downloaded frequently, tighter packing improves performance. Shawn. From zooko at zooko.com Sun Jul 12 20:31:28 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 12 Jul 2009 21:31:28 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <200907122056.31646.shawn-tahoe@willden.org> References: <200907121845.32680.shawn-tahoe@willden.org> <7EC1975D-387F-4A4C-8410-649B3659A3EA@zooko.com> <200907122056.31646.shawn-tahoe@willden.org> Message-ID: On Jul 12, 2009, at 20:56 PM, Shawn Willden wrote: > In fact, as far as I can see the UEB hash is entirely redundant -- > it's a sort of an integrity optimization that allows shares to be > checked more quickly, but as far as ensuring the integrity of the > file as a whole, it's redundant, because the read key is derived > from the content hash, so you can use it to verify integrity. Hey -- you're starting to come up with the same sort of thing that I came up with as I mentioned in my previous note -- a way to do both decryption and integrity checking on the file with only one "strong crypto worth" of bits. :-) However, you're not quite there yet, because the downloader wouldn't have the ability to verify the integrity of the individual shares, only of the final plaintext, so if the integrity check fails you wouldn't know which server(s) were serving corrupted shares. Also that scheme wouldn't allow incremental integrity checking of the file before the whole thing was downloaded. Finally, that would not work if the uploader used a randomly-chosen (not-content-hash-key) or an added-convergence secret. Still, I like the way you think... > As for the impact of the size, the smaller the URI data, the more > read cap index entries I can pack into an index file in the grid- > based burst trie (without making the files so big they're slow to > upload and retrieve), and since those files have to get uploaded > and downloaded frequently, tighter packing improves performance. Hm, this is definitely the kind of reason why I greatly value small capabilities. Can you characterize it as something like "the difference between a 24-byte immutable read cap and a 32-byte and a 64-byte is X times as many trie node fetches for an average file download"? Regards, Zooko From shawn-tahoe at willden.org Sun Jul 12 20:51:19 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 21:51:19 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> References: <200907121845.32680.shawn-tahoe@willden.org> <7BEC9350-51E5-4EB5-921E-3174C8719B72@zooko.com> Message-ID: <200907122151.20033.shawn-tahoe@willden.org> On Sunday 12 July 2009 08:50:43 pm Zooko Wilcox-O'Hearn wrote: > By the way, I'm sitting on a good idea that I haven't finished > writing up yet for how to combine the encryption key and the > integrity-checking hash together so that you have only one value > (perhaps of size 256 bits) instead of two values -- one for the key > and one for the hash. I can see several ways to do that, and it makes a lot of sense, particularly since in the case of a content-hash-key, the key is another integrity verifier, as well as a decryption key. Not so in the case of a random key, of course. Given a content-hash-key, I think the combined key and UEB hash could be even shorter than 256 bits. 192 bits would give you strong integrity assurance (since all of the 192 bits are used for that), as well as strong encryption (128 bits). > Perhaps that would solve most of your > performance issues? As I mentioned in my previous mail, I'd like to > understand more about what the performance implications are in > GridBackup. Me too. So far, I just know that read cap index uploading is significantly slowing backups. Most of this is due to the fact that the burst trie structure consists of a large number of files, and they're mutable, which makes them slower. I chose to make them mutable because they do mutate, but perhaps it would help to make them immutable. If I could reduce the size of an index entry by 30%, I could reduce the need to update trie nodes by the same amount. For the moment I've switched to uploading the indices as a set of sorted flat files, and I just have to binary search each of them in turn when looking for a specific cap. That makes uploading fast and searching slow (somewhat; the files are cached locally, so the search is performed locally). I think after I've accumulated a few hundred thousand entries in the flat files, I'll then push them all into the trie. Bulk updates of the trie are fairly quick on a per-entry basis. In general, I view the read cap index as overhead and I'd like to minimize it. Currently, the CHK URI portion of each record consumes 60 of 78 total bytes, of which only the 16-byte AES key is really essential for retrieval. > > What is the bare minimum data needed to retrieve, reassemble and > > decrypt an immutable file? Just the AES read key? > > That, and some way to find the shares, which we currently call the > "storage index". But the SID is derived from the read key, so that's implied. > That would omit not only the integrity check on the > ciphertext (to guarantee that the immutable cap you started with > could match only one file). Well, if you regenerated the read key from the plaintext and compared with the original read key, you'd know that it matches with very high probability. The exact probability depends on the number of files in the system, but even with trillions of files, the odds of a false match are on the order of winning Powerball twice in a row. > but also the integrity check on the shares > (to identify which servers are responsible for serving up corrupted > shares, in the case that the resulting file was corrupted). If the shares have different UEBs, it tells you which of them is wrong. In the case of any sort of random corruption, you'd be able to tell that anyway. If the UEBs are the same, then the UEB contains the information to tell you which share is corrupted. Assuming the read key is the primary integrity verification mechanism (and I know that's not the assumption, but it could be, and it works), then the birthday paradox is irrelevant with respect to the UEB hash, so it only needs to be long enough to resist an attacker's attempt to produce an otherwise-valid UEB with the same abbreviated hash, i.e. a second pre-image attack against a tagged SHA-256d hash (abbreviated). Shawn. From shawn-tahoe at willden.org Sun Jul 12 21:25:42 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 12 Jul 2009 22:25:42 -0600 Subject: [tahoe-dev] UEB hash size In-Reply-To: References: <200907121845.32680.shawn-tahoe@willden.org> <200907122056.31646.shawn-tahoe@willden.org> Message-ID: <200907122225.43008.shawn-tahoe@willden.org> On Sunday 12 July 2009 09:31:28 pm Zooko Wilcox-O'Hearn wrote: > Can you characterize it as something like "the difference between a > 24-byte immutable read cap and a 32-byte and a 64-byte is X times as > many trie node fetches for an average file download"? Actually, it's not an issue on the download side. The trie structure ensures that only a single trie node will be downloaded (even when the local cache is empty; with a fully-populated cache, there will be no Tahoe traffic). The issue is on the upload side, and it's a little complicated to quantify, as it depends heavily on the trie parameters and the "fullness" of trie nodes, which determines the number of trie nodes that need to be updated to add a set of entries. I don't fully understand the mathematical characteristics of the burst trie yet -- the one paper I found on the structure didn't really analyze that in detail, and my intuition about node fullness has proven to be way off -- nodes seem to be on average much less full that I'd expected, meaning there are more nodes than I'd expected. If there are m nodes (including empty nodes on already-burst levels), and n entries are inserted and if none of the nodes has to "burst", then on average it will be necessary to update m(1-(1-1/m)**n) nodes. As m grows, that tends toward n. For small m, it tends toward m. What I can't do is characterize m as a function of the number of entries in the trie and the set of trie parameters (bits per level and max entries per node). I wish I could, because besides helping me quantify this issue, it would probably also let me optimize the choice of trie parameters. I guess I need to sit down and think hard about how the trie behaves as it grows. It seems like it should be easy to get a handle on... Shawn. From trac at allmydata.org Mon Jul 13 05:37:16 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 12:37:16 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.a292665351a053e79e43595821d22934@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 07:02:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 13 Jul 2009 14:02:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.9ddcebc1eee9f16a9191090706b8334f@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:9 zooko]: > Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-) Writing tests for it might be challenging, since good tests would require the presence of a UPnP IGD to respond to the registration/de-registration, which would require providing a rudimentary (at least) IGD daemon in the code base just for testing. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From rufus.pollock at okfn.org Mon Jul 13 09:29:31 2009 From: rufus.pollock at okfn.org (Rufus Pollock) Date: Mon, 13 Jul 2009 17:29:31 +0100 Subject: [tahoe-dev] How does file deletion work? Message-ID: How does file deletion work? webapi.txt says: > DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME [...] > Note that this does not actually delete the file or directory that the name > points to from the tahoe grid -- it only removes the named reference from > this directory. If there are other names in this directory or in other > directories that point to the resource, then it will remain accessible > through those paths. Even if all names pointing to this object are removed > from their parent directories, then someone with possession of its read-cap > can continue to access the object through that cap. Right: so deleting from a directory means unlinking from that directory. If it is linked from elsewhere it won't be deleted and even if i unlink from all directories if there's a read-cap out there it will still stay so how do I *really* delete it for sure: > The object will only become completely unreachable once 1: there are no > reachable directories that reference it, and 2: nobody is holding a read- > or write- cap to the object. (This behavior is very similar to the way > hardlinks and anonymous files work in traditional unix filesystems). But how do I (or tahoe) know if someone is holding a read or write-cap? (And when we say someone we can't mean anything I've passed the read-cap to, we've got to mean something that's connected to the grid like another node right?) Also how quickly once all read or write-caps have gone does deletion happen (is this a GC thing?). Thanks in advance for any help. Rufus Pollock From shawn-tahoe at willden.org Mon Jul 13 11:14:12 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 13 Jul 2009 12:14:12 -0600 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: Message-ID: <200907131214.12791.shawn-tahoe@willden.org> On Monday 13 July 2009 10:29:31 am Rufus Pollock wrote: > Right: so deleting from a directory means unlinking from that > directory. If it is linked from elsewhere it won't be deleted and even > if i unlink from all directories if there's a read-cap out there it > will still stay so how do I *really* delete it for sure You can't! > But how do I (or tahoe) know if someone is holding a read or > write-cap? (And when we say someone we can't mean anything I've passed > the read-cap to, we've got to mean something that's connected to the > grid like another node right?) Tahoe can't in general know about all read or write caps held outside of Tahoe dirnodes. If you're holding an unlinked read cap which you don't want to get deleted, you have to periodically contact the servers holding the shares to "renew the lease". Without that, the servers will probably eventually reclaim the space. > Also how quickly once all read or write-caps have gone does deletion > happen (is this a GC thing?). It's a GC thing, and I'm not sure what the state of the GC implementation is. I probably should have just let someone who does know respond. Shawn. From ludo at gnu.org Mon Jul 13 12:50:50 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Mon, 13 Jul 2009 21:50:50 +0200 Subject: [tahoe-dev] How does file deletion work? References: <200907131214.12791.shawn-tahoe@willden.org> Message-ID: <873a90mkf9.fsf@gnu.org> Hi Shawn Willden writes: > On Monday 13 July 2009 10:29:31 am Rufus Pollock wrote: [...] >> But how do I (or tahoe) know if someone is holding a read or >> write-cap? (And when we say someone we can't mean anything I've passed >> the read-cap to, we've got to mean something that's connected to the >> grid like another node right?) > > Tahoe can't in general know about all read or write caps held outside of Tahoe > dirnodes. If you're holding an unlinked read cap which you don't want to get > deleted, you have to periodically contact the servers holding the shares > to "renew the lease". Without that, the servers will probably eventually > reclaim the space. Does that mean that storage servers are able to list the capabilities contained in a directory object? Thanks, Ludo'. From shawn-tahoe at willden.org Mon Jul 13 13:42:08 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 13 Jul 2009 14:42:08 -0600 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: <873a90mkf9.fsf@gnu.org> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> Message-ID: <200907131442.08602.shawn-tahoe@willden.org> On Monday 13 July 2009 01:50:50 pm Ludovic Court?s wrote: > Does that mean that storage servers are able to list the capabilities > contained in a directory object? Hehe. Hadn't thought that through quite far enough. No, the storage servers can't know what caps are in the directories. So those leases will have to be renewed periodically, too. Shawn. From trac at allmydata.org Mon Jul 13 19:01:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 02:01:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.076ef86795c023679c6e563abe565549@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: undecided => 1.5.0 Comment: Hm, I guess we should look into this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 19:02:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 02:02:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.0a3048016366d2980d709f60f3d4cd36@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Soultcer just reported the same thing on IRC. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 20:46:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 03:46:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" Message-ID: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Jonathan Ellis says that he was telling a co-worker: "There are a lot of docs on Tahoe, but there's nothing that says ''which operations are efficient''." -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:19:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:19:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maxmimum recursion depth exceeded in Tahoe2PeerSelector Message-ID: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> #758: maxmimum recursion depth exceeded in Tahoe2PeerSelector --------------------------------+------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- I just got this traceback from a node using the volunteergrid: {{{ /usr/local/lib/python2.6/dist- packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py, line 328 in _runCallbacks 326 self._runningCallbacks = True 327 try: 328 self.result = callback(self.result, *args, **kw) 329 finally: Locals callback > self args (, set([19, 20]), []) kw {} /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 384 in _got_response 382 383 # now loop 384 return self._loop() 385 Locals self /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 284 in _loop 282 self.contacted_peers.extend(self.contacted_peers2) 283 self.contacted_peers[:] = [] 284 return self._loop() 285 else: Locals self /home/volunteergrid/src/tahoe/src/allmydata/immutable/upload.py, line 284 in _loop 282 self.contacted_peers.extend(self.contacted_peers2) 283 self.contacted_peers[:] = [] 284 return self._loop() 285 else: Locals self }}} (And so forth until maximum recursion depth exceeded.) There are only 15 servers on the volunteergrid right now. The clause that is shown, around [source:src/allmydata/immutable/upload.py#L279 279 of upload.py] is for the case that all servers have been asked to hold a share, and then all servers have been asked to hold a second share, and this clause is to iterate and go on to ask them to hold yet a third-or- greater share. It appears that this loop never terminated before the recursion depth was exceeded. We have [source:src/allmydata/tahoe/test/test_upload.py at 20090625021809-4233b- 9cdbf53c54025466fea8ab97bed668cd0017b142#L483 tests of this case], but... Hey waitaminute! That code in upload.py says: {{{ elif self.contacted_peers2: # we've finished the second-or-later pass. Move all the remaining # peers back into self.contacted_peers for the next pass self.contacted_peers.extend(self.contacted_peers2) self.contacted_peers[:] = [] return self._loop() }}} That can't be right. It probably means to say: {{{ self.contacted_peers.extend(self.contacted_peers2) del self.contacted_peers2[:] }}} Why does that test catch this bug? But it is too late at night for me to be messing with such stuff. If someone in a different timezone or a different sleep schedule wants to fix the test to catch this bug while I sleep, that would be great! :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:34:17 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:34:17 -0000 Subject: [tahoe-dev] [tahoe-lafs] #651: handle MemoryError by failing quickly and loudly In-Reply-To: <037.129716934a290747156300e1e2c25ac5@allmydata.org> References: <037.129716934a290747156300e1e2c25ac5@allmydata.org> Message-ID: <046.0158885167c0586a2f0e4f06036e2b86@allmydata.org> #651: handle MemoryError by failing quickly and loudly --------------------------+------------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-mutable | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): People are reporting !UncoordinatedWriteErrors on the test grid (#748), and so I checked some of the test grid storage servers and saw that one -- tahoebs5.allmydata.com bs5c1 -- has 1.5 GB of virt, 475 MB res. That is running some very old versions of foolscap and TahoeLAFS: {{{ allmydata-tahoe: 1.3.0-r3747, foolscap: 0.3.2, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.26, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c8 }}} Hm... Yep, that one has !MemoryError in its twisted.log. I wonder if we could monkeypatch Python's !MemoryError class to do a {{{sys.exit()}}} or the like. I suppose this also suggests that we ought to be pushing this issue upstream to Python itself... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 13 21:36:25 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 04:36:25 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.4453153a4c189abb2be34fecf5bd8878@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Okay, so this could be caused by one or more of the following known issues: #651 (handle !MemoryError by failing quickly and loudly) (I added a note about this issue on that issue just now, #548 (mutable publish sends queries to servers that have already been asked), #547 (mapupdate(MODE_WRITE) triggers on a false boundary), #546 (mutable-file surprise shares raise inappropriate UCWE), #540 (inappropriate "uncoordinated write error" after handling a server failure). I will investigate more in the evening after work and next weekend. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:09:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:09:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #760: the command to create a server is named "create-client" Message-ID: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> #760: the command to create a server is named "create-client" ----------------------------+----------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- I just had to explain to someone on the phone that the way to create a server is to run the command 'create-client'. Maybe we should rename that command to 'create-node'? It is used for creating clients, storage servers, and web gateways, but not for creating introducers. Alternately I suppose we could split it into several commands: 'create-client', 'create-storage-server', 'create-web-gateway'? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:10:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:10:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" In-Reply-To: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> References: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> Message-ID: <046.098df38048e1a27f4794ec7a20a4ceeb@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by zooko): To close this ticket, add notes to [wiki:Performance the Performance page] about the general efficiency of all of the major operations, and link to the Performance page from [source:docs/frontends/webapi.txt the webapi.txt doc]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 07:11:10 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 14 Jul 2009 14:11:10 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.67f58e10fc5c7451ad9413f1e46945bf@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Attached is the foolscap logtool incident report. The critical excerpt of that report looks like this: {{{ # 05:32:47.692 [22866716]: entering loop # 05:32:47.693 [22866717]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.693 [22866718]: we are planning to push new seqnum=#1 # 05:32:47.693 [22866719]: 10 queries still outstanding # 05:32:47.697 [22866720]: 10 shares sent # 05:32:47.744 [22866724]: _got_write_answer from qvqv7jmm # 05:32:47.744 [22866725]: entering loop # 05:32:47.745 [22866726]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.745 [22866727]: we are planning to push new seqnum=#1 # 05:32:47.745 [22866728]: 9 queries still outstanding # 05:32:47.746 [22866729]: _got_write_answer from 6fyx5u4z # 05:32:47.746 [22866730]: entering loop # 05:32:47.747 [22866731]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.747 [22866732]: we are planning to push new seqnum=#1 # 05:32:47.747 [22866733]: 8 queries still outstanding # 05:32:47.748 [22866734]: _got_write_answer from trjdor3o # 05:32:47.748 [22866735]: entering loop # 05:32:47.749 [22866736]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.749 [22866737]: we are planning to push new seqnum=#1 # 05:32:47.749 [22866738]: 7 queries still outstanding # 05:32:47.750 [22866739]: _got_write_answer from fcmlx6em # 05:32:47.751 [22866740]: entering loop # 05:32:47.751 [22866741]: current goal: before update: , sh0 to [trjdor3o], sh1 to [xiktf6ok], sh2 to [jfdpabh3], sh3 to [z2unqjbp], sh4 to [qvqv7jmm], sh5 to [fcmlx6em], sh6 to [6fyx5u4z], sh7 to [ivjakubr], sh8 to [wfninubk], sh9 to [lwkv6cji] # 05:32:47.751 [22866742]: we are planning to push new seqnum=#1 # 05:32:47.752 [22866743]: 6 queries still outstanding # 05:32:47.753 [22866744]: _got_write_answer from fcmlx6em * 05:32:47.753 [22866745]: WEIRD our testv failed, so the write did not happen [INCIDENT-TRIGGER] * 05:32:51.776 [22866746]: somebody modified the share on us: shnum=5: I thought they had #1:R=7ahx, but testv reported #1:R=7ahx }}} Hm, so far I don't understand how the client was surprised by this chain of events. But, now it is time to go to work so I'll wonder about it later. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From rufus.pollock at okfn.org Tue Jul 14 09:01:00 2009 From: rufus.pollock at okfn.org (Rufus Pollock) Date: Tue, 14 Jul 2009 17:01:00 +0100 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: <200907131442.08602.shawn-tahoe@willden.org> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: Thanks for the responses. So as I understand it: 1. a node will keep (its chunks of) a file based on renewal requests. 2. If these don't come in the file will be marked for garbage collection. 3. If a file is listed in a directory then this will lead automatically to renewal of the relevant leases 4. However if it is no longer linked from any directory explicit renewal requests would have to be sent. 5. So by unlinking a file from all directories, in the absence, of any explicit renewal requests, it will be deleted from the grid. Does this sound right? Rufus 2009/7/13 Shawn Willden : > On Monday 13 July 2009 01:50:50 pm Ludovic Court?s wrote: >> Does that mean that storage servers are able to list the capabilities >> contained in a directory object? > > Hehe. ?Hadn't thought that through quite far enough. ?No, the storage servers > can't know what caps are in the directories. > > So those leases will have to be renewed periodically, too. > > ? ? ? ?Shawn. > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > -- Join our "100 supporters pledge": http://www.pledgebank.com/support-okfn Promoting Open Knowledge in a Digital Age - http://www.okfn.org/ From david at triendl.name Tue Jul 14 09:19:15 2009 From: david at triendl.name (David Triendl) Date: Tue, 14 Jul 2009 18:19:15 +0200 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: <20090714161915.GA5130@triendl.name> On Tue, Jul 14, 2009 at 05:01:00PM +0100, Rufus Pollock wrote: > 1. a node will keep (its chunks of) a file based on renewal requests. Yes, storage nodes keep their data until all leases expired (typically they expire after 30 days). > 2. If these don't come in the file will be marked for garbage collection. The storage nodes regularly perform scans of all data and all data with no leases (leases are stored together with chunk data) will be removed. > 3. If a file is listed in a directory then this will lead > automatically to renewal of the relevant leases No, the files in the directory need to have their leases renewed as well. This can simply be done with a single deep-check with &add-lease=true call, which will reach all directories and files beneath it and renew their leases at the currently reachable servers. > 4. However if it is no longer linked from any directory explicit > renewal requests would have to be sent. Explicit renawal requests always need to be sent, if garbage collection is enabled on the storage servers. > 5. So by unlinking a file from all directories, in the absence, of any > explicit renewal requests, it will be deleted from the grid. Yes, but you can also just abandon the whole directory, which will yield the same results. Cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090714/9ac0409c/attachment.pgp From ludo at gnu.org Tue Jul 14 16:27:47 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Wed, 15 Jul 2009 01:27:47 +0200 Subject: [tahoe-dev] How does file deletion work? References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090714161915.GA5130@triendl.name> Message-ID: <87eisi26bw.fsf@gnu.org> Hello, David Triendl writes: > On Tue, Jul 14, 2009 at 05:01:00PM +0100, Rufus Pollock wrote: [...] >> 5. So by unlinking a file from all directories, in the absence, of any >> explicit renewal requests, it will be deleted from the grid. > Yes, but you can also just abandon the whole directory, which will yield the > same results. Or, if you don't trust storage servers for being available, or for faithfully implementing the lease mechanism, anything may vanish at any time. :-) Thanks, Ludo'. From trac at allmydata.org Tue Jul 14 18:24:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 01:24:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #760: the command to create a server is named "create-client" In-Reply-To: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> References: <037.e72f7e20da581c3e2f1829f78450d89b@allmydata.org> Message-ID: <046.0feb8c36d2bb91380d823fdae7102225@allmydata.org> #760: the command to create a server is named "create-client" ----------------------------+----------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- Comment(by warner): My plan was to rename it 'create', and then add some options to specify which services you want the new node to provide. Since we want to do away with the Introducer eventually, I plan to retain 'create-introducer' as a separate command (rather than add 'tahoe create --introducer' and then get rid of it again later). The default node will have client functionality (i.e. the ability to upload and download files), a webapi server, and maybe storage functionality (I forget if we currently default to --with- storage or --no-storage). But yeah, 'create-node' has the benefits of * being longer, since creating a new client/server node is an infrequent operation and thus, according to Zipf's Law, deserves a longer name * being harder to interpret as intended for creating other things, like directories, or files I'm always on the fence as to what word to use for this client node thingy of ours. In a distributed-system sense, "node" is the right word: it's just one node out of many. If you think of things in asymmetric terms and really want to emphasize the difference between "clients" and "servers", then the node you're creating might have one or both features: sometimes you are creating a client-only node, sometimes a server-only node, and (more commonly) you are creating something that can do both. "node" has the bad quality of also being the most convenient term for elements of the file/directory DAG structure, but there we usually lean towards the longer terms "filenode" and "dirnode" (and only use the unqualified term when we need a plural: "nodes" instead of "filenodes/dirnodes" or "file/directory nodes"). So, I don't know what's the best term for the "instance of the Tahoe program (with working directory and saved state and a persistent nodeid) that you're creating for eventual launch with 'tahoe start'" thingy which would serve as the direct object in "tahoe create-XYZ". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 18:30:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 01:30:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient" In-Reply-To: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> References: <037.2012ab23a8b8616b12969fd0f90b1ef5@allmydata.org> Message-ID: <046.1f2276b581311e021047dff226365936@allmydata.org> #757: there isn't a doc that says "which operations are efficient" ---------------------------+------------------------------------------------ Reporter: zooko | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: undecided Component: documentation | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): Also, include a copy of these notes in the source tree somewhere. Adding docs to the wiki is great and all, but I'd like people to be able to grab a source tree and know that they don't need to get anything else. The biggest benefits (in my mind) of having docs on the wiki is that 1) everybody can edit them, add questions, etc, and 2) you can reference them from other wiki pages with nice short !WikiNames. The benefit of having docs in the source tree are 1) long-term persistence, 2) no version skew between the tahoe version you're looking at and the docs, and 3) local availability despite allmydata.org and/or trac being down. Maybe each file in docs/ could have a corresponding wiki page, and the file could have a note at the bottom saying "for the latest version of this document, please see URL". And we could write a script that would fetch all the docs and copy them into the source tree. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:34:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:34:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #753: use longer storage index / cap for collision resistance In-Reply-To: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> References: <038.6687183be55155531626ce3bd6b5d629@allmydata.org> Message-ID: <047.14aacb083aebc19d4268203ab97bf7ea@allmydata.org> #753: use longer storage index / cap for collision resistance ---------------------------+------------------------------------------------ Reporter: warner | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-encoding | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): Well, "It Depends". Current servers do no checking of shares. (I agree, I like server-side share-verification, except for the load and complexity it adds to the storage servers) == Immutable Files == Current code has uploaders quietly+simplisticly believing servers who say "I already have a share by that SI", without additional checking. (they don't search very far for existing shares, but if the server list is the same as it was for the first file, they'll get complete overlap). This results in the second file appearing to upload correctly but being completely unrecoverable (all shares for that SI are for the first file, so a download of the second file will get shares that fail the hash checks and fail). The uploader that we haven't written yet will probably search further for existing shares (especially once it's seen evidence of one), and will do some amount of verification of those shares (at the very least pulling down the UEB to make sure it's for the same file). This uploader will see evidence of an SI collision and probably fail the upload (with an exception that basically says "please pick a different SI, perhaps a random one"). Unless, of course, you're unlucky and wind up talking to a different servers during the two uploads. The subsequent downloader will either succeed but notice a lot of "corrupted" shares (if they can talk to the second uploader's servers) or fail due to a lot of corrupted shares (if they can't). If we further improve the servers to verify the integrity of immutable shares upon receipt, and we change the protocol to split server-selection-index from storage-index, and assume that the SI is required to be derived from the verifycap (so that servers can validate the share all the way up to the SI), then we're still in the same boat as the previous paragraph: servers who don't yet have a share will accept either the first upload or the second (since we're assuming a too-short SI, and these two files are colliding), and if the second upload succeeded due to disjoint peersets, then second-file downloaders will succeed or fail depending upon which servers they're able to talk to. == Mutable Files == We get the same issues here: if the storage-index is too small, then there will be multiple RSA keys which map to the same SI. A validating server which has accepted a share for key1 will reject an update for key2 (because it's looking at the full pubkey inside the share), but other servers might accept key2 (if they don't already have a share). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:51:05 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:51:05 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.d78f331b13587674eeba6d5bd9f3cb19@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): We need someone to review Kevan's patch. Do it now and this patch can go into TahoeLAFS v1.5! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:57:27 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:57:27 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.c7a289efb8eab4563c9b529aad537e89@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: 1.5.0 Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Comment(by zooko): [20090715025814-92b7f-d4af644430e5daef6d6ad57cc550c8faceaeb2cf] fixes "* make the Title and/or Label of the Directory page indicate that it is part of TahoeLAFS", and "* Change the wording of the 'Attach' form", but not "* Add a hyperlink with anchor text "What is this?" next to the "mutable" check-box which explains what mutable files are and how you never want to use them. Hm, maybe this also means hide the "mutable" check-box behind some sort of expert-user-mode? :-/ " -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 19:58:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 02:58:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #691: improve WUI directory page according to a new user's first impressions In-Reply-To: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> References: <038.df1297854a483c83f62c2f9c1fefac0b@allmydata.org> Message-ID: <047.c70c263d88230cb487ad5d43908cc6fc@allmydata.org> #691: improve WUI directory page according to a new user's first impressions ------------------------------------------+--------------------------------- Reporter: kpreid | Owner: zooko Type: enhancement | Status: assigned Priority: minor | Milestone: eventually Component: code-frontend-web | Version: 1.4.1 Keywords: first impressions, directory | Launchpad_bug: ------------------------------------------+--------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: We're not going to finish this for v1.5. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 21:09:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:09:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.0e721b68d60064598187b9878415835c@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings --------------------+------------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------+------------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Not urgent for TahoeLAFS v1.5, so I'm bumping it. Note that aside from the general "curiousness" that our code (or more likely our unit tests) rely on assertion-checking to be correct, turning on Python's {{{-OO}}} optimization mode might reduce CPU load a bit. I thought I observed that when benchmarking for #329/#752, but I'm not sure if it was real or an artifact of noisy benchmarking. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Tue Jul 14 22:29:41 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Tue, 14 Jul 2009 23:29:41 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: Message-ID: Folks: Not unexpectedly, my getting a day job and Brian going off on vacation (or "gallivanting", as I prefer to call it) has slowed down development of TahoeLAFS. v1.5 is late! I decided today that I really need to punt certain things, starting with the Macintosh packaging that I've been working on (tickets #762 and #763). Looking at the rest of The RoadMap (http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.5.0 ), most of these can be bumped from v1.5 once we've properly investigated the ones that *might* be critical dangers or regressions from v1.4.1... (Time passes as Zooko updates various tickets...) Okay, I've bumped #566 (after doing something to improve the situation), #691 (after doing something to improve it), #755. That leaves #705 (thanks, Kevan!), #748 (needs to be investigated a bit -- probably not critical and probably not a regression), #758 (should be easy to fix), #668 (ugh -- shall we punt this one?). Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/566 # host tahoe dependencies http://allmydata.org/trac/tahoe/ticket/668 # easy_install allmydata-tahoe is broken http://allmydata.org/trac/tahoe/ticket/691 # improve WUI directory page according to a new user's first impressions http://allmydata.org/trac/tahoe/ticket/705 # "tahoe mv" unlinks the target even when it is a directory http://allmydata.org/trac/tahoe/ticket/748 # Exception: when trying to create directory on testgrid http://allmydata.org/trac/tahoe/ticket/749 # Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings http://allmydata.org/trac/tahoe/ticket/755 # if there is an unrecoverable subdirectory, the web deep-check report loses other information http://allmydata.org/trac/tahoe/ticket/758 # maximum recursion depth exceeded in Tahoe2PeerSelector http://allmydata.org/trac/tahoe/ticket/762 # build of mac package fails due to pkgutil in zetuptoolz http://allmydata.org/trac/tahoe/ticket/763 # build of mac package fails due to some mysterious reason (not pkgutil) From toby.murray at comlab.ox.ac.uk Wed Jul 15 00:47:33 2009 From: toby.murray at comlab.ox.ac.uk (Toby Murray) Date: Wed, 15 Jul 2009 08:47:33 +0100 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: Message-ID: <1247644053.7331.0.camel@toby-laptop> On Tue, 2009-07-14 at 23:29 -0600, Zooko O'Whielacronx wrote: > That leaves #705 (thanks, Kevan!), #748 (needs to be investigated a > bit -- probably not critical and probably not a regression), I don't understand. How is not being able to create mutable files or directories on the testgrid not a regression nor critical? Cheers Toby From zookog at gmail.com Wed Jul 15 06:27:34 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 15 Jul 2009 07:27:34 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: <1247644053.7331.0.camel@toby-laptop> References: <1247644053.7331.0.camel@toby-laptop> Message-ID: Well, I guess it's the difference between bugs in the source code itself that we are about to ship as "v1.5" versus operational issues that effect TestGrid specifically versus operational issues that might typically effect other users. If it is not a new bug in the source code since the v1.4.1 then it wouldn't be a "regression" (people who are happily using v1.4.1 would not have a reason to refrain from upgrading to v1.5). If it is an operational issue that strikes rarely, doesn't cause great harm, and is easy to work around then it isn't "critical". (This part is, of course, a judgment call.) The investigation isn't complete, but so far it looks like the situation on the test grid where you can't create new directories is due to some combination of: 1. Limitations in the code that were already present in the v1.4.1 release (therefore not a regression): ticket #540 2. The limitation is that the testgrid web gateway (http://testgrid.allmydata.org:3567 ) is not handling misbehavior by some of the storage servers. That's a bug, but it isn't probably won't affect lots of users. It can be "worked around" by fixing your storage servers. 3. The misbehaving storage servers are running TahoeLAFS-v1.3-r3747, which is older than the current stable v1.4.1 release. It's possible (but again, without a complete investigation I don't know if it is true) that the cause of the MemoryError in the storage server has been fixed since then. So, I think the next step are: 1. Investigate more. Are any other storage servers besides tahoebs5.allmydata.com bs5c2 misbehaving? Do the munin graphs of bs5c2 show any interesting pattern in memory usage or other statistics? 2. Upgrade bs5c2 and reboot it, probably making TestGrid usable again. 3. ? Maybe experiment with adding some sort of kludge to hard-shutdown in case of MemoryError. I'm about to do #2, even though I don't want doing so to interfere with #1, and then I need to go to work. :-) Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/540 # inappropriate "uncoordinated write error" after handling a server failure From zookog at gmail.com Wed Jul 15 06:40:11 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 15 Jul 2009 07:40:11 -0600 Subject: [tahoe-dev] v1.5 status In-Reply-To: References: <1247644053.7331.0.camel@toby-laptop> Message-ID: Aha! And it turns out that upgrading and rebooting the storage servers on tahoebs5 don't change the failure. See ticket #748. Regards, Zooko tickets mentioned in this mail: http://allmydata.org/trac/tahoe/ticket/748 # Exception: when trying to create directory on testgrid From trac at allmydata.org Tue Jul 14 21:46:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:46:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #761: "tahoe cp $DIRCAP/$PATH $LOCAL" raises AttributeError Message-ID: <037.ffdb4fcafac540e99a10cc22625d14b0@allmydata.org> #761: "tahoe cp $DIRCAP/$PATH $LOCAL" raises AttributeError -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: eventually Component: code-frontend-cli | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- {{{tahoe cp --help}}} suggests that I try: {{{ You can also use a dircap as either FROM or TO target: tahoe cp URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html ./ # copy Zooko's wiki page to a local file }}} But when I try it, I get an exception: {{{ $ time ~/playground/allmydata/tahoe/trunk-hashedformat/bin/tahoe cp -r --node-url http://testgrid.allmydata.org:3567 URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html foo Traceback (most recent call last): File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1-r3989', 'console_scripts', 'tahoe')() File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/cli.py", line 429, in cp rc = tahoe_cp.copy(options) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 759, in copy return Copier().do_copy(options) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 444, in do_copy self.try_copy() File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 501, in try_copy return self.copy_to_directory(sources, target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 661, in copy_to_directory self.copy_files_to_target(self.targetmap[target], target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 692, in copy_files_to_target self.copy_file_into(source, name, target) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 737, in copy_file_into target.put_file(name, f) File "/data2/homezooko/playground/allmydata/tahoe/trunk- hashedformat/src/allmydata/scripts/tahoe_cp.py", line 163, in put_file pathname = os.path.join(self.pathname, name) File "/usr/lib/python2.5/posixpath.py", line 60, in join if b.startswith('/'): AttributeError: 'NoneType' object has no attribute 'startswith' }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 21:49:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 04:49:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #566: host tahoe dependencies In-Reply-To: <037.69ed23fb905f81e8424d28c6af94c997@allmydata.org> References: <037.69ed23fb905f81e8424d28c6af94c997@allmydata.org> Message-ID: <046.e07ead24aef5c7ec4dce7850ef533a6a@allmydata.org> #566: host tahoe dependencies -------------------------+-------------------------------------------------- Reporter: zooko | Owner: zooko Type: enhancement | Status: assigned Priority: major | Milestone: eventually Component: packaging | Version: 1.2.0 Keywords: | Launchpad_bug: -------------------------+-------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Okay I've downloaded the packages from the Tahoe test grid to http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs and [20090715025246 -92b7f-e7058e13ddc5be2d2538f46584b144674fce0045] adds that directory to the {{{find_links}}} in [source:setup.cfg]. Bumping this ticket out of v1.5 Milestone. To close this ticket, find a TahoeLAFS grid that is reliable and fast enough to be a component that builds of TahoeLAFS can rely upon, and upload all those packages to that grid, and change setup.cfg to point to it. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 22:09:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 05:09:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #762: build of mac package fails due to pkgutil in zetuptoolz Message-ID: <037.f1f9893c2d352042b94846d24d7182e1@allmydata.org> #762: build of mac package fails due to pkgutil in zetuptoolz ----------------------------+----------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.4.1 Keywords: mac setuptools | Launchpad_bug: ----------------------------+----------------------------------------------- http://allmydata.org/buildbot/builders/Mac%20OS-X/builds/1039/steps/mac_exe/logs/stdio {{{ File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/modulegraph/modulegraph.py", line 114, in find_module if isinstance(importer, pkg_resources.ImpWrapper): AttributeError: 'module' object has no attribute 'ImpWrapper' }}} This has something to do with the fact that the version of {{{zetuptoolz}}} bundled with TahoeLAFS in [source:misc/dependencies], called 'setuptools-0.6c12dev' replaced pkg_resources with the new pkgutil module, and modulegraph apparently uses pkg_resources. Why modulegraph's use thereof results in this incompatibility I haven't figured out yet. I have tested a version of zetuptoolz which uses the old pkg_resources style instead of the new pkgutil style -- http://allmydata.org/trac/zetuptoolz/changeset/20090703214517-92b7f- 126e534865d9104679dc424aa681cf26a26f2921 -- and shown that this error goes away with that zetuptoolz (this is why this failure doesn't happen on ootles the way it does on the "OS X" (!VirtualZooko) builder). So a good solution to this ticket would be to upgrade our bundled zetuptoolz to that new version of zetuptoolz. Although it would be nice to understand more precisely why it doesn't work this way -- maybe we could submit a patch to some other project to help. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 22:14:42 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 05:14:42 -0000 Subject: [tahoe-dev] [tahoe-lafs] #763: build of mac package fails due to some mysterious reason (not pkgutil) Message-ID: <037.bb3c128de40c25a67f89084a893ec7f6@allmydata.org> #763: build of mac package fails due to some mysterious reason (not pkgutil) -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: new Priority: major | Milestone: eventually Component: packaging | Version: 1.4.1 Keywords: mac | Launchpad_bug: -----------------------+---------------------------------------------------- I applied a patch to my local version of zetuptoolz on ootles to get past the problem described in #762. Now the "build a Mac package" step completes on ootles: http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/230/steps/dist/logs/stdio but the (new! improved!) "test the resulting Mac package" steps fails: http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/230/steps/testdmg/logs/stdio The "test the resulting Mac package" step could hopefully be amended to emit more diagnostics in this case... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 14 23:50:18 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 06:50:18 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.642238bd8fa1f70ee9a09e381944f8ef@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings ------------------------+--------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed * milestone: eventually => 1.5.0 Comment: I had some time, so I fixed it, in [3993]. All were cases where we intentionally do something wrong so as to trigger an (assertion) exception and then check the resulting behavior. All were fixed by replacing the !AssertionError with something more specific. The best rule to follow is probably this: * if an exception is worth testing, it's part of the API * !AssertionError should never be part of the API OTOH, it's a bother to create a whole new exception type and add an "if" clause (and then fret about whether the code-coverage stats are going to go down) just to do something that should be highly encouraged like adding preconditions and internal consistency checks. Maybe it means we're being overzealous in our unit tests and exercising things that we shouldn't or don't need to exercise. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:09:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:09:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.4f0bcb9bd72cf377525b9603a578d71e@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): since zooko found a !MemoryError on one of the testgrid servers, this is most likely another instance of #540. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:12:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:12:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #759: bundle newer pycryptopp into SUMO packages In-Reply-To: <037.02df60c5465488587cd253350a674c69@allmydata.org> References: <037.02df60c5465488587cd253350a674c69@allmydata.org> Message-ID: <046.0c5bf437bc93f321e040b9a3b5c7092a@allmydata.org> #759: bundle newer pycryptopp into SUMO packages ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Changes (by warner): * status: assigned => closed * resolution: => fixed Comment: Done. desert-island-test is now green. Note that we haven't actually built tarballs for a month, since one of the "Supported Builders" has gone AWOL, and the buildbot is configured to wait for all supported builders to finish a build before creating tarballs or debs. I plan to remove that coarse dependency and replace it with something smaller.. not sure what yet. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:15:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:15:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maximum recursion depth exceeded in Tahoe2PeerSelector In-Reply-To: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> References: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> Message-ID: <046.39220b9f2ef8f67bdd6861a490c0f427@allmydata.org> #758: maximum recursion depth exceeded in Tahoe2PeerSelector --------------------------------+------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------------+------------------------------------------- Comment(by warner): Huh, yeah, that code !!!is!!! odd.. your analysis feel right, but I'm too jetlagged to understand this code right now either. I want to rewrite the uploader anyways, but that's not going to happen for 1.5. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 00:21:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 07:21:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.d7f02322b0f58b099f0017dc5bd781da@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): does anyone know what percentage of home users have functional UPnP firewalls? I seem to remember reading somebody's report on the p2p-hackers mailing list that suggested that a large fraction (half?) of the boxes which claimed to support UPnP were in fact completely broken. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 05:43:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 12:43:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.7939aa57650ee40e6cd52fd9dc9bdea6@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): http://zgp.org/pipermail/p2p-hackers/2006-June/003880.html Alex Pankratov gave that 50% number on p2p-hackers in 2006. Around the same time allmydata.com implemented UPnP in their proprietary product (the one that preceded Tahoe) and measured the success rate among allmydata.com customers. Unfortunately I don't have or remember the numbers. We should ask Jim McCoy and Rob Kinninmont if they have or remember the results. I assume that deployment and robustness of UPnP has improved since then. ;-) Because of foolscap's nice design of using multiple connection hints in parallel and requiring strong SSL authentication, a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 05:49:42 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 12:49:42 -0000 Subject: [tahoe-dev] [tahoe-lafs] #759: bundle newer pycryptopp into SUMO packages In-Reply-To: <037.02df60c5465488587cd253350a674c69@allmydata.org> References: <037.02df60c5465488587cd253350a674c69@allmydata.org> Message-ID: <046.9843dc0db86da8339fbef1209fafeeb9@allmydata.org> #759: bundle newer pycryptopp into SUMO packages ---------------------------+------------------------------------------------ Reporter: zooko | Owner: warner Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): The reason it didn't build tarballs all that time was because one of the supported builders was red -- the 'mac-exe' builder on ootles. http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac- amd64%2010.4/builds/210/steps/mac_exe/logs/stdio Please don't unliterally change that code -- that's the implementation of our Buildbot Policy http://allmydata.org/trac/tahoe/wiki/BuildbotPolicy . If you think the policy should be different, let's talk about it first. In other words, I intended for it not to build tarballs all that time, because that way people who downloaded a tarball during that month would be getting one that was produced from source code that passed all the supported builders, including the 'mac-exe' builder. If that's too long to go with no new packages, then maybe this means 'mac- exe' should be removed from Supported. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 06:38:40 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 13:38:40 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.f768e9abc4867e4cf660ec598ff3c0bb@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): I upgraded and restarted the tahoebs5 storage servers, but I still get an UCWE from testgrid.allmydata.org:3456. Attached are two incident reports: -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Wed Jul 15 08:18:07 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 15 Jul 2009 09:18:07 -0600 Subject: [tahoe-dev] buildbot status update: Green! But where is Dan's ArchLinux? In-Reply-To: <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Message-ID: Folks: All the Supported Buildbots are green! Dan: you warned us when you signed up to run a Supported buildbot that the machine would come and go every day or so. That worked out fine for a while, but now it has stayed gone for about four days. Is this normal? Regards, Zooko From trac at allmydata.org Wed Jul 15 09:49:31 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 16:49:31 -0000 Subject: [tahoe-dev] [tahoe-lafs] #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings In-Reply-To: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> References: <037.7d13ad6e218f00534751fc2d00faf1e9@allmydata.org> Message-ID: <046.364ab74109539f1a8bf62c4c4b501393@allmydata.org> #749: Tahoe-LAFS fails unit tests when the "-OO" flag is passed to Python to optimize and strip docstrings ------------------------+--------------------------------------------------- Reporter: zooko | Owner: somebody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------+--------------------------------------------------- Comment(by swillden): I'm no expert on unit testing, but I think it makes sense that assertions (preconditions, postconditions, etc.) are things that don't need to be tested. I see them not as code which must be validated for correctness, but instead as code that does correctness validation, like the unit tests. Said another way, assertions and other design-by-contract artifacts are actually tests. They're just tests that are normally run all the time rather than only when we choose to explicitly invoke a test run. If you need to write test code to test your assertions, don't you also need to write test code to test your unit tests? And test code to test the test code that tests your tests, and... (try saying that five times fast) :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:04:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:04:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.a42d3cecc5c1a8d9b21b2cc9fbe37d51@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by warner): > I assume that deployment and robustness of UPnP has improved since then. ;-) Thanks for the reference. I admire your confidence :) > a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right? correct. Worst case, the node thinks it is externally visible but is not. Implementation-wise, we probably need some improvements in the overall !IntroducerClient behavior, since the current code assumes that it can get its location hints quickly at startup, and that they never change after that. We need something to handle changes to the location hints at runtime (including the initial determination, which usually requires running /sbin/ifconfig to compute and is thus asynchronous) and then send a new announcement to the Introducer each time. Foolscap is not entirely happy with changing location hints either: FURLs are computed once at startup (specifically at {{{tub.registerReference}}} time), and not updated afterwards, so we'd need some more work to make that behave smoothly. We should probably implement #445 (relay) and #50 (STUNT) eventually, too. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:23:50 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:23:50 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.5a8cab15163b603b347d33b610716527@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): > Thanks for the reference. I admire your confidence :) Waaataminute, by "admire" you mean "mock", don't you? As to the testing question that Shawn raised, I think we could write a fake IGD daemon in the test code, for example one which just returns a canned answer in response to the UPnP "are you there" and another canned answer in response to the UPnP "can I have an external port please?", and so on for all the UPnP IGD client functionality that we need. That is slightly complicated (especially if the UPnP IGD client we are using uses irreproducible randomness in its queries and expects the server to echo that randomness), but probably not trickier than the other "network server fakery" that is already in the TahoeLAFS unit tests. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 10:27:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 17:27:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.e7b2f61d8ef19a11694e4633420d9d68@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by zooko): Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as {{{miniupnp}}}, right? I'm not sure about this. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner-tahoe at allmydata.com Wed Jul 15 10:49:06 2009 From: warner-tahoe at allmydata.com (Brian Warner) Date: Wed, 15 Jul 2009 10:49:06 -0700 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> Message-ID: <20090715104906.251b900b@luther> Note that the default server configuration is conservative and does not perform any GC at all. You must explicitly enable anything which threatens data safety like GC. When you enable it, you will provide the expiration timeout, which of course determines how quickly your server's users must renew their leases. Basically, since we have a distributed filesystem (which makes reference counting expensive at best) with least-authority access semantics (which makes it impossible for the servers to count references themselves), the easiest and most reliable approach to deletion is to use leases, timers, and garbage collection. > 3. If a file is listed in a directory then this will lead > automatically to renewal of the relevant leases Nope, not without client involvement. As other folks have pointed out, servers can't read directories (since they don't hold a readcap). So clients (who *do* hold a readcap, or hold one to a parent directory, which gives them transitive access) must be responsible for performing renewals. (if you do a periodic "tahoe deep-check --add-lease ROOT:", then your statement becomes true, because your client is doing the work of finding everything reachable from that directory and renewing the leases, and merely adding/removing a file to any reachable directory is enough to start/stop renewing its lease) Explicitly renewing every single reachable file/directory from the client side is kind of a drag, as it means you (as the client) have a lot of work to do. Part of our designs on Accounting are hoping to reduce this work, but of course there's a funky tradeoff between effort (bandwidth), storage overhead, and safety. The secrets you need to renew (or cancel) a lease are generated in a way that makes them least-authority: there is a short string you could give to someone that would enable them to renew the lease on a single file for you (and not do anything else). Without any format changes (merely new code), we could build something that would let you pass a list of "lease-renewal caps" to a helper service of some sort, and have them renew your leases for you. This was the model we envisioned for the allmydata.com use case, where end users were not expected to have to do work to keep their files alive. The "tahoe manifest" command is part of this. There is also a "master lease renewal secret" which is used to compute the per-share renewal secrets: someone who has that master secret and a file's verifycap can renew the lease on its shares. One idea we've had is to mark all your leases with an account-id of some sort (perhaps just a unique number), and refresh a timer on the account-id instead of on every individual share. Basically you'd be saying "yes, I'm still alive, please keep my files alive too", and sending one message per month instead of one for every file and directory you care about. Lot less bandwidth that way. But that would only reclaim space in large chunks (or never reclaim space at all, if the user sticks around). If you're confident that you can enumerate all the files and directories that you care about, you can periodically compare this manifest against a previous version, and then send out explicit lease-cancel message for the objects that are no longer on the list. (the "cancel lease XYZ" message is the closest we've got to actual server-side deletion). But note that if you get it wrong (perhaps due to race conditions between two machines modifying a shared directory), you could accidentally lose files. Adding one lease per starting point (i.e. per root directory) per walker instance feels like it might avoid the race worries. I suspect that the best approach will be to combine this manifest-delta cancellation thing with the account-id-based timer. If clients are charged/accountable for how much space they're using, then they'll be motivated to make a reasonable-to-them tradeoff between the work done to build/compare these manifests, lease timers, and safety. Another idea (ticket #308) is to change the encryption format of dirnodes to introduce another level into the "writecap -> readcap -> verifycap" hierarchy. The new order would be "writecap -> readcap -> traversalcap -> verifycap", and a traversalcap would give somebody the ability to get the traversalcaps of all children (as well as the verifycap of the dirnode itself). Then, if you gave a hypothetical Lease Renewal Service the traversalcap of your root directory (as well as your master lease-renewing secret), they could renew all of your leases, but couldn't actually read your directories or files. (this requires "semi-private DSA keys", see http://allmydata.org/trac/pycryptopp/ticket/13 for details). You might be willing to reveal the shape of your directory structure to this service, in exchange for letting it take responsibility for your lease-renewal duties. If the service lives network-wise close to the storage servers, it may be considerably faster too. cheers, -Brian From glucnac at gmail.com Wed Jul 15 19:02:17 2009 From: glucnac at gmail.com (dan McNair) Date: Wed, 15 Jul 2009 21:02:17 -0500 (CDT) Subject: [tahoe-dev] buildbot status update: Green! But where is Dan's ArchLinux? In-Reply-To: References: <11A788B9-6EB2-40D9-92A0-F72BC414B34B@zooko.com> <27784642-5459-4370-88AF-1BCCEDC92DA7@zooko.com> Message-ID: On Wed, 15 Jul 2009, Zooko Wilcox-O'Hearn wrote: > Dan: you warned us when you signed up to run a Supported buildbot that the > machine would come and go every day or so. That worked out fine for a while, > but now it has stayed gone for about four days. Is this normal? I think I have a configuration oopsie. I recently changed how things that run on my machine all the time are started. I'll make sure the buildbut is in functioning order. Sorry! From trac at allmydata.org Wed Jul 15 15:54:52 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 15 Jul 2009 22:54:52 -0000 Subject: [tahoe-dev] [tahoe-lafs] #668: easy_install allmydata-tahoe is broken In-Reply-To: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> References: <040.d636b2f61e3a495f5f25f6f9ea61e163@allmydata.org> Message-ID: <049.1d9a4ee1d92d3adac2900981580253f5@allmydata.org> #668: easy_install allmydata-tahoe is broken -----------------------+---------------------------------------------------- Reporter: francois | Owner: zooko Type: defect | Status: assigned Priority: major | Milestone: eventually Component: packaging | Version: 1.3.0 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => eventually Comment: Sigh. I hate to allow this issue to remain unfixed, but there really isn't a good reason to hold up the TahoeLAFS v1.5 release on it. The official install instructions http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know, and all of the "flunkOnFailure" building/packaging buildsteps on all of the SupportedBuilders are passing. Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does ''not'' currently work, I hereby bump this ticket out of the v1.5 Milestone. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 19:22:02 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 02:22:02 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c6ae2c1d39a71b9a8944463eb036d0b9@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Changes (by zooko): * keywords: reliability review => reliability Comment: The comment "# we should probably pick some output that is more informative, and # put that here" can be removed from the test patch. Hm, the implementation works by using {{{urllib.urlopen()}}} to send a query for the metadata ({{{?t=json}}} and then test whether the target exists and if so whether it is a directory. If that test passes then it goes ahead and does the {{{HTTP PUT}}} to overwrite the target. The problem with this is that there is a race condition, also known as a TOCTTOU ("Time Of Check To Time Of Use") issue, where the object under the target name may be non-existent or be a file at the time the check happens, but be a directory when the subsequent PUT happens. A safer implementation would extend the semantics of the PUT to tell the webapi server "except don't do it if the target turns out to be a directory". Hm, I guess 'tahoe mv' really ought to be using {{{POST /uri/$DIRCAP/[SUBDIRS../]?t=rename}}} anyway instead of {{{PUT}}}. Hm, I see that that {{{POST}}} command has an undocumented (in [source:docs/frontends/webapi.txt]) 'replace' option indicating whether it should replace if there is already a child under the target name or abort. [source:src/allmydata/web/directory.py at 20090715025814-92b7f- d4af644430e5daef6d6ad57cc550c8faceaeb2cf#L327] I guess the right implementation of this ticket is to extend that webapi command with a {{{replace=only_files}}} option which will abort if there is a child under the target name and that child is a directory. Unsetting the 'review' keyword. Kevan: what do you think? I don't want to punt this issue out of 1.5 because it is a potentially data-losing ui issue. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Wed Jul 15 21:03:06 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Wed, 15 Jul 2009 22:03:06 -0600 Subject: [tahoe-dev] simple authority taxonomy versus a kind of privacy In-Reply-To: <20090715104906.251b900b@luther> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090715104906.251b900b@luther> Message-ID: <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> Dear Brian and tahoe-dev: I was wondering about the word "traversal cap", thinking "Isn't that what one might call a 'deep verify cap'?". Then I thought "Hey wait a second, all of our directory caps are deep.". (I'm glad about this fact -- I think that all directory access rights being transitive is simpler than having both transitive and non-transitive, while being expressive enough for our needs.) Then I thought "Hm, so why don't we just make Directory Verify Caps deep?". The answer has to do with privacy in the sense of traffic-analysis-resistance, but I don't think it is a win. More on that below. Here is a set of four figures which I hope illustrate why I hold a high value for having few types of capabilities and having their meanings be mutually orthogonal and consistent: for *conceptual simplicity*. immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Verify Cap Figure 1: taxonomy of capabilities in TahoeLAFS v1.5 If we added traversal cap to directories, it would become: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Traversal Cap Verify Cap Figure 2: taxonomy of capabilities with Traversal Cap ("Traversal Cap" in this figure could also be named "Deep Verify Cap".) If instead we changed directory verify caps to always be deep, then it would be: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap directories: Deep Write Cap Deep Read Cap Deep Verify Cap Figure 3: taxonomy of capabilities with all dir caps deep If we were then to add immutable directories (ticket #607), then it would become: immutable files: Read Cap Verify Cap mutable files: Write Cap Read Cap Verify Cap immutable directories: Deep Read Cap Deep Verify Cap mutable directories: Deep Write Cap Deep Read Cap Deep Verify Cap Figure 4: taxonomy of capabilities with immutable dirs To me, Figures 3 and 4 are conceptually simpler than Figures 1 (the current situation) or 2. Now the reason why it could be useful to have a Shallow Verify Cap -- to give someone the ability to verify the integrity of a directory without also giving them the ability to get the verify-caps of the children -- is for a kind of data-privacy. You might want to give lots of people the ability to verify the integrity of your directories without also giving them the ability to trace your directory structure -- the sizes and link structure of your directories and files. As we've recently been discussing, it might be nice for every storage server to have a verify cap to go with every share that it holds. We generally agree that "verify caps are not secret" -- everyone in the world can see everyone else's verify caps. You might not want everyone to be able to see the shape of your filesystem though! The only problem is: *they can do that anyway*. Anybody who can observe your Tahoe storage service connections (even though they are encrypted) or who operates a storage server can easily detect the exact structure of your filesystem -- which directories are linked to which other directories and files, as well as the precise size of all of the files. To defend against this sort of traffic analysis or pattern detection is somewhere between "hard" and "impossible". Our comrades over at the GNUnet project, the Freenet project, and others have been trying to develop such techniques for years (both Brian and I have contributed to such projects in the past, Brian more recently than I). Whether they're close to succeeding is not clear to me (perhaps some representative of such projects or someone whose expertise is more current than mine could speak up). But it is certain that TahoeLAFS will not offer such privacy in the next couple of releases. So on the one hand, I don't want to mush together the ability to shallow-verify and ability to deep-verify and thus preclude a user from exposing one without the other. That's why we call it "The Least-Authority Filesystem": because we believe that each authority that you can grant ought to be the least authority that you might want to grant. But on the other hand, the reason we call it "Least Authority" instead of the older term "Least Privilege" is because we recognize that the de facto ability to accomplish a goal ("authority") is what is important, not whether you have nominally been granted the official "privilege". That is: people already have the authority to deep-trace the structure of your filesystem, whether or not you officially grant them a capability to do so. There is another maxim in the capability security community (I believe it may be due to Chip Morningstar): "Never prohibit what you can't prevent.". In fact, offering separate Shallow Verify and Deep Verify Caps could endanger people instead of helping them. Imagine that you read the docs for TahoeLAFS and learn the difference between shallow- and deep- directory verify caps. Now you go out to use Tahoe or write code that uses Tahoe, and you have the idea that by choosing not to expose the deep ones you are safe from people learning about your filesystem structure. Your belief is misguided, and if you rely on your filesystem structure being private, you may suffer ill. My final argument is that we shouldn't pay a cost in conceptual complexity today for a potential future feature, if we can instead implement that feature when we later come to need it. If shallow verify caps will be useful in the future, when TahoeLAFS has grown an anonymity/privacy feature (or married a project such as GNUnet or Tor which already has one), then we can at that time implement shallow verify caps, guided by our enhanced understanding of the threat model and the defenses we will then employ. Regards, Zooko tickets mentioned in this message: http://allmydata.org/trac/tahoe/ticket/607 # DIR2:CHK tickets not mentioned in this message: http://allmydata.org/trac/tahoe/ticket/217 # DSA-based mutable files -- small URLs, fast file creation (Search in text for "shallow read-only caps".) From trac at allmydata.org Wed Jul 15 20:36:46 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:36:46 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f369936145f03f890eaab5e1aeefc819@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: undecided => 1.5.0 Comment: Hm. http://testgrid.allmydata.org:3567 currently says: {{{ Connected to 350 of 61 known storage servers }}} And its version numbers are: {{{ allmydata-tahoe: 1.4.1-r3897, foolscap: 0.4.1, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev }}} The [source:NEWS] file says that this ticket -- #653 -- has been fixed. But perhaps it hasn't? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 20:38:06 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:38:06 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.e4c0e678144806ce9c1200a1b819382d@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Terrell Russell pointed out that http://testgrid.allmydata.org:3567 currently says: {{{ Connected to 350 of 61 known storage servers }}} Perhaps this issue (ticket #748) is caused by #653. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 20:57:59 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 03:57:59 -0000 Subject: [tahoe-dev] [tahoe-lafs] #764: webapi/usage calls should be logged - standard reusable format (apache-style access.log) Message-ID: <039.de1dc7a65c8ca72ff48ec1250ed4724b@allmydata.org> #764: webapi/usage calls should be logged - standard reusable format (apache- style access.log) ----------------------------+----------------------------------------------- Reporter: terrell | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- This could/should be optional - but would allow storage server operators to more easily audit their usage. If in a standard, known format - the file would be parseable by a large number of other tools. zooko comments: If you turned on full foolscap logging to a file then you could probably grep out that part... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 21:08:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 04:08:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c9b6d2e22556b803a1f192b4d7bd87b1@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): Thanks for the review, and for seeing that -- I didn't even think about race conditions. The reason I didn't use the rename method you reference is {{{ This operation cannot move the child to a different directory. }}} which I took to mean that, if someone did {{{tahoe mv tahoe:dir/file1 tahoe:file1}}}, using that wouldn't work. Perhaps I'm mistaken, or misinterpreting something. I think I agree with your thoughts on the implementation. I don't suppose that any amount of client-side hacking is going to be enough to address the race condition above. Presumably the webapi would be a better place to address this. Then, what we'd want to do is * Write tests for the extension of the webapi command (in addition to the ones for {{{tahoe mv}}}, which are probably still a good idea even if we don't end up doing a lot of stuff there) * Alter the webapi command to pass the tests * Re-write {{{tahoe_mv.py}}} to use the new functionality Seem okay? If so, I'll start poking around/starting that. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Wed Jul 15 22:20:31 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 05:20:31 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.3ae3a075324bb5e8160559dd911e9b4a@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): From what I understand of the implementation of {{{POST /uri/$DIRCAP/[SUBDIRS../]?t=rename}}}, the webapi docs are right -- it's calling {{{move_child_to}}} (source:src/allmydata/dirnode.py at 20090713001320-92b7f- fe89f2baaab25c8eb3eb96c146177eb42108aeee#L515) with the {{{new_parent}}} argument set to the parent of the child being moved -- if I understand things correctly, anyway, that means that we can only use that command to move things around within a directory. I'm not sure how easy it'd be to tweak that to behave as we'd want. The {{{PUT}}} method seems like it'd be easy enough to modify. We could check for whatever replace option we wanted to correspond to the behavior we want to implement in {{{tahoe_mv.py}}} ({{{only_files}}} is fine by me), then examine {{{to_file}}} and {{{self.node}}} in the {{{DirectoryNodeHandler}}} to see if they're as they should be -- if so, we upload as we do now, and if not, we can raise an exception. I realize that I probably seem like a cheerleader for {{{PUT}}} at this point, but, given my understanding of the {{{POST}}} method, I'm not aware of any better options. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 05:36:35 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 12:36:35 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.5fb3583f0a60f8706bfffe0992d49722@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by zooko): You're right about {{{POST ?t=rename}}} not offering this functionality -- sorry I overlooked that. In the future, it might be good to extend {{{POST ?t=rename}}} to do that, but I agree with you that doing so would take longer and be more prone to error than other ways to solve this ticket right now. If you wanted to extend {{{POST ?t=rename}}}, you'd start [source:src/allmydata/web/directory.py at 20090715025814-92b7f- d4af644430e5daef6d6ad57cc550c8faceaeb2cf#L327 here in directory.py], and change it to look up the target node by name before calling {{{move_child_to}}}. Your proposal for how to extend {{{PUT}}} to have {{{replace=only_files}}} sounds right to me. You should push this extended semantics of {{{replace}}} all the way down into [source:src/allmydata/dirnode.py at 20090713001320-92b7f- fe89f2baaab25c8eb3eb96c146177eb42108aeee#L73 the Adder class]. That class currently takes a constructor argument {{{overwrite}}} which is a boolean. You could either change it to a multi-option argument (i.e. {{{overwrite}}} can be one of "yes", "no", "only_files"), or add a second argument named something like {{{preserve_directories}}} which is a boolean and add a precondition assertion that {{{not ((not overwrite) and preserve_directories)}}} (i.e. it is a mistake if someone passed {{{overwrite=False}}} and {{{preserve_directories=True}}} to the constructor of {{{Adder}}}.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 09:25:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 16:25:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.60e0b3962c190c9282ef7ba5f194aa3f@allmydata.org> #748: Exception: when trying to create directory on testgrid --------------------------+------------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: new Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Grr. My blog broke. [http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html My blog] is hosted on the Test Grid. That was really the last straw! I don't mind if everyone else who uses the Test Grid is inconvenienced by it being impossible to create directories, but I'm not going to leave my blog broken! So I upgraded the web gateway from: {{{ allmydata-tahoe: 1.4.1-r3897, foolscap: 0.4.1, pycryptopp: 0.5.2-1, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev }}} {{{ allmydata-tahoe: 1.4.1-r3995, foolscap: 0.4.1, pycryptopp: 0.5.15, zfec: 1.4.0-4, Twisted: 2.5.0, Nevow: 0.9.18, zope.interface: 3.3.1, python: 2.5.2, platform: Linux-Ubuntu_8.04-i686-32bit, sqlite: 3.4.2, simplejson: 1.7.3, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.16-12, setuptools: 0.6c12dev, pysqlite: 2.3.2 }}} Now instead of {{{Connected to 350 of 61 known storage servers}}} it says {{{Connected to 14 of 24 known storage servers}}} and my blog works nicely. Also, people can create directories again. This doesn't mean that this issue is resolved for v1.5 release though. We still need to investigate. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 10:44:21 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 17:44:21 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.ff3f999d3ec653612559b689968b95e3@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by terrell): the testgrid is now behaving properly - but there is still no test case. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 11:38:38 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 18:38:38 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.5dc909435caa97f8834cc7173601d2f3@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Per #748#comment:12 -- I upgraded and rebooted the testgrid web gateway server. However, the previous version that was running already had foolscap v0.4.1, so there is apparently ''some'' issue which results in more connections than servers which is still present in that version of foolscap. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 13:49:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 16 Jul 2009 20:49:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #49: UPnP In-Reply-To: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> References: <037.990325008bd9b3d720883cafbf92d5f8@allmydata.org> Message-ID: <046.c1c35ad930034ef5e5ff11d84c232419@allmydata.org> #49: UPnP ---------------------------+------------------------------------------------ Reporter: zooko | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-network Version: | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): Replying to [comment:15 zooko]: > Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as {{{miniupnp}}}, right? Makes sense to me. If it were necessary to write test cases to validate the correct functioning of every library used, you'd never have time to write any code. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From david at triendl.name Thu Jul 16 18:24:58 2009 From: david at triendl.name (David Triendl) Date: Fri, 17 Jul 2009 03:24:58 +0200 Subject: [tahoe-dev] calling all VolunteerGrid operators In-Reply-To: References: Message-ID: <20090717012458.GA8321@triendl.name> Good evening everyone, On Sat, Jul 11, 2009 at 03:50:43PM -0600, Zooko O'Whielacronx wrote: > If you are contributing a storage server to The VolunteerGrid, would > you please update the VolunteerGrid page on the wiki and describe your > server a little? For those left wondering: The url is http://allmydata.org/trac/tahoe/wiki/VolunteerGrid > Thanks. After this mail, which is going out to tahoe-dev, I'll > probably send future mails about the VolunteerGrid only to people > whose contact info is listed on that page. It would probably be easier if we had a small mailing list for volunteergrid operators/users, where we can discuss stuff like storage distributions, server outages, public nodes, experiments made, etc. Looking at the number of volunteergrid-related messages on tahoe-dev, I don't think the mailing list would be obstrusive or high-volume in any way. On a related, note, I would like to ask all volunteergrid users about how they use the volunteergrid? What kind of files did you store, what encoding did you use, what are your tahoe/volunteergrid success stories? Oh, and last but not least, I looked at Eugen Leitl's initial call for volunteers for a stable grid ( http://allmydata.org/pipermail/tahoe-dev/2009-February/001248.html ) - There are a lot of people who seemed interested but didn't add a node yet. Why not take a few minutes and join the volunteergrid? After all, everyone benefits from a broader node distribution. Cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090717/8fe30c92/attachment.pgp From zookog at gmail.com Thu Jul 16 20:20:31 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 16 Jul 2009 21:20:31 -0600 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design Message-ID: Folks: I really like the idea of "Semi-Private Keys". I invented this idea! This makes me very proud. I described the idea in lafs.pdf [1] and drew two pretty pictures (Figures 2 and 3) which show how much simpler a system of diminishing crypto-capabilities can be if it is built with semi-private keys. Apropos our recent discussion about capability taxonomy, the use of semi-private keys (or actually of semi-semi-private keys) would greatly simplify the implementation of hierarchies of diminished caps with more levels (such as if Shallow Verify caps were included in addition to Deep Write, Deep Read, and Deep Verify). In addition to being simpler, this construction would allow for an efficient off-line diminish computation. However, I really don't think we should rely on semi-private keys in our next cap design. There is no formal proof of their security, they've never been officially peer reviewed by cryptographers in a journal or conference, they are a new idea and almost all cryptographers on the planet have never heard of them. Therefore, there could be some subtle mathematical flaw in the idea which has gone unnoticed by Brian, Shawn, David-Sarah, myself, and the various other cryptographers who have looked at the idea. Also, even if the construction *is* actually safe, demanding and paranoid users such as governments and cypherpunks may refuse to rely on it out of an abundance of caution. There is another, more technical, reason not to use semi-private keys, which is that elliptic curve keys takes up twice as many bits as their security level. If our read caps contain an elliptic curve with K-bit security, then that curve has to have 2K-bits in it. Concretely, if we had a 96-bit security level (by the way I'm not yet sure that 96-bit security level is strong enough), then a read-key made up a 192-bit ecdsa semi-private key would look something like this: lafs://MR_1IvY9jxDREZmkfVyjM1eFmEqMpP8tGM9B A 128-bit security level (widely accepted) would require a 256-bit elliptic curve key, like this: lafs://MR_NHhjcZrScTqrxT73K7d4zgC4d5BzsjXYOf0JNgt2bFx And a 256-bit security level (super overkill to persuade even the most paranoid person that the crypto is not the weak point): lafs://MR_VVdUttNsfUw8eQYABYV0jGgY6hQGuoMji1LwtezvuZ4m74yhiMqjTC9v7fUOn50XzHXpggKRBpFxH3bL1jcAY3 In contrast, if we could use our current approach which doesn't rely on semi-private keys and somehow compress together the "read key" part of the cap and the "verify hash" part of the cap, then we could have 96-bit security with just one 96-bit value. However, as we recently discussed [2], short bitstrings such as hashes can be targetted by an attacker "at once". In contrast, this paper [3] seems to say that elliptic curve public keys can't be targetted en masse like that. That is, if you are given one 192-bit elliptic curve public key, it will probably cost about 2^96 to crack it. If you are given a trillion 192-bit elliptic curve public keys and you want to crack *any one* of them, it will probably *still* cost about 2^96! his conclusion is counter-intuitive to me, and I didn't really understand the math in that paper, so if anyone else wants to look at [3] and try to help us understand it that would be nice. (Note that this is the cost to crack an elliptic curve public key using the Pollard's rho method. The attack against secure hashes that we were talking about was a "brute force" attack in which you generate random ones yourself and then check if your random one collides with any of the many targets. One could do the same attack on elliptic curve public keys -- generating a random public key and then checking whether it is the same as someone else's public key. However, they are so long -- at least 192 bits -- that the 30 or 40 bits that you can shave off by generating a bilion or a trillion of them doesn't get you into range of having a chance of success.) Anyway, if the content of a cap is a secure hash output then the cap could be short, and so in order to avoid this "attack everyone at once" problem we have to make sure they aren't *too* short. Let's take our desired security level -- 96 bits -- and then add 64 bits to ensure that even if there are trillions upon trillions of targets out there (capabilities in use), the chance of success for the attacker is still negligible: 128 bit (possibly too short?): lafs://MR_4sViRyFdd9gt1WOWhBgEs6 160 bit (nice and safe): lafs://MR_S7msOFJ80xQ1WrDDiEP0M8ULxMT 256 bit (super overkill): lafs://MR_5yuGWCrBICTZEO06e0fhzYDuEqIbfKEpEBOlP239d5b Note: these caps have been formatted for human-usability by prepending them with a URL scheme (spelled here "lafs://", although other people who might be drivers of the standardization process such as Brian and Peter might prefer to spell it "tahoe://") and with "MR_" to mean "mutable file, read cap", and also the crypto bits have been base-62 encoded. Encoded in an 8-bit encoding such as ASCII or UTF-8, they now take up 32 bytes for a 128-bit cap, 37 bytes for a 160-bit cap, or 53 bytes for a 256-bit cap. When caps are not being presented to humans but instead are for computer use, such as when they are embedded inside directories, then they shouldn't be formatted like this, but should instead be basically just the pure 16, 20, 32, or 64 bytes of crypto material. So the trade-off between using semi-private keys embedded directly into the caps versus using secure hashes of traditional public keys in the caps is that the caps can be shorter for the same security level, as shown here, as long as that security level is at least 128 (or perhaps 160) bits. And, the crypto techniques we rely on can be standard. On the other hand, diminishing a capability cannot be performed as an off-line operation. E.g. if you have a read-write cap and you want to generate a read-cap, you have to find a storage server that has the "Extended Write Cap" and fetch it from there before you can generate the read-cap. (I suppose you could also cache the Extended Cap material or even include the necessary parts of it as an optional extension carried around with the cap itself!) Also, using traditional techniques instead of semi-private keys requires the format to be more complex. See lafs.pdf for details about that, but note that since I wrote that I've thought of another way to achieve "only one crypto value" using only standard crypto techniques. Note: I haven't yet published my idea for how to achieve "only one crypto value" by compressing together the read key and the verify hash. Regards, Zooko [1] http://allmydata.org/~zooko/lafs.pdf [2] http://allmydata.org/trac/tahoe/ticket/753#comment:1 [3] http://testgrid.allmydata.org:3567/file/URI:CHK:qxugzb4esi6b6clupozci7x6hu:jscs6fjjtm5jmqh3rchurubi5qmz7pxc2bf54ixt7vn64zmy5abq:3:10:305839/@@named=/multipledlog.pdf From shawn-tahoe at willden.org Thu Jul 16 21:09:04 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Thu, 16 Jul 2009 22:09:04 -0600 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design In-Reply-To: References: Message-ID: <200907162209.04414.shawn-tahoe@willden.org> On Thursday 16 July 2009 09:20:31 pm Zooko O'Whielacronx wrote: > However, I really don't think we should rely on semi-private keys in > our next cap design. There is no formal proof of their security, > they've never been officially peer reviewed by cryptographers in a > journal or conference, they are a new idea and almost all > cryptographers on the planet have never heard of them. I agree. I was only able to find a way to reduce the security by one or two bits, but I'm just a wannabe. The idea *seems* like it should be solid, but... Maybe you should publish it and then think about including it in Tahoe 3.0. > Note: I haven't yet published my idea for how to achieve "only one > crypto value" by compressing together the read key and the verify > hash. Get with it, man! It's like you have a job or something! ;-) Shawn. From trac at allmydata.org Thu Jul 16 21:51:56 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 04:51:56 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.82432138d45aceb21077a2ddfcbc56dc@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): FYI, this should have been fixed by [3872], which depended upon a feature added to foolscap-0.4.0 (i.e. foolscap itself does not fix the issue.. it required a tahoe-side change). I guess something may still be broken. We'll probably have to wait for it to become weird again and then look at the introducerclient's internal state. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 16 22:13:14 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 05:13:14 -0000 Subject: [tahoe-dev] [tahoe-lafs] #758: maximum recursion depth exceeded in Tahoe2PeerSelector In-Reply-To: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> References: <037.a61c0036d3c9fbd03c238a172a6a5b40@allmydata.org> Message-ID: <046.7bd337269e1a5c324affafe5703df0c7@allmydata.org> #758: maximum recursion depth exceeded in Tahoe2PeerSelector ------------------------------------+--------------------------------------- Reporter: zooko | Owner: Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-peerselection | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------------------+--------------------------------------- Changes (by warner): * status: new => closed * resolution: => fixed Comment: This should be fixed, by [3997]. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 06:07:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 13:07:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.7453660edfe311c06b9254b5bab8a934@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): > I guess something may still be broken. We'll probably have to wait for it to become weird again and then look at the introducerclient's internal state. :-( I guess I shouldn't have rebooted it then. Does that mean we should go ahead and boot this issue out of v1.5 Milestone? I have an idea -- let's add an assertion in the code that the number of connected storage servers <= number of known storage servers. Perhaps even a stronger, more detailed assertion if that's feasible. Here's what evidence I can gather about the problem that was exhibited. I upgraded and restarted testgrid.allmydata.org:3567 at 2009-07-16 16:17:18.835Z (on testgrid.allmydata.org's clock). There was nothing that looked too unusual in the {{{twistd.log}}} that day. There are two incidents reported, attached, from that day: {{{incident-2009-07-16-002829-gc4xv5y.flog.bz2}}} and {{{incident-2009-07-16-002846-pypgfay.flog.bz2}}}. Here is a foolscap log-viewer web service showing each of those logfiles: http://testgrid.allmydata.org:10000/ http://testgrid.allmydata.org:10001 . I have a hard time learning what I want to know from these logfiles. What I want to know (at least at first) is mostly about temporal coincidence. For starters, I'd like to be sure that these incidents occurred before I rebooted and upgraded the server, not after. However, the timestamps, such as "# 19:19:47.489 [23537270]: UNUSUAL excessive reactor delay ({'args': (25.734021902084351,), 'format': 'excessive reactor delay (%ss)', 'incarnation': ('\xf5\x16\x1dl\xb2\xf5\x85\xf9', None), 'num': 23537270, 'time': 1247710787.4896951, 'level': 23}s)" don't tell me what day it was nor what timezone the timestamp is in. Checking the status of http://foolscap.lothar.com/trac/ticket/90 suggests that the timestamps might be in PST, which is UTC-7. If that's the case then ... No, some of the (causally) earliest events in the log are from 18:01. Perhaps they were from 2009-07-15? Argh, I give up. Please tell me how to understand the timing of events in foolscap incident report files. I updated http://foolscap.lothar.com/trac/ticket/90 to plead for fully-qualified, unambiguous timestamps. The triggering incident is "download failure", but scarier to me is that there was a 26 second reactor delay. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 07:11:49 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 14:11:49 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.ae35c1066c76d0185cdddee61ac7cd25@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Oookay, I read the relevant source code and the miscounting of number of connected storage servers was fixed between [3897] (the version that exhibited the problem) and current HEAD. However, I'm not sure if that also means that whatever caused the failures on TestGrid was also fixed. Read on. Here is the path of the code that shows how "Connected to %s of %s known storage servers" was produced at version [3897]: [source:src/allmydata/web/welcome.xhtml at 3897#L55] [source:src/allmydata/web/root.py at 3897#L240] [source:src/allmydata/introducer/client.py at 3897#L277] Here is how it is produced today: [source:src/allmydata/web/welcome.xhtml at 3997#L55] [source:src/allmydata/web/root.py at 3997#L240] [source:src/allmydata/storage_client.py at 3997#L41] The old way could potentially have lots of tuples of {{{(serverid, servicename, rref)}}} for the same serverid and servicename, if new connections were established to the same serverid but the old connection was not lost, i.e. {{{notifyOnDisconnect()}}} wasn't getting called. The new way cannot possibly have an inconsistency between the number of known storage servers and the number of connected storage servers, since both are computed from the same dict -- the known servers are all the items of the dict and the connected storage servers are the ones that have an rref. Brian: what do you think about {{{notifyOnDisconnect()}}} not getting called even while new connections to the same foolscap peer are being established? That's the only explanation I can see for the observed miscounting on the web gateway that was running allmydata-tahoe [3897] and foolscap 0.4.1. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 07:36:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 14:36:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.9fce7b965325df40b8cc3aa4c1230d93@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): Hm, note that at least one of the storage servers was in a messed-up state in which it had incurred ~MemoryErrors, so it might have been failing to participate in some sort of foolscap "good-bye" negotiation, thus preventing the older connections to it from being cleaned up. This reminds of these mysterious warnings in the incident reports (I've seen a lot of this kind of message in foolscap logs): {{{ # 13:56:23.900 [867]: WEIRD Tub.connectorFinished: WEIRD, is not in [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 10:27:02 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 17 Jul 2009 17:27:02 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.b49929d6f61175dc524bf69876e70c14@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.5.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): (flogtool note: until we get [http://foolscap.lothar.com/trac/ticket/90 foolscap#90] resolved, you can use "flogtool dump --verbose" to get the raw seconds-since-epoch timestamps out of the event logs, which may give you more confidence in comparing them across servers. There is not yet something similar for "flogtool web-viewer", unfortunately. Also note that incident files '''always''' store seconds-since-epoch, so any conversion to a string time format is being done by the "flogtool" program that is processing the incident file) Yeah, one of the improvements in [3931] was to use a single data structure to keep track of storage servers, instead of two separate ones. Seemed more robust. I'd have to look more closely at the old code to figure out how {{{notifyOnDisconnect}}} could be confused. It gets fired when the TCP connection is lost, so there's no requirement that the remote side participate in any shutdown-time protocol, but it does get fired through an eventual-send, so there could conceivably have been a race condition or two. (incidentally, I removed those "Tub.connectorFinished" WEIRD messages in foolscap-0.4.2, the current release.. they were causing more problems than they were worth) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 17 20:19:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 03:19:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c149ccbb38b9dcbd99912517d902ded5@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -------------------------------+-------------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Keywords: reliability | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by kevan): Good idea with implementing the behavior in Adder. I've written some tests for Adder, and then implemented your first suggestion. If you set {{{overwrite}}} to "only_files", then it will refuse to overwrite directories. Instead of changing from {{{True}}} to "yes" and {{{False}}} to "no", I left those as they are now, since it seemed to stand less of a chance of breaking a bunch of stuff. Let me know if you want me to change this. Do we want to add tests for the new {{{overwrite}}} option to the tests for methods that use Adder (e.g.: {{{set_node}}}), or is what I have now good enough? I'll start working on the webapi stuff next, then alter {{{tahoe_mv.py}}} to use the new options. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 06:06:15 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 13:06:15 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.d64e36012f16992e4efd4eca5a89ff81@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by zooko): I would love to have this. A world map with colored pins showing where the VolunteerGrid servers are located would be very cool. Also if there were pins of a different color showing where TahoeLAFS contributors live that would make me happy too. (See [source:CREDITS the CREDITS file] for who I consider to be contributors.) However, I'm not going to prioritize setting this up, so if somebody else wants to click the "accept" button and take responsibility for this task then it is a lot more likely to get done. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 18 06:31:54 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 07:31:54 -0600 Subject: [tahoe-dev] using the volunteer grid (dogfood tasting report, continued) In-Reply-To: <1A90DE83-A23E-4682-B8A9-67CA414AE82C@zooko.com> References: <535BF38B-1D08-44A5-833D-AD7E3E113C07@zooko.com> <1A90DE83-A23E-4682-B8A9-67CA414AE82C@zooko.com> Message-ID: Folks: The VolunteerGrid seems to be working! For a while some of the music files I had stored on the VolunteerGrid were inaccessible. Today I tried again and they are back. This implies that some storage server which had some of the shares was unavailable last time, but I didn't keep a copy of the list of connected servers so I can't tell which it was. Here is the result of my deep check and repair (also attached in HTML format). Oh -- I guess the "Summary" of the directories should *not* say "Unhealthy: some versions are unrecoverable". That was the status *before* the repair. After the repair, the directories are healthy. That's a bug -- I've opened ticket #766. Regards, Zooko tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/766 # repair results Summary field says "Unhealthy" even though it is healthy after the repair, if it was unhealthy before Deep-Check-And-Repair Results for root SI=tsxqypdes5ysfme6cuqebinmdy Counters: * Objects Checked: 23 * Objects Healthy (before repair): 18 * Objects Unhealthy (before repair): 5 * Corrupt Shares (before repair): 0 * Repairs Attempted: 5 * Repairs Successful: 5 * Repairs Unsuccessful: 0 * Objects Healthy (after repair): 23 * Objects Unhealthy (after repair): 0 * Corrupt Shares (after repair): 0 Files/Directories That Had Problems: * : Unhealthy: some versions are unrecoverable * Metallica-Master of Puppets-13-of-16-fec: Unhealthy: some versions are unrecoverable * Metallica-Master of Puppets-13-of-26-fec: Unhealthy: some versions are unrecoverable * vidz: Unhealthy: some versions are unrecoverable * vidz/home_movies: Unhealthy: some versions are unrecoverable Relative Path Healthy Pre-Repair Recoverable Pre-Repair Healthy Post-Repair Storage Index Summary False True True tsxqypdes5ysfme6cuqebinmdy Unhealthy: some versions are unrecoverable 02 Master Of Puppets.flac True True True inal62aoisoa4huma5fmzq3req Healthy 03 The Thing That Should Not Be.flac True True True 4ub2wplyahagoner4npnzbshle Healthy 04 Welcome Home (Sanitarium).flac True True True jjrflgi4bpkodj5q7ygz6wz6oy Healthy 05 Disposable Heroes.flac True True True tdsby2jfikwsy67vfgoomax26e Healthy 06 Leper Messiah.flac True True True figj73pp7e7swislhr7bo3ol2u Healthy 07 Orion.flac True True True crcijgwqs7wmzd4juhcu4xpy5u Healthy 08 Damage Inc..flac True True True tydsprhi2bkdnqhzxsdzpgbjdq Healthy 2008-08-08_suarez-lnf-16x9-7142_download.mp4 True True True dwdkzbw5pl6xoos4dpqz5andti Healthy Metallica-Master of Puppets-13-of-16-fec False True True xzazzqtgytts6mp3iiv6tjrxdy Unhealthy: some versions are unrecoverable Metallica-Master of Puppets-13-of-16-fec/01 Battery.flac True True True wdbbrh2y4p77y7jpflkbucuzvy Healthy Metallica-Master of Puppets-13-of-26-fec False True True igmsz64hn6egnqu5c6e4bjmtsq Unhealthy: some versions are unrecoverable Metallica-Master of Puppets-13-of-26-fec/01 Battery.flac True True True uwgvqn372sycxncb6p3ct6rd7i Healthy Metallica-Master of Puppets-13-of-26-fec/02 Master Of Puppets.flac True True True uwqr6zia2qn2sgd2mxbwzui6ei Healthy Metallica-Master of Puppets-13-of-26-fec/03 The Thing That Should Not Be.flac True True True xollsyamtkwcg3nxhxwl4qffyq Healthy Metallica-Master of Puppets-13-of-26-fec/04 Welcome Home (Sanitarium).flac True True True 2izwujkldbas3z2oryunctseoi Healthy Metallica-Master of Puppets-13-of-26-fec/05 Disposable Heroes.flac True True True ohmxbqe2vefvltpk4x7phzgqn4 Healthy Metallica-Master of Puppets-13-of-26-fec/06 Leper Messiah.flac True True True fkt6xw5akqo4yzlo2joht7vovy Healthy Metallica-Master of Puppets-13-of-26-fec/07 Orion.flac True True True nztp537mjar6zgzkxqu5ph5yim Healthy Metallica-Master of Puppets-13-of-26-fec/08 Damage Inc..flac True True True ihe5aqjqhl3mjxhtfzxjky4f5u Healthy Metallica-Master of Puppets-4-of-6-fec True True True efqq2yffg36yvclyd5wff5wwrm Healthy vidz False True True 5mshviisrtykianham57n3gbye Unhealthy: some versions are unrecoverable vidz/home_movies False True True ggodl3dahhmp4n2d634ttzs4ie Unhealthy: some versions are unrecoverable runtime: 0.0529370307922 seconds -------------- next part -------------- An HTML attachment was scrubbed... URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20090718/62c3ce03/attachment.html From zookog at gmail.com Sat Jul 18 06:34:02 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 07:34:02 -0600 Subject: [tahoe-dev] private mailing list for VolunteerGrid Message-ID: Calling all Volunteers! I've set up a mailing list for the users and operators of the VolunteerGrid: http://allmydata.org/cgi-bin/mailman/listinfo/volunteergrid-l This is a *private* mailing list -- the list archives are available only to subscribers and my approval is required to become a subscriber. However, you are all welcome. Regards, Zooko From trac at allmydata.org Sat Jul 18 07:50:22 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 14:50:22 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.71b1ee568d8a89759069c02545916c9a@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): As I said, it's simple creating an embedded map with OSM. The only point is to get people's (and probably server's) geo-positions. I think I'll write an email to the list and ask for the data. If I get some reasonable reply, we can use that. (Putting that into the OSM database would make manegement and changing easier, though.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 08:11:32 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 15:11:32 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.4f45038beb9768140a85fa60c5259143@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by zooko): Per my comment:1 I created a user account on http://OpenStreetMap.org named "zooko" and clicked on my neighborhood of South Boulder, CO, USA. Is that sufficient to give you the information you need for my location? How can I also enter in the coordinates of things other than myself, such as my server, which is in a different location? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 09:01:47 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 16:01:47 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.891becc287eb5aa25462cb77dd64576d@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): Are you sure you ceated an account? http://www.openstreetmap.org/user/zooko tells me that "zooko" is not known... OSM is first of all a hugh database containint "interesting" stuff (roads, post boxes, houses, benches, ...). Thus you take one of the editors (eg. JOSM, Potlatch, ...) and enter known information about objects (first of all their location) into it. This is done in key/value pairs. For Tahoe, I'd probably do something like: tahoe_user=yes or tahoe_server=yes Actually, it's quite hard getting a good tag design really correct (wrt. extensibility, eg. get ir right for attaching supplied storage size afterwards etc.) For a start, it would probably nice to simply collect longitude/latitude of persons and servers and keep these in a simple CSV file or the like. OpenLayers can easily parse that. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 09:51:48 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 16:51:48 -0000 Subject: [tahoe-dev] [tahoe-lafs] #748: Exception: when trying to create directory on testgrid In-Reply-To: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> References: <043.02fa135b37a0b446275cf9b34e4d3880@allmydata.org> Message-ID: <052.1f2710dc055bc6376edff1457970a49b@allmydata.org> #748: Exception: when trying to create directory on testgrid ------------------------------+--------------------------------------------- Reporter: toby.murray | Owner: nobody Type: defect | Status: closed Priority: major | Milestone: 1.5.0 Component: code-mutable | Version: 1.4.1 Resolution: fixed | Keywords: Launchpad_bug: | ------------------------------+--------------------------------------------- Changes (by zooko): * status: new => closed * resolution: => fixed Comment: Okay, we investigated (details posted on #653), and I'm pretty that this was caused by #653, and I'm pretty sure that #653 has been fixed. Closing this as "fixed". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 10:40:58 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 17:40:58 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.c7fac9aac7160b4d26eaab73d4ff202b@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): I've now implemented a small hack to play with pin features :) http://webcontent.osm.lab.rfc822.org/tahoe/ I've used GeoRSS as a base, that is, we can have a Heading, Link and a describing text for each pin, easily. I've split it up into three groups (devs, users and servers) and would like to get your comments on it. (And I think I need help with the map extend. How can I place proper CSS there to let it extend to the rest of the screen?) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 10:44:01 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 17:44:01 -0000 Subject: [tahoe-dev] [tahoe-lafs] #642: world map with tahoe pins in it In-Reply-To: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> References: <037.fc529e9982c27b65e5456dce33bba4e8@allmydata.org> Message-ID: <046.4a8655f6db692aafee6039c9065d763f@allmydata.org> #642: world map with tahoe pins in it ---------------------+------------------------------------------------------ Reporter: zooko | Owner: somebody Type: task | Status: new Priority: minor | Milestone: undecided Component: website | Version: 1.3.0 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ Comment(by jbglaw): If I get positive responses here, I'd announce it to the mailing list. Though I'd strongly underline that publishing your *exact* location (and that of the tahoe servers) might or might not be a good idea. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Sat Jul 18 13:23:09 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sat, 18 Jul 2009 22:23:09 +0200 Subject: [tahoe-dev] private mailing list for VolunteerGrid References: Message-ID: <874ot9n3ki.fsf@gnu.org> Hello, "Zooko O'Whielacronx" writes: > This is a *private* mailing list -- the list archives are available > only to subscribers and my approval is required to become a > subscriber. Hmm, the list is private but the grid is currently public. You might want to make the grid private, then? Thanks, Ludo'. From trac at allmydata.org Sat Jul 18 13:16:47 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 20:16:47 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f9d2090af9d4bfbc743ded4a9ea7e66c@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => 1.6.0 Comment: I spoke with Brian on IRC this morning before his gallivanting schedule, and he said that it was unlikely that this problem will recur with the current versions of foolscap and of TahoeLAFS. I agree and am going to bump this ticket out of the v1.5 Milestone. Note that we can't actually close this ticket yet because there isn't a test for the original issue: duplicate entries in {{{IntroducerClient._connections}}} because ~RemoteReferences don't compare equal when they denote the same underlying TCP connection to the same peer, much less the newly discussed issue: duplicate ~RemoteReferences because (somehow) the peer is setting up new ~RemoteReferences while {{{notifyOnDisconnect()}}} is not getting called for the old ones. However, I'm fairly confident that this is (a) not a regression versus TahoeLAFS v1.4.1, and (b) not a critical bug. (In fact, it might actually be already fixed in current trunk.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Sat Jul 18 14:27:09 2009 From: warner at lothar.com (Brian Warner) Date: Sat, 18 Jul 2009 16:27:09 -0500 Subject: [tahoe-dev] simple authority taxonomy versus a kind of privacy In-Reply-To: <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090715104906.251b900b@luther> <6DE6F04A-0A70-459D-8A12-B45A58337276@zooko.com> Message-ID: <4A623E2D.4020805@lothar.com> Zooko Wilcox-O'Hearn wrote: > I was wondering about the word "traversal cap", thinking "Isn't that > what one might call a 'deep verify cap'?". Yeah, "deep-verify" is a reasonable term. Depending upon how we implement lease-renewal-caps and repair-caps, the "traversal cap" might be combineable with some other secret to generate those other caps for all reachable files and directories, so we might also create a "deep-repair" or "deep-renew" cap. But unless we go crazy and expand the dirnode structure to half a dozen columns, these other deep- caps will be based upon the deep-verify cap. Since directories are really just specialized ways of interpreting the contents of a file (mutable for now, we plan to add immutable in the future), every directory will also have the same caps you'd get on the underlying file object. There will always be a "shallow verify cap" for each directory (i.e. the regular verifycap for the underlying file), regardless of whether there is a good use for it or not. So I don't think it makes much sense to try to get rid of them. If you're concerned about people misusing them, just don't expose them in the interface, or don't give them a provocative name. > Now the reason why it could be useful to have a Shallow Verify Cap -- > to give someone the ability to verify the integrity of a directory > without also giving them the ability to get the verify-caps of the > children -- is for a kind of data-privacy. Lease management and file-repair are the two things I can imagine wanting to use lists of shallow verify caps for. A "manifest" is a list of shallow caps (using as weak a cap as you can get.. perhaps just a storage-index), with which you might: * compare against a previous manifest, to find out which files/directories have been added or removed, to cancel leases on the removals or add leases to the additions. * transform into a list of lease-renewal caps, to be given to a service that will keep your files alive while you and your computer are on vacation * transform into a list of repaircaps, for a similar repairer service The reason for having a deep-verify-cap is to provide (for those folks who are willing to give up this information) an easier way to delegate these repair and lease-maintenance tasks. Instead of building a whole manifest and giving thousands of repaircaps to your hired repair service (and rebuilding the list each time you change anything), you give them a single deep-repair-cap instead (which remains valid for a long time). > The only problem is: *they can do that anyway*. Anybody who can > observe your Tahoe storage service connections (even though they are > encrypted) or who operates a storage server can easily detect the > exact structure of your filesystem -- which directories are linked to > which other directories and files, as well as the precise size of all > of the files. I disagree with the first claim. A passive observer will scarcely find it "easy" to determine the shape of your filesystem. The foolscap messages and storage-index arguments are hidden by the encrypted links, so the only information available to the adversary is the number of bytes sent in each direction. If the client is only doing one thing at a time, they can probably match requests to responses, and distinguish between dirnode reads and file reads. Then their only way to distinguish directories is by the size of their packed representation, and they can only learn the size of files that you actually download completely (and they can't distinguish between multiple files of the same size). You'd have to do a recursive download of all your files, one at a time, to let them build up the shape of your directory structure this way. The adversary who is able to mount this attack must be able to observe all of your traffic to nearly all storage servers. A more active adversary (who runs a storage server) must still run several servers (something like numservers/k) to have a good chance of seeing most of your requests, and still only gets to learn anything about the files that you actually download. In addition, the information you get from a deep-verify-cap is significantly greater than what you can get from traffic analysis. With the verifycap, you get the exact file identities (even without downloading anything), which you can compare against other people's manifests, or against information you retrieve from unrelated storage servers. If convergence is enabled, knowing the storage-index lets you mount partial-information-guessing attacks against those files. Also, there are many Tahoe use-cases which involve trust relationships between clients and server operators: the servers might be more trusted than the outside world, so it may be ok that the client is more vulnerable to server behavior than to third party observers. My general feeling is this: the existence of traffic-analysis attacks (which allow adversaries in certain positions to learn "X") is not a strong argument to simply publish "X" to the entire world. The traffic-analysis attacks we've identified are not trivial to mount: why make the bad-guy's job easier? In addition, other people may find reasonable ways to address these problems in the future (someone might use PIR to retrieve shares): we'd be thwarting their noble efforts by publishing the X that they're trying to hide. That feeling may or may not be appropriate here, but in general I'm suspicious of arguments to explicitly abandon privacy properties in response to the threat of an attack. Anyways, I don't understand what sort of code change you're advocating: how could you "get rid of" a dirnode's shallow-verify cap? Let's just keep calling it "the verifycap of the underlying mutable file" instead of "shallow-verify" and not encourage the user to use it or believe they can get any special privacy-preserving properties from it. cheers, -Brian From trac at allmydata.org Sat Jul 18 14:35:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 21:35:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.835da7e0a16abc799bea6113d3b46f3c@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): hm, I do hope we'll be able to close this ticket eventually, without reverting back to the old code, analyzing the problem, fixing it, then discarding the fix to go back to the new code :-). Let's consider closing this as INVALID after a couple of months of correct operation of the testgrid. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 18 16:57:10 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 18 Jul 2009 23:57:10 -0000 Subject: [tahoe-dev] [tahoe-lafs] #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ In-Reply-To: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> References: <038.8c1468b9bd0661695cf6ebfd2262730f@allmydata.org> Message-ID: <047.f253ac18fa3ddbb529fcd58c9767abeb@allmydata.org> #653: introducer client: connection count is wrong, !VersionedRemoteReference needs EQ --------------------------+------------------------------------------------- Reporter: warner | Owner: warner Type: defect | Status: assigned Priority: major | Milestone: 1.6.0 Component: code-network | Version: 1.3.0 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by zooko): The only part that I'm not sure about is the way that multiple {{{RemoteReferences}}} piled up. I.e., I'm confident that there is not a bug in TahoeLAFS with regard to this issue, but I'm not confident that there isn't a bug in foolscap about this issue. Note that we didn't change the version of foolscap on the testgrid webapi (foolscap 0.4.1). Maybe if I write down what I know about it in this ticket that will help me understand it better. Could you summarize what was changed which made me think (in comment:4) that foolscap v0.4.0 fixed the ''original'' issue of having multiple {{{RemoteReferences}}} over the same TCP connection to the same peer? Whatever it was, it didn't fix the problem that was demonstrated using TahoeLAFS [3897] and foolscap 0.4.1 in which there were many more tuples of {{{(serverid, servicetype, rref)}}} than there were servers. This suggests that foolscap 0.4.1 must have been creating new {{{RemoteReferences}}} and not calling back the {{{notifyOnDisconnect()}}} method to clean up old {{{RemoteReferences}}} to the same peers. Now, TahoeLAFS in version [3997] no longer keeps tuples of {{{(peerid, servicetype, rref)}}}, instead it keeps a dict mapping from {{{peerid}}} to {{{NativeStorageClientDescriptor}}}, where a {{{NativeStorageClientDescriptor}}} has at most one {{{RemoteReference}}}. When foolscap calls {{{NativeStorageClientDescriptor._got_connection()}}}, then the {{{NativeStorageClientDescriptor}}} eventually sets {{{self.rref}}} to reference the new {{{RemoteReference}}} (see [source:src/allmydata/storage_client.py at 20090717050709-66853-80eda37caf0df376be79f45cbed728999b68a843#L218 storage_client.py line 218].) If its {{{self.rref}}} already had a reference to a {{{RemoteReference}}} then it would be overwritten by the new one, so the problem of lots of redundant {{{RemoteReference}}}s piling up certainly can't happen in TahoeLAFS anymore. But, if foolscap is generating redundant {{{RemoteReference}}}s passing them to TahoeLAFS, could this cause other problems? For example, I see that {{{NativeStorageClientDescriptor}}} calls {{{rref.notifyOnDisconnect(self._lost)}}}. If foolscap is generating redundant {{{rref}}}s, then maybe the {{{self._lost}}} will eventually be called by one of the old ones that is no longer referenced by the {{{NativeStorageClientDescriptor}}}, and if so then the current {{{rref}}} will get spuriously removed? Maybe we should add some sanity checking to {{{NativeStorageClientDescriptor}}} such as asserting that when its {{{_got_connection()}}} is called that its {{{self.rref}}} is required to be currently {{{None}}}, or asserting that when its {{{self._lost()}}} method is called that the current value of {{{self.rref}}} is the same as it was when the {{{notifyOnDisconnect()}}} was called? In the interests of a stable v1.5, maybe we could make it so that if these checks fail it does something "safe", such as if {{{self._lost()}}} is called when {{{self.rref}}} is not equal to the same thing that it was when the {{{notifyOnDisconnect()}}} was called, that it leaves the {{{self.rref}}} alone (as well as logging a high-scariness incident. What do you think? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Sat Jul 18 20:27:26 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Sat, 18 Jul 2009 21:27:26 -0600 Subject: [tahoe-dev] new logo? Message-ID: Hey folks: What do you think? http://allmydata.org/~zooko/TahoeLAFS-logo.png http://allmydata.org/~zooko/TahoeLAFS-logo.svg Like in http://allmydata.org/~zooko/network-and-reliance-topology.png , the red node represents the things that have access to the crypto keys and the black nodes represent things that don't. Regards, Zooko From ludo at gnu.org Sun Jul 19 05:57:07 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Sun, 19 Jul 2009 14:57:07 +0200 Subject: [tahoe-dev] new logo? References: Message-ID: <87eiscsue4.fsf@gnu.org> Hi Zooko, "Zooko O'Whielacronx" writes: > http://allmydata.org/~zooko/TahoeLAFS-logo.png > http://allmydata.org/~zooko/TahoeLAFS-logo.svg Do you really want to make "LAFS" part of the name? IMO it clutters the logo and the name. (Other than that, while I agree it's important to emphasize that Tahoe follows the POLA, I'm a bit skeptical about the phrase "LAFS": it's a hint to those who know what POLA is, but it doesn't tell much to those who don't, I think.) Thanks, Ludo'. From kpreid at mac.com Sun Jul 19 06:17:46 2009 From: kpreid at mac.com (Kevin Reid) Date: Sun, 19 Jul 2009 09:17:46 -0400 Subject: [tahoe-dev] new logo? In-Reply-To: References: Message-ID: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> On Jul 18, 2009, at 23:27, Zooko O'Whielacronx wrote: > Hey folks: > > What do you think? > > http://allmydata.org/~zooko/TahoeLAFS-logo.png > http://allmydata.org/~zooko/TahoeLAFS-logo.svg > > Like in http://allmydata.org/~zooko/network-and-reliance-topology.png > , the red node represents the things that have access to the crypto > keys and the black nodes represent things that don't. I think it is lacking in visual coherence: the text has nothing to do with the image, and the image has no particular geometry. I also agree that "LAFS" perhaps shouldn't be part of the prominent name. How about this (made in a hurry, to be refined later): http://switchb.org/kpreid/2009/tahoe/logo-1-o.png http://switchb.org/kpreid/2009/tahoe/logo-1-o.graffle (I'll provide a SVG version if desired; as I don't have a SVG editor this was faster.) -- Kevin Reid From michael at michael.ie Sun Jul 19 08:06:21 2009 From: michael at michael.ie (Michael Walsh) Date: Sun, 19 Jul 2009 16:06:21 +0100 Subject: [tahoe-dev] new logo? In-Reply-To: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: On Sun, Jul 19, 2009 at 2:17 PM, Kevin Reid wrote: > How about this (made in a hurry, to be refined later): > > http://switchb.org/kpreid/2009/tahoe/logo-1-o.png > http://switchb.org/kpreid/2009/tahoe/logo-1-o.graffle Ah, now that actually looks like a logo and is pretty much there IMO. From zooko at zooko.com Sun Jul 19 13:04:10 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 19 Jul 2009 14:04:10 -0600 Subject: [tahoe-dev] discussion of TahoeLAFS crypto on the cryptography mailing list Message-ID: On the cryptography mailing list there was some discussion about the fact at 112-bit elliptic curve key was recently brute-forced, so I described some of how TahoeLAFS is about to switch to (probably maybe?) 192-bit elliptic curve from RSA: http://www.mail-archive.com/cryptography at metzdowd.com/msg10652.html http://www.mail-archive.com/cryptography at metzdowd.com/msg10653.html Regards, Zooko From zooko at zooko.com Sun Jul 19 13:33:57 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 19 Jul 2009 14:33:57 -0600 Subject: [tahoe-dev] new logo? In-Reply-To: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: I like your design! I'd like to mess with it and post a variant -- could I please have an .svg version? As for naming, we have a major problem that "Tahoe" is not searchable/ unique enough to serve as the official name. Try your favorite search engine and search for "Tahoe". I just browsed through the first eighteen pages of the google search results for "Tahoe" and I finally found something that wasn't the lake in California/Nevada: it was this open source software project: http://tahoe.ca.sandia.gov . Anyway, while I'm happy for people to contribute their creative ideas about the logo, I'm not quite as happy to hear new creative ideas about the name. Anyway, I like your logo. --Zooko From sam at samason.me.uk Sun Jul 19 13:49:33 2009 From: sam at samason.me.uk (Sam Mason) Date: Sun, 19 Jul 2009 21:49:33 +0100 Subject: [tahoe-dev] new logo? In-Reply-To: References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: <20090719204933.GJ5407@samason.me.uk> On Sun, Jul 19, 2009 at 02:33:57PM -0600, Zooko Wilcox-O'Hearn wrote: > As for naming, we have a major problem that "Tahoe" is not searchable/ > unique enough to serve as the official name. Try your favorite > search engine and search for "Tahoe". I just browsed through the > first eighteen pages of the google search results for "Tahoe" and I > finally found something that wasn't the lake in California/Nevada: it > was this open source software project: http://tahoe.ca.sandia.gov . Really--for me, it comes up ninth from google.com and sixth from google.co.uk! I'm somewhat surprised that it's coming up as high as it does as the root page doesn't have any text for google to index. -- Sam http://samason.me.uk/ From kpreid at mac.com Sun Jul 19 14:32:47 2009 From: kpreid at mac.com (Kevin Reid) Date: Sun, 19 Jul 2009 17:32:47 -0400 Subject: [tahoe-dev] new logo? In-Reply-To: References: <1E725655-79DB-4540-8592-A9DBC7107A6B@mac.com> Message-ID: On Jul 19, 2009, at 16:33, Zooko Wilcox-O'Hearn wrote: > I like your design! I'd like to mess with it and post a variant -- > could I please have an .svg version? OK. Please note that this is handwritten SVG and using an editor may cause it to lose structure. Also, the rendering depends on the exact metrics of the Futura font (and, for all I know, the details of the SVG renderer); it would be more robust to embed the characters as paths into the SVG but I don't have tools handy to do that. I expect Inkscape does, though; if you do that and send me the results I'll incorporate them. -------------- next part -------------- A non-text attachment was scrubbed... Name: Tahoe logo proposal.svg Type: image/svg+xml Size: 1274 bytes Desc: not available Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090719/08799316/attachment.svg -------------- next part -------------- http://switchb.org/kpreid/2009/tahoe/logo-1-o.svg > As for naming, we have a major problem that "Tahoe" is not searchable/ > unique enough to serve as the official name. Try your favorite > search engine and search for "Tahoe". I just browsed through the > first eighteen pages of the google search results for "Tahoe" and I > finally found something that wasn't the lake in California/Nevada: it > was this open source software project: http://tahoe.ca.sandia.gov . Good point. I suggest, however, that you make sure to be consistent about "Tahoe-LAFS" vs. "TahoeLAFS". Both forms will work fine as a Google search term, but the former has more results, and also has the advantage that searchers for "Tahoe LAFS" will find it. -- Kevin Reid From warner at lothar.com Sun Jul 19 21:31:51 2009 From: warner at lothar.com (Brian Warner) Date: Mon, 20 Jul 2009 00:31:51 -0400 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design In-Reply-To: References: Message-ID: <4A63F337.4050204@lothar.com> Zooko O'Whielacronx wrote: > the use of semi-private keys (or actually of semi-semi-private keys) > would greatly simplify the implementation of hierarchies of diminished > caps with more levels For reference, the semi-private ECDSA-based mutable file scheme we've been considering (hereafter abbreviated SP-DSA) would have a DSA hierarchy with two intermediate keys, and symmetric-encryption secrets (used for data confidentiality) assigned as follows: 1 = DSA signing key 2 = first intermediate key (derived with math from 1) 3 = second intermediate key (derived with math from 2) 4 = verifying key (derived with math from 3) A = writekey (hash of 1), encrypts child writecaps in a dirnode B = readkey (hash of A), encrypts filenames and child readcaps in dirnode C = traversalkey/deep-verify-key (hash of B), encrypts child traversalcaps Then the actual filecaps would be: writecap = 1 (allows full access) readcap = 2 (no access to child writecaps or forming new signatures) traversalcap = 3 (verify and access to traversalcaps only, i.e. deep-verify) verifycap = 4 or hash of 4 storage-index = verifycap = 4 or hash of 4 The server-side shares would hold a copy of "4" (the verifying key). This scheme is great: it gives us all the properties I've ever wanted for mutable files, including: * one crypto value for all caps (which I think is K bits for the writekey, and 2K bits for the rest) * servers can verify every bit of the share, all the way up to the storage-index (i.e. servers can detect valid shares stored under the wrong storage-index) * full offline attenuation (aka "diminish computation"): given any cap, you can compute any weaker cap without talking to a server * related to that, given a writecap you can create new shares without talking to a server, which would let you replace a file that was completely wiped out (our current RSA-based scheme doesn't allow this: if you lose all shares, you'll never get the signing key back, so you'll never be able to create new shares, and basically you have to give up on that slot) > However, I really don't think we should rely on semi-private keys in > our next cap design. There is no formal proof of their security, I fully agree with this argument. (although I am reluctant about it.. I think it's a neat idea, and I really like the simple design described above, and all the properties it has. If I weren't so paranoid about crypto tools, I'd say go for it. Curses! If only this trick were invented twenty years ago and thoroughly reviewed by now. Next time I get access to a time machine, I'll go back and explain it to zooko's teenage self so he can invent it earlier :-). > There is another, more technical, reason not to use semi-private keys, Ok, here's where I get confused. > In contrast, if we could use our current approach which doesn't rely > on semi-private keys and somehow compress together the "read key" part > of the cap and the "verify hash" part of the cap, then we could have > 96-bit security with just one 96-bit value. However, as we recently > discussed [2], short bitstrings such as hashes can be targetted by an > attacker "at once". In contrast, this paper [3] seems to say that > elliptic curve public keys can't be targetted en masse like that. There are too many negatives in that paragraph for me to follow. (I must admit that my time is very fragmented right now.. I'm only getting to read the mailing list in 10-minute bursts, and I don't have a nice quiet place to think about things, and won't until I get back from this trip in a few weeks. It's entirely possible that I would understand what you're saying if I were at home instead of travelling). I think you are comparing SP-DSA (above) against an as-yet-undescribed scheme which depends upon some not-yet-invented compression trick. I don't think I can respond completely meaningfully until you invent and/or describe this other scheme :-). I think much of your text is stating a lower bound (of maybe 128 or 160 bits) on the crypto strings to avoid birthday attacks, which I completely agree with. You've mentioned your compression trick to me once or twice before, and although I've forgotten the details, I think I remember concluding that it didn't provide all the properties we needed. Let me list those properties here: * offline derivation of storage-index from any cap: necessary to get any share data, including full pubkeys or other values which were too large to store in the cap * full server-side verification of shares, up to the storage-index: the server should be able to detect any bitflip in the share, and should also be able to detect a valid share for SI=1 being mistakenly placed in a file meant to hold SI=2. (this basically requires the storage-index to equal the verifycap or H(verifycap)) * I think it's important, but not absolutely critical, to be able to perform offline attenuation from writecap to readcap, without first having to fetch e.g. the full pubkey from at least one server. While we store both child writecap+readcap in our dirnodes (so many use-cases will simply be able to snarf the pre-attenuated readcap from the dirnode), we must also populate both writecap and readcap in those dirnodes when we fill them. dirnode.set_children() (which is given a bunch of child writecaps) will be very expensive if we must first fetch each child's pubkey. For the same reason, I think we need offline attenuation from writecap/readcap to traversalcap. * I also think it's important (but less so) to have offline attenuation from any cap to a verifycap. This would let us build a repair manifest by reading all the dirnodes (without this property, we would also have to touch all the filenodes, which would be considerably more work). Fortunately, any scheme that uses verifycap=SI will automatically provide this property too. * it would be nice to be able to generate the strongest secret (probably the signing key) from the writecap without talking to a server. (note that our current RSA scheme doesn't provide this property). This would let you fix a lost-all-shares situation, as described above. Again, I could live without this. So, excluding your as-yet-undescribed compression trick, the best non-semiprivate DSA mutable file scheme that I've been able to come up with would look like: 1 = signing key 2 = verifying key (derived with math from 1) 3 = storage-index (either 2 or hash of 2, depending upon length) A = writekey (hash of 1), encrypts child writecaps in a dirnode B = readkey (hash of A), encrypts filenames and child readcaps in dirnode C = traversalkey/deep-verify-key (hash of B), encrypts child traversalcaps writecap = 1 readcap = B+3 traversalcap = C+3 verifycap = 3 The confidentiality comes from the A/B/C chain, and the integrity comes from the 1/2/3 chain. Excepting the master secret (1), there's no way to get a single bit to perform both jobs (and still be able to derive a storage-index from any cap). I care about short writecaps and (to a lesser extent) short readcaps.. traversalcaps and verifycaps can be long. With X-bit EC-DSA signing keys and Y-bit hashes (perhaps X=128 and Y=128), this would give us the following cap lengths: writecap: X (128) readcap: 2*Y (256) traversalcap: 2*Y (256) verifycap: Y (128) I think I'm ok with those lengths, although if we raise X or Y to get a more comfortable margin then they might get a bit unwieldy. (Note that the length of the verifying key is irrelevant here, if we store the full verifying key in the share and merely put its hash in the readcap/traversalcap/verifycap. This is actually very close to what we do with RSA keys, except that the RSA signing key is also too long to put in the cap, so we use writecap=A instead of writecap=1 and have a scheme to encrypt the signing key and store it in the share.) Since ECDSA signing keys are short enough to use directly as writecaps, it would allow regeneration of slots in which all shares were lost. Likewise, since writecaps hold the full signing key (from which the verifying key can be computed), we get full offline attenuation. I'll wait to say more until Zooko describes his compression trick. In the brief picture he showed me earlier tonight (which was actually for immutable files), it looked like the readcap was derived partially from the hash of the verifycap, and if SI=verifycap or SI=H(verifycap), that would mean it's impossible to derive the SI from the readcap, which would make it very difficult to actually locate the right shares. On the other hand, if the storage-index is something else, then the server can't completely verify the share. Hm, you know, what would the cap lengths be for the SP-DSA scheme anyways? If the intermediate keys need to be 2K bits long, and if a K-bit hash gives us K bits of integrity (this may be false), then the SP-DSA scheme and the non-SP DSA scheme I described may give us similar cap lengths, and we can achieve our goals without relying upon SP-DSA. Or would Y need to equal 2K (i.e. 256 bits), making the non-SP-DSA scheme have 384-bit readcaps versus SP-DSA's 256-bit readcaps? cheers, -Brian From trac at allmydata.org Mon Jul 20 03:33:51 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 20 Jul 2009 10:33:51 -0000 Subject: [tahoe-dev] [tahoe-lafs] #768: OpenBSD support in util/iputil.py Message-ID: <039.f8493439cdd03b7ab4e9cb3f85200267@allmydata.org> #768: OpenBSD support in util/iputil.py ---------------------+------------------------------------------------------ Reporter: tdelaet | Owner: somebody Type: defect | Status: new Priority: major | Milestone: undecided Component: code | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ To support new OpenBSD releases, a few extra entries are needed in the plaform map in util/iputil.py. Tahoe does not work with me unless I add 'openbsd4': 'bsd' to the platform map. I assume that 'openbsd3': 'bsd' also needs to be added but I did not test that. After this addition tahoe works for me on OpenBSD 4.5 on sparc64. (tested with revision 3996 from snapshots) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Mon Jul 20 07:17:28 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 20 Jul 2009 08:17:28 -0600 Subject: [tahoe-dev] v1.5.0 is feature-complete! Message-ID: Thanks to Kevan Carstensen for implementing and Brian Warner for reviewing, the last issue, ticket #705, has been resolved. There are no more open tickets in the v1.5.0 Milestone, and I plan to defer any patches which change the code until after the 1.5 release. Any further patches will just be to docs (e.g. the relnotes.txt file). So, test it out! Upgrade your grids to the current trunk and report any anomalies! I'll write up Release Notes and do the whole how_to_make_a_tahoe_release process [1] as my day-job schedule permits, so probably some morning this week before my children wake up and need breakfast and help with their chores. :-) Regards, Zooko [1] http://allmydata.org/trac/tahoe/browser/docs/ how_to_make_a_tahoe_release.txt tickets mentioned in this e-mail: http://allmydata.org/trac/tahoe/ticket/705 # "tahoe mv" unlinks the target even when it is a directory From trac at allmydata.org Mon Jul 20 11:22:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 20 Jul 2009 18:22:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #705: "tahoe mv" unlinks the target even when it is a directory In-Reply-To: <037.a8a194300e8b675536e90909f435f643@allmydata.org> References: <037.a8a194300e8b675536e90909f435f643@allmydata.org> Message-ID: <046.c0d401d103c7fcaded49070118c8873d@allmydata.org> #705: "tahoe mv" unlinks the target even when it is a directory -----------------------------------+---------------------------------------- Reporter: zooko | Owner: kevan Type: defect | Status: closed Priority: critical | Milestone: 1.5.0 Component: code-frontend-cli | Version: 1.4.1 Resolution: fixed | Keywords: reliability Launchpad_bug: | -----------------------------------+---------------------------------------- Comment(by kevan): Between those and the {{{POST}}} method that zooko mentioned earlier, we could probably stand to open a new ticket (obviously for after 1.5.0) with the things that we still want to do here. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 20 14:45:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 20 Jul 2009 21:45:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #629: 'tahoe backup' doesn't tolerate 8-bit filenames In-Reply-To: <038.485461e6d4d6a1baef902f268cb98750@allmydata.org> References: <038.485461e6d4d6a1baef902f268cb98750@allmydata.org> Message-ID: <047.f22816558f8a7d55bf23c9c783154a8c@allmydata.org> #629: 'tahoe backup' doesn't tolerate 8-bit filenames -------------------------------+-------------------------------------------- Reporter: warner | Owner: francois Type: defect | Status: assigned Priority: major | Milestone: undecided Component: code-frontend-cli | Version: 1.3.0 Keywords: backup unicode | Launchpad_bug: -------------------------------+-------------------------------------------- Comment(by midnightmagic): Uh.. one might be me. :-) I have ill-encoded filenames as a leftover from old, old data stores: I actually have direct backups from my Amiga, for example, that have survived all the machine disasters I've had (so far) and persist in my backup stores. I have files from my old Commodore-64 too, for that matter. Sometimes it's a matter of trying to keep as pristine a backup of those files as possible so that I can use them in things like emulators, or even actual antique hardware, that keeps me from doing any sorts of conversions. Also, the OS filesystem seems to ignore encodings for the most part in its command-line activities: and so these artifacts persist. (And I'd love to store them in a tahoe backup grid without transforming them with.. for example, tar or something.) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Mon Jul 20 15:53:39 2009 From: warner at lothar.com (Brian Warner) Date: Mon, 20 Jul 2009 18:53:39 -0400 Subject: [tahoe-dev] two reasons not to use semi-private keys in our new cap design In-Reply-To: <4A63F337.4050204@lothar.com> References: <4A63F337.4050204@lothar.com> Message-ID: <4A64F573.9010907@lothar.com> Brian Warner wrote: > The confidentiality comes from the A/B/C chain, and the integrity > comes from the 1/2/3 chain. Excepting the master secret (1), there's > no way to get a single bit to perform both jobs (and still be able to > derive a storage-index from any cap). I should clarify this a bit. We spent some time talking about this a year ago, during earlier work on DSA-based mutable filecaps. At that time, we were only thinking about server-side validation of shares to prevent a "roadblock attack", in which someone who learns your storage index (perhaps the first server you talk to) quickly runs to all the other servers and uploads junk data under that SI, preventing you from storing a share under the same index. These days, we'd like to replace the mutable-file "write enabler" (a shared secret that allows a writecap holder to modify an existing share) with a signed share-update message: specifically, we'd like the server to accept any update which results in 1) a valid share, with 2) a higher sequence number than the previous version. (this would let us stop using an encrypted channel to the storage servers, and enable mutable-file repair from non-writecaps, among other things). So we are now more interested in full server-side validation than we were before. Each filecap has some number of bits. Each bit can fulfill one or more purposes. Some of the purposes (and their abbreviations for this note) are: * "C": confidentiality * "I": Integrity * "Ir": Integrity, but only useable by the holder of a readcap For reference, our current URI:CHK: immutable filecaps have two fields: 128 bits of "C" (the readkey), and 256 bits of "I" (the UEB hash). Likewise, the current URI:SSK: mutable writecaps have 128 "C" bits (the writekey) and 256 bits of "I" (the pubkey hash). Our verifycaps are 256 "I" bits, and the storage-index is 128 bits that provide nothing, not even "I". Hashing a field with "C" bits causes them to lose their "C" status. Anything hashed from the share data counts as an "I" bit, but if a secret is mixed in, it only counts as an "Ir" bit. The shortest conceivable writecap/readcap design would have every bit offer both "C" and "I". I believe this to be impossible: if the bit is used for confidentiality, you can't reveal it to a non-readcap-holder, meaning it can't be used for integrity checking by anyone but a readcap-holder. So I think the best you could do is to have readcap bits offer both "C" and "Ir". To enable full server-side validation, we'd like every bit of the storage-index to offer "I": integrity checking that is usable by everybody. The work we were doing a year ago involved mutable readcaps which had one section with "C+Ir" bits and a second section with pure "I" bits. This gave readcap holders a full Ir+I bits of integrity checking, and gave storage servers a smaller (but non-zero) I bits of integrity checking (enough to discourage a roadblock attack but not enough to do full validation and stop using the write-enabler). I think we were planning on a mutable readcap with 128 C+Ir bits and an extra 64 I bits. For immutable files, you have to encrypt+encode the file before you can get the secure hash that forms the root of the integrity check (because we want to validate the individual shares too). Since you must commit to the encryption keys before you get those bits, so you can't just derive the encryption keys from that hash. But you could combine that hash with some other secret, and hide the main encryption key by encrypting it with that secret, and then the bits of that secret would each provide C+Ir. I think this is closely related to Zooko's scheme. Mutable files are easier, because you can sign anything you like. Anything that will get you to the right public key will serve as an I bit: the private key, a semi-private key, or a hash of the public key. The private and semi-private key bits also count as C bits, so writecaps (made by using the private key directly) can be short. Anyways, I hope that there's some way of applying this sort of "C/I/Ir" analysis to filecap schemes that will let us either build something with all the properties we want, or prove that we can't get them so we can stop fussing about and start building some acceptable subset :-). cheers, -Brian From trac at allmydata.org Mon Jul 20 20:44:53 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 21 Jul 2009 03:44:53 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form Message-ID: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- I just tried to run {{{tahoe --version}}} from a .deb I built from trunk (on my sid system), and it fails with the following error: {{{ pkg_resources.VersionConflict: (pycryptopp 0.5.2-1 (/usr/lib/python2.5 /site-packages), Requirement.parse('pycryptopp>=0.5.15')) }}} I should update the debian/control files to make this requirement visible to the debian package manager (i.e. it should not have let me install this tahoe), but apart from that, we need updated pycryptopp debs before we can release 1.5.0 . I think we're ok with zfec and foolscap packages. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Tue Jul 21 07:41:22 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 08:41:22 -0600 Subject: [tahoe-dev] Status of Windows package? Message-ID: <200907210841.22868.shawn-tahoe@willden.org> Hi, I know that someone (sorry, don't recall who) was looking into building and installing Tahoe on Windows a while back. What's the status of that? Assuming it works, how difficult is it? I'd like to start deploying my own friendnet with my family, and they're mostly on Windows. For the more experienced Pythonistas out there, what is the recommended way to create simple installation packages for Windows? I'm thinking about creating an MSI that includes the Python interpreter (2.6), all dependencies and Tahoe. Does that make sense? Thanks, Shawn. From shawn-tahoe at willden.org Tue Jul 21 07:59:17 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 08:59:17 -0600 Subject: [tahoe-dev] HTTP and HTTPS access to a node Message-ID: <200907210859.17379.shawn-tahoe@willden.org> Is it possible to configure a node to listen via both HTTP and HTTPS at the same time? I understand I can set "web.port" in tahoe.cfg to configure SSL access, but I'd like to leave non-SSL access available as well. Ideally I don't want to run two nodes to do this (this is on a Linux Virtual Server with limited RAM -- the node is actually a helper only, if that matters). Thanks, Shawn. From robk-tahoe at allmydata.com Tue Jul 21 08:03:42 2009 From: robk-tahoe at allmydata.com (rob kinninmont) Date: Tue, 21 Jul 2009 16:03:42 +0100 Subject: [tahoe-dev] Status of Windows package? In-Reply-To: <200907210841.22868.shawn-tahoe@willden.org> References: <200907210841.22868.shawn-tahoe@willden.org> Message-ID: <6CF6DD93-977E-49B6-844B-32BFEC9F3E5A@allmydata.com> At Allmydata we used py2exe to build a (suite of) .exe file(s) including all the bundling of python libraries etc. It's not perfect (it doesn't integrate terribly well with .egg files for instance) but it works pretty well. We also used InnoSetup to build setup.exe installers for all the pieces. So, that is to say, what you're asking about is a 'solved problem', or at least once was ;-) I'm not sure how much of that build stuff made it from .com into the open source tahoe repo (I'd look into it, but am busy on something else at the mo) but the above tools are to some extent "known to work". If you have questions, problems, feel free to ask me :-) cheers, rob On Jul 21, 2009, at 3:41 PM, Shawn Willden wrote: > Hi, > > I know that someone (sorry, don't recall who) was looking into > building and > installing Tahoe on Windows a while back. What's the status of that? > Assuming it works, how difficult is it? I'd like to start deploying > my own > friendnet with my family, and they're mostly on Windows. > > For the more experienced Pythonistas out there, what is the > recommended way to > create simple installation packages for Windows? I'm thinking about > creating > an MSI that includes the Python interpreter (2.6), all dependencies > and > Tahoe. Does that make sense? > > Thanks, > > Shawn. > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev From shawn-tahoe at willden.org Tue Jul 21 08:26:50 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 09:26:50 -0600 Subject: [tahoe-dev] Status of Windows package? In-Reply-To: <6CF6DD93-977E-49B6-844B-32BFEC9F3E5A@allmydata.com> References: <200907210841.22868.shawn-tahoe@willden.org> <6CF6DD93-977E-49B6-844B-32BFEC9F3E5A@allmydata.com> Message-ID: <200907210926.50626.shawn-tahoe@willden.org> On Tuesday 21 July 2009 09:03:42 am rob kinninmont wrote: > At Allmydata we used py2exe to build a (suite of) .exe file(s) > including all the bundling of python libraries etc. It's not perfect > (it doesn't integrate terribly well with .egg files for instance) but > it works pretty well. I have fiddled with py2exe, and it seemed to be okay, except, as you say, in the area of dependencies which aren't packaged but might be installed as eggs. > We also used InnoSetup to build setup.exe installers for all the pieces. > > So, that is to say, what you're asking about is a 'solved problem', or > at least once was ;-) I'm not sure how much of that build stuff made > it from .com into the open source tahoe repo (I'd look into it, but am > busy on something else at the mo) but the above tools are to some > extent "known to work". I looked into that question a while back and it appeared that none of the build stuff made it from the .com work. Is there some way we could get it moved over? > If you have questions, problems, feel free to ask me :-) I'll do that! Thanks, Shawn. From trac at allmydata.org Tue Jul 21 06:26:36 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 21 Jul 2009 13:26:36 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.2d8fa9b22950097b1d44c2118bf87b0a@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Whenever a new patch is committed to the pycryptopp repository, the buildbot builds new .deb packages for a couple of platforms -- Debian- unstable-i386 and Ubuntu-Jaunty-amd64: http://allmydata.org/buildbot-pycryptopp/builders/BlackDew%20debian- unstable-i386/builds/10/steps/stdeb/logs/stdio Probably the resulting .deb would work for you. How shall we arrange for it to be installed into an apt repository where your {{{apt-get}}} will find it? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 21 06:40:02 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 21 Jul 2009 13:40:02 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.c2cfbce0abd1774ed86f930455610522@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by arthur): here is a copy of the debian files generated for i386 : http://testgrid.allmydata.org:3567/uri/URI%3ACHK%3Aaws3mawi42itcztaotbeguefry%3A5nufi4nprsitbklcimg3rokoaxccitrpr4xi7yapjvbcpr672x4a%3A3%3A10%3A5438390 Content : pycryptopp_0.5.15-1.diff.gz pycryptopp_0.5.15-1.dsc pycryptopp_0.5.15-1_i386.build pycryptopp_0.5.15-1_i386.changes pycryptopp_0.5.15-1.tar.gz pycryptopp_0.5.15_i386.build pycryptopp_0.5.15.orig.tar.gz pycryptopp_0.5.15.tar.gz pycryptopp-0.5.15.tar.gz python-pycryptopp_0.5.15-1_i386.deb python-pycryptopp-dbg_0.5.15-1_i386.deb -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 21 07:42:05 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 21 Jul 2009 14:42:05 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.1cd4f6a0d8683e87164dc6e121fcb6c0@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Old description: > I just tried to run {{{tahoe --version}}} from a .deb I built from trunk > (on my sid system), and it fails with the following error: > > {{{ > pkg_resources.VersionConflict: (pycryptopp 0.5.2-1 (/usr/lib/python2.5 > /site-packages), Requirement.parse('pycryptopp>=0.5.15')) > }}} > > I should update the debian/control files to make this requirement visible > to the debian package manager (i.e. it should not have let me install > this tahoe), but apart from that, we need updated pycryptopp debs before > we can release 1.5.0 . > > I think we're ok with zfec and foolscap packages. New description: I just tried to run {{{tahoe --version}}} from a .deb I built from trunk (on my sid system), and it fails with the following error: {{{ pkg_resources.VersionConflict: (pycryptopp 0.5.2-1 (/usr/lib/python2.5 /site-packages), Requirement.parse('pycryptopp>=0.5.15')) }}} I've updated the debian/control files to make this requirement visible to the debian package manager (i.e. it should not let me install this tahoe), but apart from that, we need updated pycryptopp debs before we can release 1.5.0 . We need them for all of the same platforms for which we provide Tahoe debs: etch/edgy/feisty/gutsy/hardy/hardy-amd64 . I think we're ok with zfec and foolscap packages. -- Comment(by warner): I updated the tahoe debian/control rules to declare a dependency upon pycryptopp-0.5.15: the tahoe .debs now correctly refuse to install. Zooko: I set up a bunch of "flappserver upload-file" services for uploading tahoe .debs to the repository on hanford (which gets mirrored to allmydata.org). Let's do the same for the pycryptopp debs. In {{{buildslave at hanford}}}, do this one or more times: {{{ flappserver add ~/.flappserver -C DIST/main/binary-i386 upload-file ~/tahoe-debs/dists/DIST/main/binary-i386 }}} (setting DIST appropriately for each time) Each time you run that, it will emit a furl. Copy this to the buildslave, stored in something like ~/main-deb.furl . Then add an upload step to the buildmaster config, which runs: {{{ flappclient --furlfile ~/main-deb.furl upload-file ../pycryptopp*.deb }}} You should also copy the "tahoe-update-apt.furl" and have a step to update the APT index by running {{{ flappclient --furlfile ~/tahoe-update-apt.furl run-command }}} (take a look at any of the tahoe deb builders for an example) I don't know how you ought manage the variety of platforms, though. We build and host tahoe debs for etch/edgy/feisty/gutsy/hardy/hardy-amd64, so it'd be nice to provide pycryptopp debs for all of those. The tahoe .deb process only actually has rules for etch, lenny, and sid: basically we use the etch rules for the py2.4 platforms (etch and edgy), sid for sid, and lenny for everything else. But we run those rules on actual edgy/feisty/etc systems so they declare the right dependencies (in general you can't install a newer deb on an older system, because it will declare a dependency upon newer versions of libc, etc). You'd need to find some similar mapping for pycryptopp, and create debs for all the same platforms we do for Tahoe. It sounds like you're only currently creating debs for one of those platforms (sid), and that .deb probably won't install on any of the earlier ones. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Tue Jul 21 08:51:37 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 21 Jul 2009 09:51:37 -0600 Subject: [tahoe-dev] Status of Windows package? In-Reply-To: <200907210926.50626.shawn-tahoe@willden.org> References: <200907210841.22868.shawn-tahoe@willden.org> <6CF6DD93-977E-49B6-844B-32BFEC9F3E5A@allmydata.com> <200907210926.50626.shawn-tahoe@willden.org> Message-ID: <656A8FD2-ACD4-4E78-B549-A026E7AB51FD@zooko.com> Check out the behavior of the Windows buildslave. There is a step called "installer". http://allmydata.org/buildbot/builders/windows/builds/1556 There is a different code-base named tahoe-w32-client: http://allmydata.org/trac/tahoe-w32-client/browser It also has a buildbot that you can inspect, to see how it is built. I believe that tahoe-w32-client requires a tahoe-lafs package, and then adds Windows filesystem integration (CIFS/SMB) resulting in a "Virtual Drive" on your Windows desktop, and it adds a GUI Backup Application. If you are looking to package your own Tahoe-based things, you might investigate using bbfreeze instead of py2exe: http://pypi.python.org/pypi/bbfreeze I haven't used it, but it seems to have good features and to be actively developed. Regards, Zooko From shawn-tahoe at willden.org Tue Jul 21 08:55:25 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 09:55:25 -0600 Subject: [tahoe-dev] Lock files in Tahoe directories Message-ID: <200907210955.25547.shawn-tahoe@willden.org> I have a situation where I need to allow multiple clients to update directories full of mutable files in a controlled way. On a local file system, I'd use lockfiles to synchronize/serialize access. I'm looking for some way to do that effectively with Tahoe files. According to mutable.txt, the Prime Coordination Directive is "Don't Do That". Specifically, uncoordinated writes to tahoe mutable slots are just asking for bad things to happen. However, it occurs to me that perhaps I can coordinate writes through Tahoe if I use different slots to do it. The idea is that I'll create a "lockfile directory", in which each client will write a lockfile named, say, ".lock". This directory and its lockfiles will be populated in a controlled way so that no two clients are updating it at once. The lockfiles will all be empty to begin with. When a client needs to update shared files, it will do the following: 1. Scan the lockfile dir and load all lockfile contents. 2. If any lockfile is non-empty, sleep a random amount of time (say, 20-60s) and go to step 1. 3. Write a timestamp and textual description of the planned update into its lockfile. 4. Wait a few seconds (say 10-20) and re-read all lockfile contents. 5. If any lockfile other than its own is non-empty, truncate its lockfile and go to step 1. 6. Make updates to shared files. 7. Truncate the lockfile. The use of a timestamp and update description in step 3 is just to facilitate manual debugging. The big question is whether or not the delay in step 4 is sufficient and, indeed, whether *any* amount of delay is sufficient to guarantee that write conflicts cannot occur. I guess the best approach would be to implement a coordination server, as suggested by mutable.txt. Perhaps if someone who knows the codebase outlined where I should look to get started on that, mayabe I could do that. I would think coordination should be a service offered by a node, rather than a node type, probably enabled by setting "[coordinator]/enabled=true" in tahoe.cfg, similar to the helper config. Comments, suggestions? Thanks, Shawn. From shawn-tahoe at willden.org Tue Jul 21 08:59:35 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 09:59:35 -0600 Subject: [tahoe-dev] Lock files in Tahoe directories In-Reply-To: <200907210955.25547.shawn-tahoe@willden.org> References: <200907210955.25547.shawn-tahoe@willden.org> Message-ID: <200907210959.35221.shawn-tahoe@willden.org> On Tuesday 21 July 2009 09:55:25 am Shawn Willden wrote: > According to mutable.txt, the Prime Coordination Directive is "Don't Do > That". Specifically, uncoordinated writes to tahoe mutable slots are just > asking for bad things to happen. Sorry to follow up to myself, but I just noticed another possible option: What if all clients are using the same helper to make updated to the shared dirs? Does the internal serialization mentioned in mutable.txt allow the helper to ensure that updates are serialized? If that's the case, then for the moment I could probably just use normal lock file techniques and rely on the helper to ensure that all clients have a consistent view of the behavior. I could probably write some tests to check whether or not that works, but I'd like to know whether or not it theoretically should work before I bother. Shawn. From warner at lothar.com Tue Jul 21 09:04:09 2009 From: warner at lothar.com (Brian Warner) Date: Tue, 21 Jul 2009 12:04:09 -0400 Subject: [tahoe-dev] HTTP and HTTPS access to a node In-Reply-To: <200907210859.17379.shawn-tahoe@willden.org> References: <200907210859.17379.shawn-tahoe@willden.org> Message-ID: > Is it possible to configure a node to listen via both HTTP and HTTPS > at the > same time? Not without code changes. They wouldn't be too hard to implement, though. I'd suggest making web.port take a comma-separated list of strports, then having web.py create and attach a new listening port for each one. Should be maybe 5 lines, plus tests and docs. cheers, -Brian From shawn-tahoe at willden.org Tue Jul 21 09:07:40 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 10:07:40 -0600 Subject: [tahoe-dev] HTTP and HTTPS access to a node In-Reply-To: References: <200907210859.17379.shawn-tahoe@willden.org> Message-ID: <200907211007.40686.shawn-tahoe@willden.org> On Tuesday 21 July 2009 10:04:09 am Brian Warner wrote: > > Is it possible to configure a node to listen via both HTTP and HTTPS > > at the > > same time? > > Not without code changes. They wouldn't be too hard to implement, > though. I'd suggest making web.port take a comma-separated list of > strports, then having web.py create and attach a new listening port > for each one. Should be maybe 5 lines, plus tests and docs. I'll look into that, thanks. Shawn. From zooko at zooko.com Tue Jul 21 09:22:48 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 21 Jul 2009 10:22:48 -0600 Subject: [tahoe-dev] v1.5.0 is feature-complete! In-Reply-To: References: Message-ID: <8BBDC2CE-4956-495A-8390-466ACFCB6B16@zooko.com> On Monday,2009-07-20, at 8:17 , Zooko Wilcox-O'Hearn wrote: > So, test it out! Upgrade your grids to the current trunk and > report any anomalies! Ookay, nobody has reported any anomalies. Let's try this: Test it out! Upgrade your grids to the current trunk and report any successes! :-) Regards, Zooko From terrellrussell at gmail.com Tue Jul 21 09:27:36 2009 From: terrellrussell at gmail.com (Terrell Russell) Date: Tue, 21 Jul 2009 12:27:36 -0400 Subject: [tahoe-dev] v1.5.0 is feature-complete! In-Reply-To: <8BBDC2CE-4956-495A-8390-466ACFCB6B16@zooko.com> References: <8BBDC2CE-4956-495A-8390-466ACFCB6B16@zooko.com> Message-ID: <4A65EC78.2090002@gmail.com> works for me. darcs trunk. terrell On 7/21/09 12:22 PM, Zooko Wilcox-O'Hearn wrote: > On Monday,2009-07-20, at 8:17 , Zooko Wilcox-O'Hearn wrote: > >> So, test it out! Upgrade your grids to the current trunk and >> report any anomalies! > > Ookay, nobody has reported any anomalies. Let's try this: > > Test it out! Upgrade your grids to the current trunk and report any > successes! :-) > > Regards, > > Zooko From warner at lothar.com Tue Jul 21 12:37:06 2009 From: warner at lothar.com (Brian Warner) Date: Tue, 21 Jul 2009 15:37:06 -0400 Subject: [tahoe-dev] Lock files in Tahoe directories In-Reply-To: <200907210959.35221.shawn-tahoe@willden.org> References: <200907210955.25547.shawn-tahoe@willden.org> <200907210959.35221.shawn-tahoe@willden.org> Message-ID: <15BC6699-9CBA-4E94-BECB-51E51E0F4A92@lothar.com> > What if all clients are using the same helper to make updated to the > shared > dirs? Does the internal serialization mentioned in mutable.txt > allow the > helper to ensure that updates are serialized? Nope, unfortunately the helper only handles immutable uploads. If it helped with mutable publishes too, then yeah, that would probably cover it. But it doesn't. (we only implemented one of the four things that a Helper might help with, since immutable-download, mutable-publish, and mutable-retrieve weren't as important). I think that a central lock manager is the way to go. A long time ago we talked about this, and I think we were planning on storing a FURL in the mutable file somewhere, and the rule would be that clients who see a FURL there are obligated to contact it and claim a lock before starting the read-modify-write sequence. Our RSA mutable-file design doesn't leave space for arbitrary extra data, though.. maybe I'll add a note to the DSA design notes page about providing extension space for the future... cheers, -Brian From shawn-tahoe at willden.org Tue Jul 21 12:56:37 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 21 Jul 2009 13:56:37 -0600 Subject: [tahoe-dev] Lock files in Tahoe directories In-Reply-To: <15BC6699-9CBA-4E94-BECB-51E51E0F4A92@lothar.com> References: <200907210955.25547.shawn-tahoe@willden.org> <200907210959.35221.shawn-tahoe@willden.org> <15BC6699-9CBA-4E94-BECB-51E51E0F4A92@lothar.com> Message-ID: <200907211356.37570.shawn-tahoe@willden.org> On Tuesday 21 July 2009 01:37:06 pm Brian Warner wrote: > I think that a central lock manager is the way to go. A long time ago > we talked about this, and I think we were planning on storing a FURL > in the mutable file somewhere mutable.txt in the docs mentions putting the mutable server FURL in the dirnode metadata. That's not quite as good as putting it in the mutable file itself, since another dirnode linking the same mutable file could omit the FURL, but it would be much easier to implement. Shawn. From warner at lothar.com Tue Jul 21 13:31:08 2009 From: warner at lothar.com (Brian Warner) Date: Tue, 21 Jul 2009 16:31:08 -0400 Subject: [tahoe-dev] Lock files in Tahoe directories In-Reply-To: <200907211356.37570.shawn-tahoe@willden.org> References: <200907210955.25547.shawn-tahoe@willden.org> <200907210959.35221.shawn-tahoe@willden.org> <15BC6699-9CBA-4E94-BECB-51E51E0F4A92@lothar.com> <200907211356.37570.shawn-tahoe@willden.org> Message-ID: <4A66258C.8040501@lothar.com> Shawn Willden wrote: > mutable.txt in the docs mentions putting the mutable server FURL in the > dirnode metadata. That's not quite as good as putting it in the mutable file > itself, since another dirnode linking the same mutable file could omit the > FURL, but it would be much easier to implement. True. The lack of node metadata is frustrating.. you just can't quite simulate it with edge metadata. -Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 259 bytes Desc: OpenPGP digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090721/84b97bd5/attachment.pgp From warner at lothar.com Tue Jul 21 13:51:29 2009 From: warner at lothar.com (Brian Warner) Date: Tue, 21 Jul 2009 16:51:29 -0400 Subject: [tahoe-dev] Lock files in Tahoe directories In-Reply-To: <200907210955.25547.shawn-tahoe@willden.org> References: <200907210955.25547.shawn-tahoe@willden.org> Message-ID: <4A662A51.1060502@lothar.com> Shawn Willden wrote: > The idea is that I'll create a "lockfile directory", in which each > client will write a lockfile named, say, ".lock". This > directory and its lockfiles will be populated in a controlled way so > that no two clients are updating it at once. The lockfiles will all be > empty to begin with. > The big question is whether or not the delay in step 4 is sufficient > and, indeed, whether *any* amount of delay is sufficient to guarantee > that write conflicts cannot occur. Nope. I think it's the CAP theorem that says you can't get both consistency and availability in the presence of partitions. The closer you get to perfection in one axis, the worse it gets in the other ones. In vague terms, using mutable files to improve consistency in mutable files is a percentage thing: if you assume some sort of probability distribution of conflict given a certain write frequency and delay time, then using one mutable file as a lock for another might get you a P1*P2 probability of conflict. Using a coordination server is a lot simpler, and will get you perfect consistency, at the expense of availability: if you can't reach the coordination server, you aren't allowed to do anything. Also because (fundamentally speaking) connections are merely a useful fiction created out of non-guaranteed messages and timeouts, there are lots of failure modes which could cause lost progress. The complexity of this stuff is what prompted us to write down the Prime Coordination Directive ("don't do that!") and move on to other less-impossible tasks, like Accounting :-). > I guess the best approach would be to implement a coordination server, > as suggested by mutable.txt. Perhaps if someone who knows the codebase > outlined where I should look to get started on that, mayabe I could do > that. I would think coordination should be a service offered by a > node, rather than a node type, probably enabled by setting > "[coordinator]/enabled=true" in tahoe.cfg, similar to the helper > config. Yeah, definitely. We're certainly moving to a model in which you start a generic "node", running one or more services, and a coordinator would be one of those service types. (maybe use a more descriptive name.. "write-coordinator" ? "consistency-coordinator" ? I'm not sure.) You could have each coordinator announce itself through the Introducer, and then other clients could subscribe to hear about them. Or you could have the coordinator-providing node simply write its FURL to a local file (like we do with the helper) and then manually copy it into a [client]coordinator.furl= entry in the clients who should use it. You'd have to decide whether all mutable files will be accessed through the coordinator, or if there should be some flag to enable/disable the coordinator for each dirnode.add_child operation (it'd probably be easier to add the coordinator FURL to the parent dirnode's edge metadata for everything). You might also want a tahoe.cfg setting to stop using the coordinator altogether (even if the edge metadata said it should be used), if the coordinator goes away and you want to regain availability at the expense of consistency. Also, it might be easier to just have a single client-wide coordinator FURL, instead of putting a separate one into each mutable file object. If all of your clients use the same one, then you'll get the same consistency properties. We were only talking about having per-file coordinators to make the system more scalable: if your grid only has tens or hundreds of clients, it'd be simpler to just use a single static coordinator. cheers, -Brian From trac at allmydata.org Tue Jul 21 13:19:55 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 21 Jul 2009 20:19:55 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.2cb9114fc66474dff4df8deb5727f14f@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by warner): Oh, or of course you could just log in to each of our debian/ubuntu buildslaves in turn, build .deb/.orig.tar.gz/.diff.gz packages, scp them over to hanford:~buildslave/tahoe-debs/... , and run 'make' to update the index and remirror to org. Any level of automation you want to apply to this process would make your life easier in the long run. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Tue Jul 21 15:25:54 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 21 Jul 2009 16:25:54 -0600 Subject: [tahoe-dev] Fwd: Zooko's semi-private keys References: <20090721191112.2111814F6E1@finney.org> Message-ID: <93A8450B-F015-412D-9F05-8AD946781267@zooko.com> Folks: Hal Finney is a very good thinker about cryptography. Years ago, the Freenet developers used to call him by the nickname "Gandalf" because he would wander into their mailing list and bequeath sage wisdom. I've sometimes wished he would do the same for Tahoe-LAFS, so I'm delighted to see that he posted the following to the cryptography mailing list. Regards, Zooko Begin forwarded message: > From: hal at finney.org ("Hal Finney") > Date: Tuesday, July,2009-07-21 13:11:12 0 MDT > To: cryptography at metzdowd.com, jamesd at echeque.com, > tanja at hyperelliptic.org, zooko at zooko.com > Subject: Zooko's semi-private keys > > Zooko's proposal for "semi-private keys" is worthy of discussion > here I think. The full idea is in http://allmydata.org/~zooko/ > lafs.pdf but I will present it here for your enjoyment (I should > emphasize that I played no part in any of the development of this > idea, I just read his PDF). Apologies in advance for any > misunderstandings or misrepresentations I make. I also want to > apologize for using the term "secret key" rather than Zooko's > preferred "private key" so that we can call it SK and then the > public key is PK. The semi-private key will be SPK. Below, "^" > refers to repeated iteration of the group operation, in a given group. > > With most public key systems, knowing the SK allows you to deduce > the PK, but of course you can't go the other direction: > > SK -> PK > PK /-> SK > > Zooko wishes to introduce an intermediate data structure, the semi- > private key or SPK, which fits in between: > > SK -> SPK > SPK -> PK > PK /-> SPK > SPK /-> SK > > From the SK you can deduce the SPK, and from the SPK the PK, but > you can't go backwards. > > The reason for this is he wants to issue signatures with the SK > which can be verified by people holding the SPK and by (different) > people holding the PK; and further he wants to be able to create an > encryption key from the SPK, which will (therefore) also be known > to the holder of the SK, but not to holders of the PK. In this way > the holder of an SK can have two groups of verifiers: PK holders > can check signatures but not read data, while SPK holders can both > check signatures and read data. > > The part that makes this interesting is that it is desired that all > of SK, SPK and PK are as small as possible for a given security > level. This is because of the goal in the Tahoe object-capability > distributed file system of using these keys as identifiers for > files. There would be 3 different identifiers for a file in this > system, corresponding to the SK, SPK and PK associated with that > file, and each identifier would give the different levels of > cryptographic access described above. > > If we didn't have this requirement, we could solve it trivially by > augmenting the SK with an ordinary encryption key k, and defining > SPK as (PK,k). Then the information arrows above would be met. But > we have increased the SK and SPK key sizes by the size of k, which > may be a very substantial percentage when we are dealing with EC keys. > > Zooko proposes a specific alternative in his paper, a modification > of DSA (equivalently, ECDSA). In ordinary DSA, SK is a secret > exponent x in some group, and PK is g^x for some generator g of the > group. Zooko defines: > > y = H(g^x) using hash function H > > and proposes that: > > SK is xy > SPK is g^x > PK is g^(xy) > > It is easy to see that this satisfies the information arrows SK -> > SPK -> PK. There are no obvious ways to go the other way, but > analysis is needed to verify that. As far as key size, neither SK > nor PK gets any bigger, and SPK is the same size as PK. > > DSA signatures are then done in the usual way, using xy in place of > the usual secret exponent x. For reference, and slightly > simplified, the usual DSA formulas are: > > r = g^k for some random k; > s is derived from sk * rx = h (mod the group order) > > where h is the message hash. Then Zooko's modification merely > replaces x by xy (keep in mind that sometimes y is defined as the > public key g^x in DSA descriptions, but here y is different, y = H > (g^x)): > > r = g^k for some random k; > s is derived from sk * rxy = h (mod the group order) > > Zooko asks for cryptographic community review of this proposal. I > believe I can sketch a couple of simple proofs in the random oracle > model that being able to forge signatures, knowing either SK or > SPK, would allow forging ordinary DSA. This message is getting > pretty long though, so perhaps others will wish to chime in. > > The remaining questions are whether PK /=> SPK holds, and SPK /=> > SK. The second part is easily shown to reduce to discrete log. > Suppose we have an oracle which, given g^x, produces x*H(g^x). Then > we can just divide by H(g^x) and get x, turning it into a discrete > log oracle. > > The first is equivalent to: knowing g^(xy) is it impossible to > deduce g^x, where y = H(g^x). Define Y = g^x, then y = H(Y) and g^ > (xy) = Y^H(Y). The question is then: > > Given Y^H(Y) can we deduce Y? > > Ideally we'd like to show that an oracle that could solve this > could be used to find arbitrary discrete logs or break some other > standard assumption, but I can't see how to do it. (I also can't > see how to go the other way, from a discrete log oracle to > something that can solve this - seems like a hard problem.) > > Hal Finney From trac at allmydata.org Wed Jul 22 05:59:15 2009 From: trac at allmydata.org (tahoe-lafs) Date: Wed, 22 Jul 2009 12:59:15 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.46e4044ab9e019c1ed7f1df87f80204e@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: new Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Hm, I don't personally own or operate any of the Debian buildslaves currently connected to the pycryptopp buildbot -- http://allmydata.org /buildbot-pycryptopp/waterfall -- just the jaunty/amd64 one (yukyuk). I guess in order to automatically produce .deb's for these various flavors of Debian, we need to add buildslaves for pycryptopp running each flavor? I'm not going to make time in the forseeable future to set up lots of buildslaves for various flavors of Debian, although I can definitely do Jaunty/amd64 right now... Hm, the first step in your recipe doesn't work: {{{ buildslave at hanford:~$ flappserver add ~/.flappserver -C jaunty/main /binary-amd64 upload-file ~/tahoe-debs/dists/jaunty/main/binary-amd64 Traceback (most recent call last): File "/usr/bin/flappserver", line 18, in run_flappserver() File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 471, in run_flappserver r = dispatch(command, so) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 440, in dispatch return c.run(options) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 191, in run s = build_service(service_basedir, None, service_type, service_args) File "/usr/lib/python2.5/site-packages/foolscap/appserver/services.py", line 279, in build_service raise UnknownServiceType(service_type) foolscap.appserver.services.UnknownServiceType: -C }}} Anyway, as I was saying, I'll spend some time setting up automation to produce .deb's for jaunty/amd64, and I'll also set up a server owned by allmydata.com for lenny/i386, but I don't care about the various other flavors of Debian enough to set up .deb-producing automation for them. If someone else wants to volunteer to do the buildslave side of this recipe for a Debian flavor that they care about then I'll do the buildmaster side of the recipe. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Wed Jul 22 12:02:07 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Wed, 22 Jul 2009 13:02:07 -0600 Subject: [tahoe-dev] HTTP and HTTPS access to a node In-Reply-To: References: <200907210859.17379.shawn-tahoe@willden.org> Message-ID: <200907221302.08250.shawn-tahoe@willden.org> On Tuesday 21 July 2009 10:04:09 am Brian Warner wrote: > > Is it possible to configure a node to listen via both HTTP and HTTPS > > at the same time? > > Not without code changes. They wouldn't be too hard to implement, > though. I'd suggest making web.port take a comma-separated list of > strports, then having web.py create and attach a new listening port > for each one. Should be maybe 5 lines, plus tests and docs. It is pretty simple, but I'm unsure about some of the side-effects. One that I've come across is the node.url file. Should I write all of the URLs to this file? It looks like there are a number of users of this file, and they expect to find a single URL in it. Shawn. From shawn-tahoe at willden.org Wed Jul 22 15:22:20 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Wed, 22 Jul 2009 16:22:20 -0600 Subject: [tahoe-dev] darcs patch: Client support for listening on multiple ports Message-ID: <1248301340.179576.28279.nullmailer@mordsith.willden.org> Wed Jul 22 16:16:54 MDT 2009 Shawn Willden * Client support for listening on multiple ports -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/x-darcs-patch Size: 36484 bytes Desc: A darcs patch for your repository! Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090722/a24a629d/attachment-0001.bin From shawn-tahoe at willden.org Wed Jul 22 15:34:02 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Wed, 22 Jul 2009 16:34:02 -0600 Subject: [tahoe-dev] darcs patch: Client support for listening on multiple ports In-Reply-To: <1248301340.179576.28279.nullmailer@mordsith.willden.org> References: <1248301340.179576.28279.nullmailer@mordsith.willden.org> Message-ID: <200907221634.02508.shawn-tahoe@willden.org> On Wednesday 22 July 2009 04:22:20 pm Shawn Willden wrote: > Wed Jul 22 16:16:54 MDT 2009 Shawn Willden > * Client support for listening on multiple ports This still needs unit tests. If someone can point me in the right direction, I'll implement them and then attach the patch to ticket #770. I'm also not certain how to handle the node.url file, and there may be other similar issues. I'm sure this won't go into the tree until after 1.5 is released, but since my nodes are now providing both HTTP and HTTPS services I'm okay with that :-) Shawn. From trac at allmydata.org Thu Jul 23 08:32:16 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 23 Jul 2009 15:32:16 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.aca7bc2e911ea53ca36fafccb580e6d1@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * cc: blackdew@? (added) * keywords: => debian * status: new => assigned Comment: I set up a pycryptopp buildslave on a server owned by allmydata.com named "slave-etch": http://allmydata.org/buildbot-pycryptopp/builders/allmydata.com%20debian- etch-i686 I looked around, but allmydata.com doesn't seem to run a Debian lenny server. As soon as Brian tells me what's wrong with the flappserver setup (above), I'll set that up to receive .deb's at hanford. Meanwhile, does someone want to volunteer to run a Debian lenny (5.0) buildslave? We already have a Debian sid (unstable) buildslave thanks to Black Dew: http://allmydata.org/buildbot-pycryptopp/builders/BlackDew%20debian- unstable-i386/builds/11/steps/stdeb/logs/stdio Oh, I see that "stdeb 0.3-bdew-custom-fixes (/usr/lib/python2.5/site- packages/stdeb-0.3_bdew_custom_fixes-py2.5.egg)" doesn't satisfy the requirement that I just added of {{{stdeb >= 0.3}}}. :-) I'll loosen that requirement to just {{{stdeb}}}. Black Dew: what custom fixes are in that package? I think you reported them all to stdeb upstream (Andrew Straw), but I'm not sure. Black Dew: would you be willing to set up a flapp client to upload the resulting .deb's for Debuan unstable to our apt repository? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 23 10:16:49 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 23 Jul 2009 17:16:49 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.d32b160e41b42f5b7948af0097c251ce@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by warner): sorry, use lowercase "-c" or "--comment", not uppercase "-C". You could manually build pycryptopp debs on each of the allmydata-owned debian buildslaves: we have one for each debian release that we make tahoe .debs for (obviously). This is what I do each time I make a new foolscap release, regardless of whether tahoe depends upon it the new version or not. I have a script to reduce the typing involved, though. The list is in the {{{AdminServers}}} on the (allmydata-only) dev trac instance, or in the comments on the buildmaster config's slavename list. The reason I'm concerned about this is because the tahoe debs that we've been building for years now are useless without all of the support debs. pycryptopp is the only one that we're missing. Until we have updated pycryptopp debs, users of those older platforms (including Hardy, the current Ubuntu long-term-support release) will be forced to build their own, a relatively difficult and annoying process for someone used to simply typing "apt-get install allmydata-tahoe". In addition, servers which are already running hardy (such as the allmydata.com hosts) will be unable to upgrade until their admins manually build pycryptopp debs. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 23 10:24:17 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 23 Jul 2009 17:24:17 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.cc8c2deea1ab62490a22604d1bdb7131@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): I would be happy for Tahoe-LAFS (and pycryptopp) to be apt-get install'able by people who use various Debian platforms and who configure their {{{/etc/apt/sources.list}}} to point to allmydata.org's apt repositories. However, I have very little time to work on fun stuff (Tahoe-LAFS) nowadays, and manually building .deb's doesn't qualify as "fun". If someone pays me to do it, or if someone else does it, then .deb's will manually get built. Fortunately, setting up automation to automatically build .deb's counts as "fun". ;-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ndurner at googlemail.com Thu Jul 23 17:04:04 2009 From: ndurner at googlemail.com (Nils Durner) Date: Fri, 24 Jul 2009 01:04:04 +0100 Subject: [tahoe-dev] GNU Bazaar on Tahoe Message-ID: <4A68FA74.4060503@web.de> Hi, the current development branch (> 1.17; revision 4559) of GNU Bazaar (http://bazaar-vcs.org) supports Tahoe LAFS as storage backend. Here's how to set it up: 1. Install GNU Bazaar > 1.17: bzr checkout lp:bzr; cd bzr; python setup.py install 2. Enable Tahoe's FTP frontend as described in docs/frontends/FTP-and-SFTP.txt. 3. Create a new Bazaar repository (optional): bzr init-repo --1.9 --no-trees ftp://user at localhost:8021/ 4. Create a new branch: bzr init ftp://user at localhost:8021/my_branch 5. Checkout: bzr checkout ftp://user at localhost:8021/my_branch 6. Add files cd my_branch; touch testfile; bzr add testfile 7. Commit: bzr commit -m start Common problems: * bzr fails with "bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 5: ordinal not in range(128)" Bazaar tries to determine the local user name if it is not part of the FTP URL. If the name contains special characters like umlauts, bzr fails with this error message. Solution: specify the FTP login as part of the URL: ftp://username at host * Large pushes (uploads of several revisions) are very slow, bzr warns that "FTP server does not support file appending natively. Performance may be severely degraded! (502 Command 'APPE' not implemented)" Due to the limited command set of the Twisted FTP server, file appends have to be done on the client side. This means that the current write group is downloaded and reuploaded (with the next revision appended) for each revision in the current push. You can probably work around this problem by preparing a suitable branch locally and uploading it using an ordinary FTP tool. * bzr warning: "FTP Could not set permissions to 0700 on /my_branch/.bzr. 502 Command 'SITE' not implemented" File permissions are not preserved, but that doesn't really matter. * Ineffective file locking Protection against concurrent writes is achieved through lock files, which does not work reliably with Tahoe AFAIK. Bigger teams with a shared branch should probably therefore employ an automatic gatekeeper (http://bazaar-vcs.org/Workflows#Decentralized%20with%20automatic%20gatekeeper) Benefits: * "SCM in the cloud" o central repository without a central server o inherent backup * secure storage o only team members have access to the files (IT staff doesn't) * granting read-only access is easily possible (via read caps) * (non-bzr) access possible via all available Tahoe frontends: HTTP, S/FTP, CLI, ... Regards, Nils From trac at allmydata.org Thu Jul 23 22:48:59 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 24 Jul 2009 05:48:59 -0000 Subject: [tahoe-dev] [tahoe-lafs] #771: tahoe ls doesn't work on files Message-ID: <037.e542a2265e42d3a41c8e58e5d1f98a9e@allmydata.org> #771: tahoe ls doesn't work on files ---------------------+------------------------------------------------------ Reporter: kevan | Owner: nobody Type: defect | Status: new Priority: major | Milestone: undecided Component: unknown | Version: 1.4.1 Keywords: | Launchpad_bug: ---------------------+------------------------------------------------------ If I have an immutable file {{{file2}}} at {{{testdir/file2}}} on my remote filesystem, and do {{{tahoe ls testdir/file2}}}, I get {{{ Traceback (most recent call last): File "/Users/kacarstensen/Documents/code/tahoe/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1-r4026', 'console_scripts', 'tahoe')() File "/Users/kacarstensen/Documents/code/tahoe/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/Users/kacarstensen/Documents/code/tahoe/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/Users/kacarstensen/Documents/code/tahoe/src/allmydata/scripts/cli.py", line 412, in list rc = tahoe_ls.list(options) File "/Users/kacarstensen/Documents/code/tahoe/src/allmydata/scripts/tahoe_ls.py", line 70, in list childtype = child[0] KeyError: 0 }}} If I do {{{tahoe ls --json testdir/file2}}}, I get {{{ [ "filenode", { "mutable": false, "ro_uri": "URI:LIT:a_ro_cap", "size": 13 } ] }}} If I do {{{tahoe ls --json testdir/}}}, I get {{{ [ "dirnode", { "rw_uri": "URI:DIR2:rw_cap", "verify_uri": "URI:DIR2-Verifier:verify_cap", "ro_uri": "URI:DIR2-RO:ro_cap", "children": { "dir2": [ "dirnode", { "mutable": true, "verify_uri": "URI:DIR2-Verifier:verify_cap", "rw_uri": "URI:DIR2:rw_cap", "ro_uri": "URI:DIR2-RO:ro_cap", "metadata": { "ctime": 1248055314.2892411, "tahoe": { "linkmotime": 1248055314.2892411, "linkcrtime": 1248055314.2892411 }, "mtime": 1248055314.2892411 } } ], "test.pdf": [ "filenode", { "mutable": false, "verify_uri": "URI:CHK-Verifier:verify_cap", "metadata": { "ctime": 1246685054.166369, "tahoe": { "linkmotime": 1246685054.166369, "linkcrtime": 1246685054.166369 }, "mtime": 1246685054.166369 }, "ro_uri": "URI:CHK:ro_cap", "size": 700145 } ], "file2": [ "filenode", { "mutable": false, "metadata": { "ctime": 1247192345.969193, "tahoe": { "linkmotime": 1247192345.969193, "linkcrtime": 1247192345.969193 }, "mtime": 1247192345.969193 }, "ro_uri": "URI:LIT:ro_cap", "size": 13 } ] }, "mutable": true } ] }}} If I do {{{tahoe ls testdir/}}}, it works as normal. I think the problem is that we haven't correctly implemented parsing for {{{GET}}}ing a filecap. Instead of doing, as in source:/src/allmydata/scripts/tahoe_ls.py#L55, {{{ children = {childname: d} }}} thus losing the {{{child[0]}}} item that the loop is looking for, we should do something like {{{ children = {childname: [nodetype, d]} }}} Unfortunately, this too fails because the loop wants to find metadata that isn't there. I think the best solution here would be (one way or another) to follow source:/docs/frontends/webapi.txt#L402 -- that is, to {{{GET}}} the file's information using the containing directory's dircap, rather than the filecap of the file. That way, we have the metadata that we want for, e.g., {{{tahoe ls -l}}}. I'm just kind of stuck on the best way to do that. The most workable idea I've come up with is to check the results from the first {{{GET}}}, and, if they're for a {{{filenode}}} without metadata, attempt to re-fetch them by using {{{get_alias}}} on the parent directory (which we could infer by parsing the {{{where}}} option. This seems kind of roundabout, though -- it seems like it'd be easier to determine, before the initial {{{GET}}}, whether the rootcap returned from {{{get_alias}}} corresponded to a directory or a file, and do that check there. I'm just not sure if there's an easy way to do that, hence this long-winded explanation. If there's a more obvious way, I'm open to that, too. I'm attaching a patch with a test that demonstrates this bug. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 24 06:07:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 24 Jul 2009 13:07:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.35fb49eb3561932a08b582459ec6adef@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): > sorry, use lowercase "-c" or "--comment", not uppercase "-C". I thought I tried that... Yes, I did: {{{ buildslave at hanford:~$ flappserver add ~/.flappserver -c jaunty/main /binary-amd64 upload-file ~/tahoe-debs/dists/jaunty/main/binary-amd64 Traceback (most recent call last): File "/usr/bin/flappserver", line 18, in run_flappserver() File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 471, in run_flappserver r = dispatch(command, so) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 440, in dispatch return c.run(options) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 191, in run s = build_service(service_basedir, None, service_type, service_args) File "/usr/lib/python2.5/site-packages/foolscap/appserver/services.py", line 279, in build_service raise UnknownServiceType(service_type) foolscap.appserver.services.UnknownServiceType: -c buildslave at hanford:~$ flappserver add ~/.flappserver --comment jaunty/main /binary-amd64 upload-file ~/tahoe-debs/dists/jaunty/main/binary-amd64 Traceback (most recent call last): File "/usr/bin/flappserver", line 18, in run_flappserver() File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 471, in run_flappserver r = dispatch(command, so) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 440, in dispatch return c.run(options) File "/usr/lib/python2.5/site-packages/foolscap/appserver/cli.py", line 191, in run s = build_service(service_basedir, None, service_type, service_args) File "/usr/lib/python2.5/site-packages/foolscap/appserver/services.py", line 279, in build_service raise UnknownServiceType(service_type) foolscap.appserver.services.UnknownServiceType: --comment }}} Since I get only about an hour to work on Tahoe-LAFS every day (at most, and in several small bursts), and since you and I are out of sync so that it takes about 24 hours for every round trip between us, maybe you could set up the flappserver on hanford? Or, if you don't want to, then please write back and tell me what's wrong with my current flappserver setup command-line. :-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Fri Jul 24 06:09:27 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 24 Jul 2009 13:09:27 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.42673758ece4bef29f1feeeffe5e9ffe@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * cc: bdew (added) Comment: We now have three Debian buildslaves which are successfully generating .deb's: http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- jaunty-yukyuk http://allmydata.org/buildbot-pycryptopp/builders/BlackDew%20debian- unstable-i386 http://allmydata.org/buildbot-pycryptopp/builders/allmydata.com%20debian- etch-i686 The one that I don't have a login to is the one that I most want to have .deb's for -- Debian unstable. Hopefully Black Dew will be willing to set up the client side flapp service to upload the .deb's that his buildslave is building. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Fri Jul 24 06:33:29 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Fri, 24 Jul 2009 07:33:29 -0600 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated Message-ID: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> [cross-posted to tahoe-dev at allmydata.org and cryptography at metzdowd.com] Disclosure: Cleversafe is to some degree a competitor of my Tahoe- LAFS project. On the other hand, I tend to feel positive towards them because they open-source much of their work. Our "Related Projects" page has included a link to cleversafe for years now, I briefly collaborated with some of them on a paper about erasure coding last year, and I even spoke briefly with them about the idea of becoming an employee of their company this year. I am tempted to ignore this idea that they are pushing about encryption being overrated, because they are wrong and it is embarassing. But I've decided not to ignore it, because people who publicly spread this kind of misinformation need to be publicly contradicted, lest they confuse others. Cleversafe has posted a series of blog entries entitled "3 Reasons Why Encryption is Overrated". http://dev.cleversafe.org/weblog/?p=63 # 3 Reasons Why Encryption is Overrated http://dev.cleversafe.org/weblog/?p=95 # Response Part 1: Future Processing Power http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: Complexities of Key Management http://dev.cleversafe.org/weblog/?p=178 # Response Part 3: Disclosure Laws It begins like this: """ When it comes to storage and security, discussions traditionally center on encryption. The reason encryption ? or the use of a complex algorithm to encode information ? is accepted as a best practice rests on the premise that while it?s possible to crack encrypted information, most malicious hackers don?t have access to the amount of computer processing power they would need to decrypt information. But not so fast. Let?s take a look at three reasons why encryption is overrated. """ Ugh. The first claim -- the today's encryption is vulnerable to tomorrow's processing power -- is a common goof, which is easy to make by conflating historical failures of cryptosystems due to having too small of a crypto value with failures due to weak algorithms. Examples of the former are DES, which failed because its 56-bit key was small enough to fall to brute force, and the bizarre "40-bit security" policies of the U.S. Federal Government in the 90's. An example of the latter is SHA1, whose hash output size is *not* small enough to brute-force, but which is insecure because, as it turns out, the SHA1 algorithm allows the generation of colliding inputs much quicker than a brute force search would. Oh boy, I see that in the discussion following the article "Future Processing Power", the author writes: """ I don?t think symmetric ciphers such as AES-256 are under any threat of being at risk to brute force attacks any time this century. """ What? Then why is he spreading this Fear, Uncertainty, and Doubt? Oh and then it gets *really* interesting: it turns out that cleversafe uses AES-256 in an All-or-Nothing Transform as part of their "Information Dispersal" algorithm. Okay, I would like to understand better the cryptographic effects of that (and in particular, whether this means that the cleversafe architecture is just as susceptible to AES-256 failing as an encryption scheme such as is used in the Tahoe-LAFS architecture). But, it is time for me to stop reading about cryptography and get ready to go to work. :-) Regards Zooko --- Tahoe, the Least-Authority Filesystem -- http://allmydata.org store your data: $10/month -- http://allmydata.com/?tracking=zsig I am available for work -- http://zooko.com/r?sum?.html From zookog at gmail.com Fri Jul 24 07:21:54 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Fri, 24 Jul 2009 08:21:54 -0600 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form Message-ID: Folks: I grepped our apache log files to find out how many unique IP addresses had used "apt-get" on our debian repositories for different debian distributions and architectures. These are *attempts* to download. If they got a 404-not-found, they still get counted in this list. The count starts April 12 (just before the Tahoe-LAFS v1.4 release) and goes until today. If my scripts are counting the right things, the results are: ARCH is i386 etch 21 feisty 0 gutsy 1 hardy 24 jaunty 0 lenny 1 sid 1 ARCH is amd64 etch 2 feisty 0 gutsy 0 hardy 4 jaunty 1 lenny 1 sid 0 ARCH is arm etch 0 feisty 0 gutsy 0 hardy 0 jaunty 0 lenny 1 sid 0 ARCH is solaris etch 0 feisty 0 gutsy 0 hardy 1 jaunty 0 lenny 0 sid 0 (Solaris?? That means a user of Nexenta tried using our apt-repository. It wasn't me.) Here is a count of how many people downloaded the various release tarballs: allmydata-tahoe-1.4.1.zip 262 allmydata-tahoe-1.3.0.zip 109 allmydata-tahoe-1.2.0.tar.bz2 87 allmydata-tahoe-1.2.0.tar.gz 78 allmydata-tahoe-1.2.0.zip 45 allmydata-tahoe-1.4.1.tar.bz2 10 allmydata-tahoe-1.4.1.tar.gz 9 allmydata-tahoe-1.4.1-SUMO.zip 8 allmydata-tahoe-1.3.0.tar.gz 6 allmydata-tahoe-1.4.1-SUMO.tar.bz2 6 allmydata-tahoe-1.4.1-SUMO.tar.gz 5 allmydata-tahoe-1.3.0.tar.bz2 3 allmydata-tahoe-1.20.tar.bz2 1 allmydata-tahoe-tahoe-1.3.0.zip 1 allmydata-tahoe-1.4.1.zi 1 Note: The .zip files are the official way to download Tahoe-LAFS as described on http://allmydata.org/source/tahoe/trunk/docs/install.html . To use apt-get they would have to read the sentence at the bottom of that page and follow the link and then search in text on the next page for "debian". To get any of the other source tarballs other than th e .zip file, they would have to do similarly or else edit the URL manually. (The last four filenames listed above are typos from editing the URL manually.) A .tar.gz source dist is also hosted on the Python Package Index: http://pypi.python.org/pypi/allmydata-tahoe At the moment, it says that it has been downloaded 130 times (but that is counting downloads, not unique IP addresses). From zookog at gmail.com Fri Jul 24 07:47:41 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Fri, 24 Jul 2009 08:47:41 -0600 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: References: Message-ID: Aha! I *thought* something smelled funny about these results. I was looking only at one particular access.log file, not the full set of access.log files that extend from April 12 to today. The real results are coming soon... From zookog at gmail.com Fri Jul 24 08:14:53 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Fri, 24 Jul 2009 09:14:53 -0600 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: References: Message-ID: ARCH is i386 dapper 4 edgy 1 etch 84 feisty 4 gutsy 17 hardy 159 intrepid 8 jaunty 2 lenny 6 sid 3 stable 0 testing 0 unstable 0 ARCH is amd64 dapper 0 edgy 0 etch 10 feisty 1 gutsy 0 hardy 23 intrepid 3 jaunty 1 lenny 7 sid 0 stable 0 testing 0 unstable 0 ARCH is powerpc dapper 0 edgy 0 etch 0 feisty 0 gutsy 2 hardy 0 intrepid 0 jaunty 0 lenny 0 sid 0 stable 0 testing 0 unstable 0 ARCH is arm dapper 0 edgy 0 etch 0 feisty 0 gutsy 0 hardy 0 intrepid 0 jaunty 0 lenny 1 sid 0 stable 0 testing 0 unstable 0 ARCH is solaris dapper 0 edgy 0 etch 0 feisty 0 gutsy 0 hardy 1 intrepid 0 jaunty 0 lenny 0 sid 0 stable 0 testing 0 unstable 0 Hm, is this right? This comes from counting lines that match debian/dists/${DIST}/tahoe/binary-${ARCH}.*APT in the access.log. Regards, Zooko From arthur.lutz at gmail.com Fri Jul 24 09:07:59 2009 From: arthur.lutz at gmail.com (Arthur Lutz) Date: Fri, 24 Jul 2009 18:07:59 +0200 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: References: Message-ID: Hi, You have to take into consideration that certain repos are not complete. So for example on one of my lenny boxes I'm actually pointing to etch because no packages were available for lenny. Or on a jaunty I am pointing on hardy. It's cool that it works but you should take that into account in your stats if it ponders the efforts into building debs. Anyone trying to get the packages into the official distributions ? or a PPA for ubuntu ? Arthur On Fri, Jul 24, 2009 at 5:14 PM, Zooko O'Whielacronx wrote: > ARCH is i386 dapper 4 edgy 1 etch 84 feisty 4 gutsy 17 hardy 159 > intrepid 8 jaunty 2 lenny 6 sid 3 stable 0 testing 0 unstable 0 > ARCH is amd64 dapper 0 edgy 0 etch 10 feisty 1 gutsy 0 hardy 23 > intrepid 3 jaunty 1 lenny 7 sid 0 stable 0 testing 0 unstable 0 > ARCH is powerpc dapper 0 edgy 0 etch 0 feisty 0 gutsy 2 hardy 0 > intrepid 0 jaunty 0 lenny 0 sid 0 stable 0 testing 0 unstable 0 > ARCH is arm dapper 0 edgy 0 etch 0 feisty 0 gutsy 0 hardy 0 > intrepid 0 jaunty 0 lenny 1 sid 0 stable 0 testing 0 unstable 0 > ARCH is solaris dapper 0 edgy 0 etch 0 feisty 0 gutsy 0 hardy 1 > intrepid 0 jaunty 0 lenny 0 sid 0 stable 0 testing 0 unstable 0 > > Hm, is this right? This comes from counting lines that match > > debian/dists/${DIST}/tahoe/binary-${ARCH}.*APT > > in the access.log. > > Regards, > > Zooko > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20090724/3f74ce52/attachment.htm From rufus.pollock at okfn.org Fri Jul 24 11:16:22 2009 From: rufus.pollock at okfn.org (Rufus Pollock) Date: Fri, 24 Jul 2009 19:16:22 +0100 Subject: [tahoe-dev] How does file deletion work? In-Reply-To: <20090715104906.251b900b@luther> References: <200907131214.12791.shawn-tahoe@willden.org> <873a90mkf9.fsf@gnu.org> <200907131442.08602.shawn-tahoe@willden.org> <20090715104906.251b900b@luther> Message-ID: 2009/7/15 Brian Warner : > > Note that the default server configuration is conservative and does not > perform any GC at all. You must explicitly enable anything which threatens > data safety like GC. When you enable it, you will provide the expiration > timeout, which of course determines how quickly your server's users must > renew their leases. And in the absence of GC (which may threaten data safety): how do I do file deletion? (From my reading of the rest of your email I think the answer is you can't ...) > Basically, since we have a distributed filesystem (which makes reference > counting expensive at best) with least-authority access semantics (which > makes it impossible for the servers to count references themselves), the > easiest and most reliable approach to deletion is to use leases, timers, and > garbage collection. Right, though couldn't one have alternatives like each node having an optional configuration variable holding the public key of an "administrator". The private key would then act as a capability token granting the ability to e.g. delete nodes etc >> 3. If a file is listed in a directory then this will lead >> automatically to renewal of the relevant leases > > Nope, not without client involvement. As other folks have pointed out, > servers can't read directories (since they don't hold a readcap). So clients > (who *do* hold a readcap, or hold one to a parent directory, which gives them > transitive access) must be responsible for performing renewals. > > (if you do a periodic "tahoe deep-check --add-lease ROOT:", then your > statement becomes true, because your client is doing the work of finding > everything reachable from that directory and renewing the leases, and merely > adding/removing a file to any reachable directory is enough to start/stop > renewing its lease) And this can be done by any storage node with access to that root directory right? [...] > If you're confident that you can enumerate all the files and directories that > you care about, you can periodically compare this manifest against a previous > version, and then send out explicit lease-cancel message for the objects that > are no longer on the list. (the "cancel lease XYZ" message is the closest > we've got to actual server-side deletion). But note that if you get it wrong > (perhaps due to race conditions between two machines modifying a shared > directory), you could accidentally lose files. Adding one lease per starting > point (i.e. per root directory) per walker instance feels like it might avoid > the race worries. Maybe I've missed it in the webapi.txt but can you send lease cancellation via the webapi (and what other lease operations be done that way?) [...] > Another idea (ticket #308) is to change the encryption format of dirnodes to > introduce another level into the "writecap -> readcap -> verifycap" > hierarchy. The new order would be "writecap -> readcap -> traversalcap -> > verifycap", and a traversalcap would give somebody the ability to get the > traversalcaps of all children (as well as the verifycap of the dirnode > itself). Then, if you gave a hypothetical Lease Renewal Service the > traversalcap of your root directory (as well as your master lease-renewing > secret), they could renew all of your leases, but couldn't actually read your > directories or files. (this requires "semi-private DSA keys", see > http://allmydata.org/trac/pycryptopp/ticket/13 for details). You might be > willing to reveal the shape of your directory structure to this service, in > exchange for letting it take responsibility for your lease-renewal duties. If > the service lives network-wise close to the storage servers, it may be > considerably faster too. And certainly is the case where I don't care about giving out information because I'm running a grid for open data! Thanks once again for your deleted explanations. Regards, Rufus From rweeks at gmail.com Fri Jul 24 17:14:40 2009 From: rweeks at gmail.com (Russ Weeks) Date: Fri, 24 Jul 2009 17:14:40 -0700 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated In-Reply-To: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> References: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> Message-ID: <4158677e0907241714k5c70313bra58b5182d605046d@mail.gmail.com> Yikes, that All-Or-Nothing transform, that's an interesting algorithm. Seems to me that it reduces the security of AES-256 to the security of the dispersal algorithm. If I control a subset of malicious nodes within the distributed storage system, and I can convince the sender (via DDOS or some network coordinate trickery, perhaps) to spread K slices of user data amongst my nodes, then I can recover the user data. Why would I take a nice, robust, well-understood algorithm like AES-256 and hobble it with my in-house dispersal algorithm? Because key management is hard? It _is_ hard, definitely, and I don't quite understand how TahoeLAFS approaches the problem (I guess it has to do with these 'caps' you guys keep talking about), but we shouldn't ignore the problem just because it's hard. As for Reason #1, that computers get faster and faster: pick a key size sufficiently large for you to retire well before your customers come calling with pitchforks and torches. As for Reason #3, that disclosure laws are a PITA: Any storage system based on distributing erasure-encoded slices is going to enjoy those benefits, right? I don't see how All-or-Nothing is a big win over a key-management infrastructure. -Russ On Fri, Jul 24, 2009 at 6:33 AM, Zooko Wilcox-O'Hearn wrote: > [cross-posted to tahoe-dev at allmydata.org and cryptography at metzdowd.com] > > Disclosure: ?Cleversafe is to some degree a competitor of my Tahoe- > LAFS project. ?On the other hand, I tend to feel positive towards > them because they open-source much of their work. ?Our "Related > Projects" page has included a link to cleversafe for years now, I > briefly collaborated with some of them on a paper about erasure > coding last year, and I even spoke briefly with them about the idea > of becoming an employee of their company this year. ?I am tempted to > ignore this idea that they are pushing about encryption being > overrated, because they are wrong and it is embarassing. ?But I've > decided not to ignore it, because people who publicly spread this > kind of misinformation need to be publicly contradicted, lest they > confuse others. > > Cleversafe has posted a series of blog entries entitled "3 Reasons > Why Encryption is Overrated". > > http://dev.cleversafe.org/weblog/?p=63 # 3 Reasons Why Encryption is > Overrated > http://dev.cleversafe.org/weblog/?p=95 # Response Part 1: Future > Processing Power > http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: > Complexities of Key Management > http://dev.cleversafe.org/weblog/?p=178 # Response Part 3: Disclosure > Laws > > It begins like this: > > """ > When it comes to storage and security, discussions traditionally > center on encryption. ?The reason encryption ? or the use of a > complex algorithm to encode information ? is accepted as a best > practice rests on the premise that while it?s possible to crack > encrypted information, most malicious hackers don?t have access to > the amount of computer processing power they would need to decrypt > information. > > But not so fast. ?Let?s take a look at three reasons why encryption > is overrated. > """ > > Ugh. > > The first claim -- the today's encryption is vulnerable to tomorrow's > processing power -- is a common goof, which is easy to make by > conflating historical failures of cryptosystems due to having too > small of a crypto value with failures due to weak algorithms. > Examples of the former are DES, which failed because its 56-bit key > was small enough to fall to brute force, and the bizarre "40-bit > security" policies of the U.S. Federal Government in the 90's. ?An > example of the latter is SHA1, whose hash output size is *not* small > enough to brute-force, but which is insecure because, as it turns > out, the SHA1 algorithm allows the generation of colliding inputs > much quicker than a brute force search would. > > Oh boy, I see that in the discussion following the article "Future > Processing Power", the author writes: > > """ > I don?t think symmetric ciphers such as AES-256 are under any threat > of being at risk to brute force attacks any time this century. > """ > > What? ?Then why is he spreading this Fear, Uncertainty, and Doubt? > Oh and then it gets *really* interesting: it turns out that > cleversafe uses AES-256 in an All-or-Nothing Transform as part of > their "Information Dispersal" algorithm. ?Okay, I would like to > understand better the cryptographic effects of that (and in > particular, whether this means that the cleversafe architecture is > just as susceptible to AES-256 failing as an encryption scheme such > as is used in the Tahoe-LAFS architecture). > > But, it is time for me to stop reading about cryptography and get > ready to go to work. ?:-) > > Regards > > Zooko > --- > Tahoe, the Least-Authority Filesystem -- http://allmydata.org > store your data: $10/month -- http://allmydata.com/?tracking=zsig > I am available for work -- http://zooko.com/r?sum?.html > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > From sam at samason.me.uk Fri Jul 24 17:15:38 2009 From: sam at samason.me.uk (Sam Mason) Date: Sat, 25 Jul 2009 01:15:38 +0100 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated In-Reply-To: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> References: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> Message-ID: <20090725001538.GG5407@samason.me.uk> On Fri, Jul 24, 2009 at 07:33:29AM -0600, Zooko Wilcox-O'Hearn wrote: > Cleversafe has posted a series of blog entries entitled "3 Reasons > Why Encryption is Overrated". > > http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: Complexities of Key Management I won't pretend that I understand Cleversafe, but this post seems to spend a lot of time explaining how keys are hard to look after but then happily ignores the problem of how to keep their "masked key" safe (which appears to me to be a very similar problem). Or am I missing something? -- Sam http://samason.me.uk/ From trac at allmydata.org Sat Jul 25 11:15:57 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 25 Jul 2009 18:15:57 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows Message-ID: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Before the official Tahoe-LAFS v1.5.0 release, I would like for someone to follow the official install doc (http://allmydata.org/source/tahoe/trunk/docs/install.html ) on a Windows system. If you've never installed Tahoe-LAFS on your Windows system before, all the better! We have automated tests of build and certain install steps on Windows (http://allmydata.org/buildbot/builders/windows , http://allmydata.org/buildbot/builders/BlackDew%20windows-mingw32-py26 ) and cygwin (http://allmydata.org/buildbot/builders/cygwin ), but I would feel better if an actual human tried it. Hm, say Black Dew, why is your Windows buildslave off-line? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Sat Jul 25 12:02:00 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sat, 25 Jul 2009 13:02:00 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> Folks: Anybody out there who loves Windows? Okay, how about this: anybody out there who has a working Windows system and is willing to use it for an hour or so to make sure Tahoe-LAFS installs cleanly? Thanks! Regards, Zooko On Saturday,2009-07-25, at 12:15 , tahoe-lafs wrote: > #773: run through installation on Windows > ----------------------- > +---------------------------------------------------- > Reporter: zooko | Owner: somebody > Type: task | Status: new > Priority: major | Milestone: 1.5.0 > Component: packaging | Version: 1.4.1 > Keywords: win32 | Launchpad_bug: > ----------------------- > +---------------------------------------------------- > Before the official Tahoe-LAFS v1.5.0 release, I would like for > someone to > follow the official install doc > (http://allmydata.org/source/tahoe/trunk/docs/install.html ) on a > Windows > system. If you've never installed Tahoe-LAFS on your Windows system > before, all the better! We have automated tests of build and certain > install steps on Windows (http://allmydata.org/buildbot/builders/ > windows , > http://allmydata.org/buildbot/builders/BlackDew%20windows-mingw32- > py26 ) > and cygwin (http://allmydata.org/buildbot/builders/cygwin ), but I > would > feel better if an actual human tried it. > > Hm, say Black Dew, why is your Windows buildslave off-line? > > -- > Ticket URL: > tahoe-lafs > secure decentralized file storage grid > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev From zooko at zooko.com Sat Jul 25 12:09:05 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sat, 25 Jul 2009 13:09:05 -0600 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: References: Message-ID: <4D736E36-0EE3-4CF8-92E1-7124F712803D@zooko.com> On Friday,2009-07-24, at 10:07 , Arthur Lutz wrote: > Anyone trying to get the packages into the official distributions ? > or a PPA for ubuntu ? POX and I are working on packaging various Tahoe-LAFS dependencies (e.g. pycryptopp, zfec, pyutil, zbase32, argparse) for Debian. I don't know anything about PPAs for Ubuntu. Why are they better than the apt repos hosted by allmydata.org? Regards, Zooko From jresch at cleversafe.com Sat Jul 25 14:00:16 2009 From: jresch at cleversafe.com (Jason Resch) Date: Sat, 25 Jul 2009 16:00:16 -0500 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated In-Reply-To: <4158677e0907241714k5c70313bra58b5182d605046d@mail.gmail.com> References: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> <4158677e0907241714k5c70313bra58b5182d605046d@mail.gmail.com> Message-ID: <4A6B7260.6070301@cleversafe.com> Russ Weeks wrote: > Yikes, that All-Or-Nothing transform, that's an interesting algorithm. > > It certainly is. When first described by Ron Rivest, it was mostly a curiosity with little practical benefit, and therefore had been mostly forgotten. However, when it is combined with an information dispersal algorithm, the combined system has many of the properties of the Shamir Secret Sharing Scheme, but maintaining the perfect storage efficiency of the IDA. > Seems to me that it reduces the security of AES-256 to the security of > the dispersal algorithm. If I control a subset of malicious nodes > within the distributed storage system, and I can convince the sender > (via DDOS or some network coordinate trickery, perhaps) to spread K > slices of user data amongst my nodes, then I can recover the user > data. > To see the security benefits of this approach it is best to think of AES as a special case of this system where K = N = 2. Where one "slice" is the encrypted data and the other slice is the key. To compromises this system an attacker must compromise data from two locations, the location where encrypted data is kept and the location where the key is kept. Disperal+AONT can improve upon this, as K can be greater than 2, forcing an attacker to have to compromise a large number of locations. Additionally, N can be greater than K, therefore yielding a higher degree of reliability, where in the simple case of AES, one would have to make copies of the key and data to increase the reliability (but this decreases confidentiality). The system you envisage is one in which the owner of the system is different from the user of the system. You are correct that this system would not be secure if one stored >=K number of slices on untrusted machines, but the same is true for AES: one couldn't store their key and encrypted data on machines they didn't trust and expect their data to be safe. Of all the ways to store secret keys reliabily and confidentially, secret sharing schemes are the best known technique. Dispersal+AONT allows one to store their data itself in the same way that people have been storing their keys when using a secret sharing scheme. Therefore it can be somewhat redundant to use a traditional key based system along with dispersal+aont, because the question them becomes, "How do you secure that secret key?" If the technique for securing the key is the same as the data, there is little practical benefit. Of course this requires that the machines storing slices must be as secure and trusted as locations one would store shares of their key. Nothing prevents one from using both an existing encryption system and then dispersal+aont, this will yield greater confidentiality but reduced reliability. > Why would I take a nice, robust, well-understood algorithm like > AES-256 and hobble it with my in-house dispersal algorithm? Because > key management is hard? It _is_ hard, definitely, and I don't quite > understand how TahoeLAFS approaches the problem (I guess it has to do > with these 'caps' you guys keep talking about), but we shouldn't > ignore the problem just because it's hard. > > It's not just a question of technical or logistical difficulty of a key management system. The bigger concern is _how_ exactly your key management system actually secures your keys. Often it is stored in a single physical medium, and the loss of that one device storing the key means one loses all data that it was encrypted with. Or, the theft or compromise of that one system means the attacker has access to all of your data. As Zooko noted before, AONT utilizes AES in an unmodified way, the only difference is how the key is protected. Dispersal+AONT doesn't ignore the problem of key management, rather it solves the key management problem by eliminating it (or more accurately collapsing it into the same problem of how the data is managed). > As for Reason #1, that computers get faster and faster: pick a key > size sufficiently large for you to retire well before your customers > come calling with pitchforks and torches. > In the response #1 post, the problem is not only exponentially increasing computing power, but advances in mathematical theory and quantum computers which threaten the asymmetric algorithms which so many key management systems are dependent upon. Such advances leading to breaks in RSA or ECC could come at any time, because it's never been proven that those problems have no efficient solutions. Use of asymmetric algorithms in key management systems is very common because all the secret keys used by one person can be protected by a single asymmetric key. Asymmetric algorithms are completely unnecessary when Dispersal+AONT are used, and therefore this approach offers greater time-resistance. > As for Reason #3, that disclosure laws are a PITA: Any storage system > based on distributing erasure-encoded slices is going to enjoy those > benefits, right? I don't see how All-or-Nothing is a big win over a > key-management infrastructure. > Without the AONT applied prior to dispersal, some slices, we call "data slices" would contain recognizable information from the original input data. This could be avoided by computing all "code slices" which are slices produced by the erasure codes, but this is computationally expensive, and unecessary when AONT is applied. P.S. Many of the questions you and Zooko raised were brought up by others in a previous discussion thread. You might find then interesting: http://groups.google.com/group/cloud-computing/browse_thread/thread/bd763d933d40394e/451f43e9eafee6da However replies to that thread seemed to have stop appearing, so if you have comments regarding any of them you should post them to this mailing list. Thanks, Jason > -Russ > > On Fri, Jul 24, 2009 at 6:33 AM, Zooko Wilcox-O'Hearn wrote: > >> [cross-posted to tahoe-dev at allmydata.org and cryptography at metzdowd.com] >> >> Disclosure: Cleversafe is to some degree a competitor of my Tahoe- >> LAFS project. On the other hand, I tend to feel positive towards >> them because they open-source much of their work. Our "Related >> Projects" page has included a link to cleversafe for years now, I >> briefly collaborated with some of them on a paper about erasure >> coding last year, and I even spoke briefly with them about the idea >> of becoming an employee of their company this year. I am tempted to >> ignore this idea that they are pushing about encryption being >> overrated, because they are wrong and it is embarassing. But I've >> decided not to ignore it, because people who publicly spread this >> kind of misinformation need to be publicly contradicted, lest they >> confuse others. >> >> Cleversafe has posted a series of blog entries entitled "3 Reasons >> Why Encryption is Overrated". >> >> http://dev.cleversafe.org/weblog/?p=63 # 3 Reasons Why Encryption is >> Overrated >> http://dev.cleversafe.org/weblog/?p=95 # Response Part 1: Future >> Processing Power >> http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: >> Complexities of Key Management >> http://dev.cleversafe.org/weblog/?p=178 # Response Part 3: Disclosure >> Laws >> >> It begins like this: >> >> """ >> When it comes to storage and security, discussions traditionally >> center on encryption. The reason encryption ? or the use of a >> complex algorithm to encode information ? is accepted as a best >> practice rests on the premise that while it?s possible to crack >> encrypted information, most malicious hackers don?t have access to >> the amount of computer processing power they would need to decrypt >> information. >> >> But not so fast. Let?s take a look at three reasons why encryption >> is overrated. >> """ >> >> Ugh. >> >> The first claim -- the today's encryption is vulnerable to tomorrow's >> processing power -- is a common goof, which is easy to make by >> conflating historical failures of cryptosystems due to having too >> small of a crypto value with failures due to weak algorithms. >> Examples of the former are DES, which failed because its 56-bit key >> was small enough to fall to brute force, and the bizarre "40-bit >> security" policies of the U.S. Federal Government in the 90's. An >> example of the latter is SHA1, whose hash output size is *not* small >> enough to brute-force, but which is insecure because, as it turns >> out, the SHA1 algorithm allows the generation of colliding inputs >> much quicker than a brute force search would. >> >> Oh boy, I see that in the discussion following the article "Future >> Processing Power", the author writes: >> >> """ >> I don?t think symmetric ciphers such as AES-256 are under any threat >> of being at risk to brute force attacks any time this century. >> """ >> >> What? Then why is he spreading this Fear, Uncertainty, and Doubt? >> Oh and then it gets *really* interesting: it turns out that >> cleversafe uses AES-256 in an All-or-Nothing Transform as part of >> their "Information Dispersal" algorithm. Okay, I would like to >> understand better the cryptographic effects of that (and in >> particular, whether this means that the cleversafe architecture is >> just as susceptible to AES-256 failing as an encryption scheme such >> as is used in the Tahoe-LAFS architecture). >> >> But, it is time for me to stop reading about cryptography and get >> ready to go to work. :-) >> >> Regards >> >> Zooko >> --- >> Tahoe, the Least-Authority Filesystem -- http://allmydata.org >> store your data: $10/month -- http://allmydata.com/?tracking=zsig >> I am available for work -- http://zooko.com/r?sum?.html >> _______________________________________________ >> tahoe-dev mailing list >> tahoe-dev at allmydata.org >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev >> >> > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20090725/802c55f3/attachment.htm From jresch at cleversafe.com Sat Jul 25 14:01:43 2009 From: jresch at cleversafe.com (Jason Resch) Date: Sat, 25 Jul 2009 16:01:43 -0500 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption isOverrated In-Reply-To: <4158677e0907241714k5c70313bra58b5182d605046d@mail.gmail.com> Message-ID: <404AAC62CDC41F418D5D484FE86D84C001448C61@csi-01-ex.atlarge.net> Russ Weeks wrote: > Yikes, that All-Or-Nothing transform, that's an interesting algorithm. > It certainly is. When first described by Ron Rivest, it was mostly a curiosity with little practical benefit, and therefore had been mostly forgotten. However, when it is combined with an information dispersal algorithm, the combined system has many of the properties of the Shamir Secret Sharing Scheme, but maintaining the perfect storage efficiency of the IDA. > Seems to me that it reduces the security of AES-256 to the security of > the dispersal algorithm. If I control a subset of malicious nodes > within the distributed storage system, and I can convince the sender > (via DDOS or some network coordinate trickery, perhaps) to spread K > slices of user data amongst my nodes, then I can recover the user > data. To see the security benefits of this approach it is best to think of AES as a special case of this system where K = N = 2. Where one "slice" is the encrypted data and the other slice is the key. To compromises this system an attacker must compromise data from two locations, the location where encrypted data is kept and the location where the key is kept. Disperal+AONT can improve upon this, as K can be greater than 2, forcing an attacker to have to compromise a large number of locations. Additionally, N can be greater than K, therefore yielding a higher degree of reliability, where in the simple case of AES, one would have to make copies of the key and data to increase the reliability (but this decreases confidentiality). The system you envisage is one in which the owner of the system is different from the user of the system. You are correct that this system would not be secure if one stored >=K number of slices on untrusted machines, but the same is true for AES: one couldn't store their key and encrypted data on machines they didn't trust and expect their data to be safe. Of all the ways to store secret keys reliabily and confidentially, secret sharing schemes are the best known technique. Dispersal+AONT allows one to store their data itself in the same way that people have been storing their keys when using a secret sharing scheme. Therefore it can be somewhat redundant to use a traditional key based system along with dispersal+aont, because the question them becomes, "How do you secure that secret key?" If the technique for securing the key is the same as the data, there is little practical benefit. Of course this requires that the machines storing slices must be as secure and trusted as locations one would store shares of their key. Nothing prevents one from using both an existing encryption system and then dispersal+aont, this will yield greater confidentiality but reduced reliability. > > Why would I take a nice, robust, well-understood algorithm like > AES-256 and hobble it with my in-house dispersal algorithm? Because > key management is hard? It _is_ hard, definitely, and I don't quite > understand how TahoeLAFS approaches the problem (I guess it has to do > with these 'caps' you guys keep talking about), but we shouldn't > ignore the problem just because it's hard. > It's not just a question of technical or logistical difficulty of a key management system. The bigger concern is _how_ exactly your key management system actually secures your keys. Often it is stored in a single physical medium, and the loss of that one device storing the key means one loses all data that it was encrypted with. Or, the theft or compromise of that one system means the attacker has access to all of your data. As Zooko noted before, AONT utilizes AES in an unmodified way, the only difference is how the key is protected. Dispersal+AONT doesn't ignore the problem of key management, rather it solves the key management problem by eliminating it (or more accurately collapsing it into the same problem of how the data is managed). > As for Reason #1, that computers get faster and faster: pick a key > size sufficiently large for you to retire well before your customers > come calling with pitchforks and torches. In the response #1 post, the problem is not only exponentially increasing computing power, but advances in mathematical theory and quantum computers which threaten the asymmetric algorithms which so many key management systems are dependent upon. Such advances leading to breaks in RSA or ECC could come at any time, because it's never been proven that those problems have no efficient solutions. Use of asymmetric algorithms in key management systems is very common because all the secret keys used by one person can be protected by a single asymmetric key. Asymmetric algorithms are completely unnecessary when Dispersal+AONT are used, and therefore this approach offers greater time-resistance. > As for Reason #3, that disclosure laws are a PITA: Any storage system > based on distributing erasure-encoded slices is going to enjoy those > benefits, right? I don't see how All-or-Nothing is a big win over a > key-management infrastructure. Without the AONT applied prior to dispersal, some slices, we call "data slices" would contain recognizable information from the original input data. This could be avoided by computing all "code slices" which are slices produced by the erasure codes, but this is computationally expensive, and unnecessary when AONT is applied. P.S. Many of the questions you and Zooko raised were brought up by others in a previous discussion thread. You might find then interesting: http://groups.google.com/group/cloud-computing/browse_thread/thread/bd763d933d40394e/451f43e9eafee6da However replies to that thread seemed to have stop appearing, so if you have comments regarding any of them you should post them to this mailing list. Thanks, Jason > > -Russ > > On Fri, Jul 24, 2009 at 6:33 AM, Zooko Wilcox-O'Hearn wrote: >> [cross-posted to tahoe-dev at allmydata.org and cryptography at metzdowd.com] >> >> Disclosure: Cleversafe is to some degree a competitor of my Tahoe- >> LAFS project. On the other hand, I tend to feel positive towards >> them because they open-source much of their work. Our "Related >> Projects" page has included a link to cleversafe for years now, I >> briefly collaborated with some of them on a paper about erasure >> coding last year, and I even spoke briefly with them about the idea >> of becoming an employee of their company this year. I am tempted to >> ignore this idea that they are pushing about encryption being >> overrated, because they are wrong and it is embarassing. But I've >> decided not to ignore it, because people who publicly spread this >> kind of misinformation need to be publicly contradicted, lest they >> confuse others. >> >> Cleversafe has posted a series of blog entries entitled "3 Reasons >> Why Encryption is Overrated". >> >> http://dev.cleversafe.org/weblog/?p=63 # 3 Reasons Why Encryption is >> Overrated >> http://dev.cleversafe.org/weblog/?p=95 # Response Part 1: Future >> Processing Power >> http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: >> Complexities of Key Management >> http://dev.cleversafe.org/weblog/?p=178 # Response Part 3: Disclosure >> Laws >> >> It begins like this: >> >> """ >> When it comes to storage and security, discussions traditionally >> center on encryption. The reason encryption - or the use of a >> complex algorithm to encode information - is accepted as a best >> practice rests on the premise that while it's possible to crack >> encrypted information, most malicious hackers don't have access to >> the amount of computer processing power they would need to decrypt >> information. >> >> But not so fast. Let's take a look at three reasons why encryption >> is overrated. >> """ >> >> Ugh. >> >> The first claim -- the today's encryption is vulnerable to tomorrow's >> processing power -- is a common goof, which is easy to make by >> conflating historical failures of cryptosystems due to having too >> small of a crypto value with failures due to weak algorithms. >> Examples of the former are DES, which failed because its 56-bit key >> was small enough to fall to brute force, and the bizarre "40-bit >> security" policies of the U.S. Federal Government in the 90's. An >> example of the latter is SHA1, whose hash output size is *not* small >> enough to brute-force, but which is insecure because, as it turns >> out, the SHA1 algorithm allows the generation of colliding inputs >> much quicker than a brute force search would. >> >> Oh boy, I see that in the discussion following the article "Future >> Processing Power", the author writes: >> >> """ >> I don't think symmetric ciphers such as AES-256 are under any threat >> of being at risk to brute force attacks any time this century. >> """ >> >> What? Then why is he spreading this Fear, Uncertainty, and Doubt? >> Oh and then it gets *really* interesting: it turns out that >> cleversafe uses AES-256 in an All-or-Nothing Transform as part of >> their "Information Dispersal" algorithm. Okay, I would like to >> understand better the cryptographic effects of that (and in >> particular, whether this means that the cleversafe architecture is >> just as susceptible to AES-256 failing as an encryption scheme such >> as is used in the Tahoe-LAFS architecture). >> >> But, it is time for me to stop reading about cryptography and get >> ready to go to work. :-) >> >> Regards >> >> Zooko >> --- >> Tahoe, the Least-Authority Filesystem -- http://allmydata.org >> store your data: $10/month -- http://allmydata.com/?tracking=zsig >> I am available for work -- http://zooko.com/r?sum?.html >> _______________________________________________ >> tahoe-dev mailing list >> tahoe-dev at allmydata.org >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev >> > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev From shawn-tahoe at willden.org Sat Jul 25 14:53:11 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sat, 25 Jul 2009 15:53:11 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> Message-ID: <200907251553.11890.shawn-tahoe@willden.org> On Saturday 25 July 2009 01:02:00 pm Zooko Wilcox-O'Hearn wrote: > Folks: > > Anybody out there who loves Windows? Okay, how about this: anybody > out there who has a working Windows system and is willing to use it > for an hour or so to make sure Tahoe-LAFS installs cleanly? I'll fire up a clean XP VM and give it a try. This has been on my to-do list for a while anyway. Oops. My kids just informed me we're going to see Harry Potter. I guess I'll do it this evening, after the movie. Shawn. From trac at allmydata.org Sat Jul 25 14:35:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 25 Jul 2009 21:35:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location In-Reply-To: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> References: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> Message-ID: <046.3b8683c054e8a2b8580841e6b7d00f3d@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by robk): I pretty much just asked this question of zooko (he was online at the time) in part it was because I hadn't found the docs/configuration.txt file - because I naturally go looking for info on the web, hence trac, hence had to go digging to find config instructions. (but that's another matter) tub.location="some manually crafted stuff" handles the case I'm facing. but "auto+some_manual_cruft" would be nice. sign me up for a minor-priority vote in favour ;-) -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 25 15:41:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 25 Jul 2009 22:41:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.9a82f21b8ba63c20afd582798752f87d@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): So I skipped the {{{--comment}}} option, which I couldn't figure out, but it worked without the {{{--comment}}} option. Now we have automatic upload of pycryptopp .deb's and automatic rebuild of the apt repo index so that "apt-get" can find and download the .deb's! So far the jaunty-amd64 buildslave (my workstation) is the only one doing this: http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- jaunty-yukyuk The next step is for other buildslave operators to agree to install the furl files into their buildslave base directory, and then I'll set the buildmaster to invoke the "upload-deb" and "update-apt-repo" on those buildslaves. The furl files are confidential (knowledge of them gives the ability to upload the debs and regenerate the apt repo index, respectively), so write to zooko at zooko.com asking for yours and I'll e-mail you yours back. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sat Jul 25 16:21:16 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sat, 25 Jul 2009 23:21:16 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.c9d82e618a986eba8b03eb6c98856774@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Hm, {{{sudo apt-get install python-pycryptopp}}} doesn't work on my system even after the buildbot appears to have uploaded the pycryptopp .deb and updated the apt repo for jaunty on amd64 in this step: http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- jaunty-yukyuk/builds/23/steps/update%20apt%20repo/logs/stdio It says: """ MAIN yukyuk:~$ sudo apt-get install python-pycryptopp Reading package lists... 0% Reading package lists... 3% Reading package lists... Done Building dependency tree Reading state information... Done Package python-pycryptopp is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source E: Package python-pycryptopp has no installation candidate """ My {{{/etc/apt/sources.list}}} includes: {{{ deb http://allmydata.org/debian/ jaunty main tahoe }}} Is there a manual step that needs to be done on the apt repository server to initialize the jaunty repo for the first time? (I think before this automated step started getting run there were no "jaunty"-flavored debs on this server.) By the way, once we have the new automated apt-repo for pycryptopp sorted out then we need to update this wiki page, adding jaunty and setting the flags correctly for the imminent v1.5.0 release: http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages -- Ticket URL: tahoe-lafs secure decentralized file storage grid From michael at michael.ie Sat Jul 25 17:35:53 2009 From: michael at michael.ie (Michael Walsh) Date: Sun, 26 Jul 2009 01:35:53 +0100 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> Message-ID: What's the config of the build machines? I see on zandrs windows-native-tahoe buildslave, as I'm doing he uses Python.orgs build (instead of cygwins python package). What compiler - VS 2003? Or setup.py -c mingw32? - Is it your own compile of PyWin32 or using a package? I haven't compiled anything using mingw on windows since... mnet actually and I'm unsure what stuff from it I actually need installed. My Config: VS 2008 is installed; cygwin and setup.py executed from this env; python.org 2.5.4 EXPORTed path to cygwin; PyWin32 package for Python 2.5; allmydata-tahoe-1.4.1-r4026.tar.gz error: Setup script exited with error: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. Thanks, Michael On Sat, Jul 25, 2009 at 8:02 PM, Zooko Wilcox-O'Hearn wrote: > Folks: > > Anybody out there who loves Windows? ?Okay, how about this: anybody > out there who has a working Windows system and is willing to use it > for an hour or so to make sure Tahoe-LAFS installs cleanly? > > Thanks! > > Regards, > > Zooko > > On Saturday,2009-07-25, at 12:15 , tahoe-lafs wrote: > >> #773: run through installation on Windows >> ----------------------- >> +---------------------------------------------------- >> ?Reporter: ?zooko ? ? ?| ? ? ? ? ? Owner: ?somebody >> ? ? ?Type: ?task ? ? ? | ? ? ? ? ?Status: ?new >> ?Priority: ?major ? ? ?| ? ? ? Milestone: ?1.5.0 >> Component: ?packaging ?| ? ? ? ? Version: ?1.4.1 >> ?Keywords: ?win32 ? ? ?| ? Launchpad_bug: >> ----------------------- >> +---------------------------------------------------- >> ?Before the official Tahoe-LAFS v1.5.0 release, I would like for >> someone to >> ?follow the official install doc >> ?(http://allmydata.org/source/tahoe/trunk/docs/install.html ) on a >> Windows >> ?system. ?If you've never installed Tahoe-LAFS on your Windows system >> ?before, all the better! ?We have automated tests of build and certain >> ?install steps on Windows (http://allmydata.org/buildbot/builders/ >> windows , >> ?http://allmydata.org/buildbot/builders/BlackDew%20windows-mingw32- >> py26 ) >> ?and cygwin (http://allmydata.org/buildbot/builders/cygwin ), but I >> would >> ?feel better if an actual human tried it. >> >> ?Hm, say Black Dew, why is your Windows buildslave off-line? >> >> -- >> Ticket URL: >> tahoe-lafs >> secure decentralized file storage grid >> _______________________________________________ >> tahoe-dev mailing list >> tahoe-dev at allmydata.org >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > From shawn-tahoe at willden.org Sat Jul 25 22:22:54 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sat, 25 Jul 2009 23:22:54 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907251553.11890.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> <200907251553.11890.shawn-tahoe@willden.org> Message-ID: <200907252322.55298.shawn-tahoe@willden.org> On Saturday 25 July 2009 03:53:11 pm Shawn Willden wrote: > On Saturday 25 July 2009 01:02:00 pm Zooko Wilcox-O'Hearn wrote: > > Folks: > > > > Anybody out there who loves Windows? Okay, how about this: anybody > > out there who has a working Windows system and is willing to use it > > for an hour or so to make sure Tahoe-LAFS installs cleanly? > > I'll fire up a clean XP VM and give it a try. This has been on my to-do > list for a while anyway. Okay, here's what I did: 0. Installed a clean Windows XP system in a VMWare virtual machine. 1. Went to http://allmydata.org/source/tahoe/trunk/docs/install.html. BTW, that page looks horrible on IE6. Huge type. 2. It said I need Python, so I followed the link, found the latest 2.5 on the list (2.5.4), clicked through and downloaded the MSI. Took all the defaults. 3. Back to the instructions. The link is to a 1.4 zip file. Couldn't find a 1.5 zip file. Got a clean checkout with darcs, removed the _darcs directory, zipped it up and copied it over to Windows. 4. Back to the instructions. Unpacked the zip into c:\tahoe 5. Opened a cmd.exe window. Went to c:\tahoe and ran "python setup.py build". Got "'python' is not recognized as an internal or external command, operable program or batch file." Changed it to "c:\Python25\python setup.py build". It ran for a while, then failed with "ImportError: No module named win32api". 6. Back to the instructions. Noticed the bit at the bottom with a link to the InstallDetails wiki page. Noticed it said I needed pywin32. 7. Downloaded and installed pywin32-214.win32-py2.5.exe, then ran the build again. This time it finished with the error "Python was built with Visual Studio version 7.1 and extensions need to be build with the same version of the compiler, but it isn't installed." It was trying to build zfec-1.4.5. 8. Based on the mention of mingw in Michael Walsh's error message, downloaded the mingw installer. One interesting problem was that IE6 crashed every time I tried to download. Installed FF to work around that. Did a full install of mingw. 9. Re-ran build, with "-c mingw32", but got the same compiler version error. At this point, it looks like my only option is to (a) install cygwin, (b) get python2.5 sources, (c) get python-mingw, (d) build python with mingw and then (e) retry with the mingw-built python. How does this work on the Windows buildslaves? Are they using a mingw-build python? Shawn. From blackdew at gmail.com Sat Jul 25 23:37:14 2009 From: blackdew at gmail.com (Black Dew) Date: Sun, 26 Jul 2009 09:37:14 +0300 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907252322.55298.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> <200907251553.11890.shawn-tahoe@willden.org> <200907252322.55298.shawn-tahoe@willden.org> Message-ID: > How does this work on the Windows buildslaves? ?Are they using a mingw-build > python? My buildslave has the normal python from python.org and builds using MinGW. I can't check right now, but possibly my buildslave has the default compiler set in distutils config file, maybe the -c isn't passed on from tahoe setup.py to the dependencies. From shawn-tahoe at willden.org Sat Jul 25 23:50:24 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 00:50:24 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907252322.55298.shawn-tahoe@willden.org> Message-ID: <200907260050.24396.shawn-tahoe@willden.org> On Sunday 26 July 2009 12:37:14 am Black Dew wrote: > > How does this work on the Windows buildslaves? ?Are they using a > > mingw-build python? > > My buildslave has the normal python from python.org and builds using MinGW. > > I can't check right now, but possibly my buildslave has the default > compiler set in distutils config file, maybe the -c isn't passed on > from tahoe setup.py to the dependencies. Yeah, I found a site that clued me into the distutils.cfg. That works. I'm gradually getting further, and typing another e-mail with my progress notes. Thanks, Shawn. From shawn-tahoe at willden.org Sat Jul 25 23:59:20 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 00:59:20 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907252322.55298.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907251553.11890.shawn-tahoe@willden.org> <200907252322.55298.shawn-tahoe@willden.org> Message-ID: <200907260059.20332.shawn-tahoe@willden.org> On Saturday 25 July 2009 11:22:54 pm Shawn Willden wrote: > At this point, it looks like my only option is to (a) install cygwin, (b) > get python2.5 sources, (c) get python-mingw, (d) build python with mingw > and then (e) retry with the mingw-built python. Lots of googling turned up this page: http://boodebr.org/main/python/build-windows-extensions which gave me the trick: You have to create a distutils.cfg file that tells distutils to use mingw32. So... 10. Created a distutils.cfg file as specfied at http://boodebr.org/main/python/build-windows-extensions then tried building again. Failed trying to build pyopenssl with the error "Cannot find ssleay32.a". So, the openssl libraries apparently need to be installed. 11. Installed pyopenssl package from: http://launchpadlibrarian.net/29331282/pyOpenSSL-0.9.win32-py2.5.exe then tried building again. Dang. It still tries to download and build pyopenssl, even though it's already installed in site-packages. 12. Hacked around that problem by copying ssleay32.dll from the site-packages installation to C:\Windows\system32. Tried building again, and it succeeded! 13. Run the tests. Oddly, it rebuilt a bunch of stuff, including pycryptopp. 22 tests failed, it looks like all of them because they tried to use OpenSSL and failed to load a DLL. Copied the other DLL (libeay32.dll) to system32. Same problem. 14. Run "python setup.py install". Succeeded. Try the tests again. Same failures. Not good. 15. Try to start tahoe node. Fails with the DLL load error. :-( Shawn. From blackdew at gmail.com Sun Jul 26 00:34:16 2009 From: blackdew at gmail.com (Black Dew) Date: Sun, 26 Jul 2009 10:34:16 +0300 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <4D736E36-0EE3-4CF8-92E1-7124F712803D@zooko.com> References: <4D736E36-0EE3-4CF8-92E1-7124F712803D@zooko.com> Message-ID: > I don't know anything about PPAs for Ubuntu. ?Why are they better > than the apt repos hosted by allmydata.org? I don't know how it works in Ubuntu, but the benefit of getting tahoe (and dependencies) into Debian proper (if someone is wiling to do it) would be automatic building and testing for every platform supported by debian (an that's a lot of platforms :P) on debian's official buildd's. From trac at allmydata.org Sun Jul 26 00:07:26 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 26 Jul 2009 07:07:26 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.1fb9fa40d84af04b4a2b29d94e27b6d7@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by bdew): I was away for few days and it looks like it's (crappy) internet connection is half-dead, i'll try to fix it later today. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Sun Jul 26 05:07:55 2009 From: trac at allmydata.org (tahoe-lafs) Date: Sun, 26 Jul 2009 12:07:55 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.64baabb31cd7fe4bcfee55ff1060e09d@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): In the long term we'll probably switch to the new fork of setuptools named "Distribute" http://mail.python.org/pipermail/distutils- sig/2009-July/012665.html and contribute patches to that project. For now it wouldn't hurt for you to install {{{setuptools}}} (actually {{{zetuptoolz}}} 0.6c12dev on your system, but on the other hand we already have .deb's for {{{zfec}}} so it is okay if your system can't produce .deb's of {{{zfec}}} at the moment. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From hughejp at mac.com Sun Jul 26 06:18:07 2009 From: hughejp at mac.com (james hughes) Date: Sun, 26 Jul 2009 21:18:07 +0800 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated In-Reply-To: <2454178A-89EE-4124-A411-7C278EDDB570@mac.com> References: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> <2454178A-89EE-4124-A411-7C278EDDB570@mac.com> Message-ID: <5036710B-C150-4334-BDFB-A7382F7D8224@mac.com> This was also posted to cryptography at metzdowd.com On Jul 24, 2009, at 9:33 PM, Zooko Wilcox-O'Hearn wrote: > [cross-posted to tahoe-dev at allmydata.org and > cryptography at metzdowd.com] > > Disclosure: Cleversafe is to some degree a competitor of my Tahoe- > LAFS project. ... > I am tempted to ignore this idea that they are pushing about > encryption being overrated, because they are wrong and it is > embarassing. ....and probably patent pending regardless of there being significant amounts of prior art for their work. One reference is ?POTSHARDS: Secure Long-Term Storage Without Encryption? by Storer, Greenan, and Miller at http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf. Maybe they did include this in their application. I certainly do not know. They seem to have one patent http://tinyurl.com/njq8yo and 7 pending. http://tinyurl.com/ntpsj9 ... > But I've decided not to ignore it, because people who publicly > spread this kind of misinformation need to be publicly contradicted, > lest they confuse others. ... The trick is cute, but I argue largely irrelevant. Follows is a response to this web page that can probably be broadened to be a criticism of any system that claims security and also claims that key management of some sort is not a necessary evil. > http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: > Complexities of Key Management I agree with many of your points. I would like to make a few of my own. 1) If you are already paying the large penalty to Reed Solomon the encrypted data, the cost of your secret sharing scheme is a small additional cost to bear, agreed. Using the hash to ?prove? you have all the pieces is cute and does turn Reed Solomon into an AONT. I will argue that if you were to do a Blakley key split of a random key, and append each portion to each portion of the encrypted file you would get similar performance results. I will give you that your scheme is simpler to describe. 2) In my opinion, key management is more about process than cryptography. The whole premise of Shamir and Blakley is that each share is independently managed. In your case, they are not. All of the pieces are managed by the same people, possibly in the same data center, etc. Because of this, some could argue that the encryption has little value, not because it is bad crypto, but because the shares are not independently controlled. I agree that if someone steals one piece, they have nothing. They will argue, that if someone can steal one piece, it is feasible to steal the rest. 3) Unless broken drives are degaussed, if they are discarded, they can be considered lost. Because of this, there will be drive loss all the time (3% per year according to several papers). As long as all N pieces are not on the same media, you can actually lose the media, no problem. This can be expanded to a loss of a server, raid controllers, NAS box, etc. without problem as long as there is only N-1 pieces, no problem. What if you lose N chunks (drives, systems, etc.) over time, are you sure you have not lost control of someone?s data? Have you tracked what was on each and every lost drive? What is your process when you do a technology refresh and retire a complete configuration? If media destruction is still necessary, will resulting operational process really any easier or safer than if the data were just split? 4) What do you do if you believe your system has been compromised by a hacker? Could they have read N pieces? Could they have erased the logs? 5) I also suggest that there is other prior art out there for this kind of storage system. I suggest the paper ?POTSHARDS: Secure Long- Term Storage Without Encryption? by Storer, Greenan, and Miller at http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf because it covers the same space, and has a good set of references to other systems. My final comment is that you raised the bar, yes. I will argue that you did not make the case that key management is not needed. Secrets are still needed to get past the residual problems described in these comments. Keys are small secrets that can be secured at lower cost that securing the entire bulk of the data. Your system requires the bulk of the data to to be protected, and thus in the long run, does not offer operational efficiency that simple bulk encryption with a traditional key management provides. Jim --------------------------------------------------------------------- The Cryptography Mailing List Unsubscribe by sending "unsubscribe cryptography" to majordomo at metzdowd.com From zooko at zooko.com Sun Jul 26 06:53:40 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 26 Jul 2009 07:53:40 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907252322.55298.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> <200907251553.11890.shawn-tahoe@willden.org> <200907252322.55298.shawn-tahoe@willden.org> Message-ID: <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> On Saturday,2009-07-25, at 23:22 , Shawn Willden wrote: > 1. Went to http://allmydata.org/source/tahoe/trunk/docs/ > install.html. BTW, that page looks horrible on IE6. Huge type. Uh, well, the page doesn't have any style sheet or other style indicators. It just says "

About Tahoe

...". > 2. It said I need Python, so I followed the link, found the latest > 2.5 on the list (2.5.4), clicked through and downloaded the MSI. > Took all the defaults. Good. > 3. Back to the instructions. The link is to a 1.4 zip file. > Couldn't find a 1.5 zip file. Got a clean checkout with darcs, > removed the _darcs directory, zipped it up and copied it over to > Windows. Oh, if you get a chance to do something like this again soon, please get a recent snapshot from http://allmydata.org/source/tahoe/ snapshots/ . > 5. Opened a cmd.exe window. Went to c:\tahoe and ran "python > setup.py build". Got "'python' is not recognized as an internal or > external command, operable program or batch file." Changed it to "c: > \Python25\python setup.py build". Hm, so installing Python didn't add that directory to your path. > 6. Back to the instructions. Noticed the bit at the bottom with a > link to the InstallDetails wiki page. Noticed it said I needed > pywin32. Okay I've updated install.html to mention pywin32 earlier. > 7. Downloaded and installed pywin32-214.win32-py2.5.exe, then ran > the build again. This time it finished with the error "Python was > built with Visual Studio version 7.1 and extensions need to be > build with the same version of the compiler, but it isn't > installed." It was trying to build zfec-1.4.5. Okay, so now there are two layers of how to improve things here. On the lower layer, I want to make it easier and better-documented for people to build the packages such as zfec and pycryptopp which require a C/C++ compiler. On the upper layer, we are supposed to be providing binary packages (eggs) of those dependencies, at least on Windows, so that people installing Tahoe-LAFS, which is pure Python, do not require a C/C++ compiler. So why isn't *that* layer currently working? The _auto_deps.py file [1] says that Tahoe-LAFS requires zfec >= 1.1.0. The setup.cfg file [2] says that the build process should look at a certain directory on http://allmydata.org named "tahoe-dep-eggs" [3]. That directory contains a file named "zfec-1.4.4-py2.5-win32.egg". When you did the build step you described above -- your step 7 -- it should not have attempted to compile zfec from source! Could you inspect the stdout from the build process to see if it considered directory [3] and if it considered downloading "zfec-1.4.4-py2.5-win32.egg"? Oh! It is because you're building v1.4.1 and the setup.cfg of that version [4] does not say to look at the "tahoe-dep-eggs" dir [3]. Good, so that mystery is explained. If you try again, then the "upper layer" of pre-built binaries of the dependencies should work with a recent pre-1.5 snapshot. Regards, Zooko [1] http://allmydata.org/trac/tahoe/browser/_auto_deps.py [2] http://allmydata.org/trac/tahoe/browser/setup.cfg [3] http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ [4] http://allmydata.org/trac/tahoe/browser/setup.cfg? rev=20090414025636-92b7f-e77128b954ced21ebd04d82c615ee0f8f9cffd49 From david-sarah at jacaranda.org Sun Jul 26 07:01:50 2009 From: david-sarah at jacaranda.org (David-Sarah Hopwood) Date: Sun, 26 Jul 2009 15:01:50 +0100 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <9192C834-6A0A-45AA-B8FE-EA1B58814BE6@zooko.com> <200907251553.11890.shawn-tahoe@willden.org> <200907252322.55298.shawn-tahoe@willden.org> <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> Message-ID: <4A6C61CE.5020403@jacaranda.org> Zooko Wilcox-O'Hearn wrote: > On Saturday,2009-07-25, at 23:22 , Shawn Willden wrote: > >> 1. Went to http://allmydata.org/source/tahoe/trunk/docs/ >> install.html. BTW, that page looks horrible on IE6. Huge type. > > Uh, well, the page doesn't have any style sheet or other style > indicators. It just says "

About Tahoe

...". The problem is the double

here:

How To Install Tahoe

-- David-Sarah Hopwood ? http://davidsarah.livejournal.com From shawn-tahoe at willden.org Sun Jul 26 08:14:48 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 09:14:48 -0600 Subject: [tahoe-dev] download statistics Re: [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: References: <4D736E36-0EE3-4CF8-92E1-7124F712803D@zooko.com> Message-ID: <200907260914.48380.shawn-tahoe@willden.org> On Sunday 26 July 2009 01:34:16 am Black Dew wrote: > > I don't know anything about PPAs for Ubuntu. ?Why are they better > > than the apt repos hosted by allmydata.org? I think the only thing that might be "better" about PPAs is that more Ubuntu users are familiar with the PPAs. The process of using one is identical, though (add to sources.list, install). > I don't know how it works in Ubuntu, but the benefit of getting tahoe > (and dependencies) into Debian proper (if someone is wiling to do it) > would be automatic building and testing for every platform supported > by debian (an that's a lot of platforms :P) on debian's official > buildd's. And getting it into Debian makes it much easier to get it into Ubuntu proper. Shawn. From shawn-tahoe at willden.org Sun Jul 26 08:28:30 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 09:28:30 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907252322.55298.shawn-tahoe@willden.org> <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> Message-ID: <200907260928.30733.shawn-tahoe@willden.org> On Sunday 26 July 2009 07:53:40 am Zooko Wilcox-O'Hearn wrote: > > 3. Back to the instructions. The link is to a 1.4 zip file. > > Couldn't find a 1.5 zip file. Got a clean checkout with darcs, > > removed the _darcs directory, zipped it up and copied it over to > > Windows. > > Oh, if you get a chance to do something like this again soon, please > get a recent snapshot from http://allmydata.org/source/tahoe/ > snapshots/ . Okay, Trying that now... I did look there before, but I just looked at version numbers, not dates, and I was looking for 1.5, not 1.4.1-r4026. Anway, I removed everything I had installed (all python packages, mingw, python itself), then installed python 2.5.4 and pywin32-214. I wish I'd had the foresight to snapshot the VM after installing SP2, but I didn't. After I get through this, I'll start over again, with a fresh XP install. Then I unpacked the latest snapshot and ran the build. Got a warning on the zope.interface install saying that an optional extension could not be compiled. Doh! Downloaded a tarball for pycryptopp, not an egg. It said: ----------------- Processing dependencies for allmydata-tahoe==1.4.1-r4026 Searching for pycryptopp>=0.5.15 Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/ Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ Best match: pycryptopp 0.5.15 Downloading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/pycryptopp-0 .5.15.tar.gz Processing pycryptopp-0.5.15.tar.gz ------------------ So it appears it looked for an egg, but got the tarball. Ah ha! I looked on that page and there is no egg for python 2.5 win32. There is for python 2.6. I'm off to try that. I'd rather use 2.6 anyway; I only used 2.5 because that's what the installation directions said. > > 5. Opened a cmd.exe window. Went to c:\tahoe and ran "python > > setup.py build". Got "'python' is not recognized as an internal or > > external command, operable program or batch file." Changed it to "c: > > \Python25\python setup.py build". > > Hm, so installing Python didn't add that directory to your path. No. I ended up adding it later. Had to add mingw as well. Adding paths on Windows is a PITA. And I think maybe I needed to add something else, which caused my trouble building openssl. A library path, maybe. Shawn. From shawn-tahoe at willden.org Sun Jul 26 08:46:52 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 09:46:52 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907260928.30733.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <296E707F-39A0-4F56-8D1A-FB4288E2ABF1@zooko.com> <200907260928.30733.shawn-tahoe@willden.org> Message-ID: <200907260946.53163.shawn-tahoe@willden.org> On Sunday 26 July 2009 09:28:30 am Shawn Willden wrote: > Ah ha! I looked on that page and there is no egg for python 2.5 win32. > There is for python 2.6. I'm off to try that. I'd rather use 2.6 anyway; > I only used 2.5 because that's what the installation directions said. Didn't work with 2.6 either. I did the following: 1. Removed all 2.5 stuff. 2. Installed 2.6.2 from the MSI 3. Did NOT instally pywin32. I noticed there was a link on the deps page, so I wondered if maybe setup would grab it. 4. Unpacked the latest snapshot 5. Ran \python26\python setup.py build. It got through a bunch of stuff (didn't install pywin32) but ultimately failed trying to build pycryptopp: ------ Installed c:\tahoe\src Processing dependencies for allmydata-tahoe==1.4.1-r4026 Searching for pycryptopp>=0.5.15 Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/ Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ Best match: pycryptopp 0.5.15 Downloading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/pycryptopp-0 .5.15.tar.gz Processing pycryptopp-0.5.15.tar.gz Running pycryptopp-0.5.15\setup.py -q bdist_egg --dist-dir c: \docume~1\admini~1\ locals~1\temp\easy_install-edumad\pycryptopp-0.5.15\egg-dist-tmp-rr4q2q setup.py darcsver: Failure from attempt to find version tags with 'darcs changes ', and pycryptopp\_version.py already exists, so leaving it alone. error: Setup script exited with error: Unable to find vcvarsall.bat ------- I think it didn't try to use the egg because the egg is version 0.5.12, while the tarball is 0.5.15. Shawn. From shawn-tahoe at willden.org Sun Jul 26 09:04:50 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 10:04:50 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907260946.53163.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907260928.30733.shawn-tahoe@willden.org> <200907260946.53163.shawn-tahoe@willden.org> Message-ID: <200907261004.50584.shawn-tahoe@willden.org> On Sunday 26 July 2009 09:46:52 am Shawn Willden wrote: > error: Setup script exited with error: Unable to find vcvarsall.bat Installing mingw32 and a distutils.cfg file didn't fix that. Looks like an open issue: http://bugs.python.org/issue2698 I'm sure I could work around this by installing MSVC. I guess I'll do that, though I hate it. Shawn From shawn-tahoe at willden.org Sun Jul 26 09:54:30 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 10:54:30 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907261004.50584.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907260946.53163.shawn-tahoe@willden.org> <200907261004.50584.shawn-tahoe@willden.org> Message-ID: <200907261054.30722.shawn-tahoe@willden.org> On Sunday 26 July 2009 10:04:50 am Shawn Willden wrote: > I'm sure I could work around this by installing MSVC. I guess I'll do > that, though I hate it. Well... even that has problems :-) ------ Creating library build\temp.win32-2.6\Release\cryptopp\_pycryptopp.lib and object build\temp.win32-2.6\Release\cryptopp\_pycryptopp.exp _pycryptoppmodule.obj : error LNK2019: unresolved external symbol "void __cdecl init_sha256(struct _object *)" (?init_sha256@@YAXPAU_object@@@Z) referenced in function _init_pycryptopp build\lib.win32-2.6\pycryptopp\_pycryptopp.pyd : fatal error LNK1120: 1 unresolved externals error: Setup script exited with error: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1120 ------ This seems very odd. Unresolved external? Hmm. I can't figure out where this stuff is actually being built to go look at the details. Trying to see where these files are located, I ran the build again. Interestingly, this time it didn't even fiddle with pycryptopp BUT it failed to build zfec 1.4.5: ------- cl : Command line warning D9002 : ignoring unknown option '-std=c99' _fecmodule.c zfec/_fecmodule.c(120) : warning C4018: '<' : signed/unsigned mismatch zfec/_fecmodule.c(557) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(557) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(557) : error C2143: syntax error : missing ')' before 'type' zfec/_fecmodule.c(557) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(557) : error C2065: 'i' : undeclared identifier zfec/_fecmodule.c(557) : warning C4018: '<' : signed/unsigned mismatch zfec/_fecmodule.c(557) : warning C4552: '<' : operator has no effect; expected operator with side-effect zfec/_fecmodule.c(557) : error C2065: 'i' : undeclared identifier zfec/_fecmodule.c(557) : error C2059: syntax error : ')' zfec/_fecmodule.c(558) : error C2146: syntax error : missing ';' before identifier 'printf' zfec/_fecmodule.c(558) : error C2065: 'i' : undeclared identifier zfec/_fecmodule.c(569) : error C2143: syntax error : missing ';' before 'const' zfec/_fecmodule.c(570) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(571) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(579) : error C2275: 'fec_t' : illegal use of this type as an expression c:\documents and settings\administrator\local settings\temp\easy_install-zdrtxo\zfec-1.4.5\zfec\fec.h( 15) : see declaration of 'fec_t' zfec/_fecmodule.c(579) : error C2059: syntax error : 'const' zfec/_fecmodule.c(580) : error C2065: 'fec' : undeclared identifier zfec/_fecmodule.c(580) : warning C4047: 'function' : 'const fec_t *' differs in levels of indirection from 'int' zfec/_fecmodule.c(580) : warning C4024: 'fec_encode' : different types for formal and actual parameter 1 zfec/_fecmodule.c(580) : error C2065: 'blocks' : undeclared identifier zfec/_fecmodule.c(580) : warning C4047: 'function' : 'const gf *const *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(580) : warning C4024: 'fec_encode' : different types for formal and actual parameter 2 zfec/_fecmodule.c(580) : error C2065: 'outblocks' : undeclared identifier zfec/_fecmodule.c(580) : warning C4047: 'function' : 'gf *const *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(580) : warning C4024: 'fec_encode' : different types for formal and actual parameter 3 zfec/_fecmodule.c(580) : error C2065: 'block_nums' : undeclared identifier zfec/_fecmodule.c(580) : warning C4047: 'function' : 'const unsigned int *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(580) : warning C4024: 'fec_encode' : different types for formal and actual parameter 4 zfec/_fecmodule.c(589) : error C2143: syntax error : missing ';' before 'const' zfec/_fecmodule.c(590) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(591) : error C2143: syntax error : missing ';' before 'type' zfec/_fecmodule.c(593) : error C2065: 'fec' : undeclared identifier zfec/_fecmodule.c(593) : warning C4047: 'function' : 'const fec_t *' differs in levels of indirection from 'int' zfec/_fecmodule.c(593) : warning C4024: 'fec_decode' : different types for formal and actual parameter 1 zfec/_fecmodule.c(593) : error C2065: 'inpkts' : undeclared identifier zfec/_fecmodule.c(593) : warning C4047: 'function' : 'const gf *const *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(593) : warning C4024: 'fec_decode' : different types for formal and actual parameter 2 zfec/_fecmodule.c(593) : error C2065: 'outpkts' : undeclared identifier zfec/_fecmodule.c(593) : warning C4047: 'function' : 'gf *const *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(593) : warning C4024: 'fec_decode' : different types for formal and actual parameter 3 zfec/_fecmodule.c(593) : error C2065: 'indexes' : undeclared identifier zfec/_fecmodule.c(593) : warning C4047: 'function' : 'const unsigned int *const ' differs in levels of indirection from 'int' zfec/_fecmodule.c(593) : warning C4024: 'fec_decode' : different types for formal and actual parameter 4 error: Setup script exited with error: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' failed with exit status 2 ------ Seems like some serious disagreement between the code and Microsoft's compiler about what C should look like. Or a very wrong verions of a zfec header file. I'm not sure what to try next. I guess I'll go back to 2.5 and the mingw compiler. Shawn. From jresch at cleversafe.com Sun Jul 26 10:02:13 2009 From: jresch at cleversafe.com (Jason Resch) Date: Sun, 26 Jul 2009 12:02:13 -0500 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption isOverrated In-Reply-To: <5036710B-C150-4334-BDFB-A7382F7D8224@mac.com> Message-ID: <404AAC62CDC41F418D5D484FE86D84C001448C77@csi-01-ex.atlarge.net> james hughes wrote: > This was also posted to cryptography at metzdowd.com > > On Jul 24, 2009, at 9:33 PM, Zooko Wilcox-O'Hearn wrote: > >> [cross-posted to tahoe-dev at allmydata.org and >> cryptography at metzdowd.com] >> >> Disclosure: Cleversafe is to some degree a competitor of my Tahoe- >> LAFS project. > ... >> I am tempted to ignore this idea that they are pushing about >> encryption being overrated, because they are wrong and it is >> embarassing. > > ....and probably patent pending regardless of there being significant > amounts of prior art for their work. One reference is "POTSHARDS: > Secure Long-Term Storage Without Encryption" by Storer, Greenan, and > Miller at http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf. Maybe > they did include this in their application. I certainly do not know. > They seem to have one patent > http://tinyurl.com/njq8yo > and 7 pending. > http://tinyurl.com/ntpsj9 Thanks for the link to POTSHARDS, it was interesting. However, POTSHARDS is doing something different, it uses the Shamir Secret Sharing Scheme. Therefore if you had a k of n system where k = 10, and n = 16 (a common configuration we use) then to store 1 GB of data using POTSHARDS will require N*(1 GB) or 16 GB of data to store. Cleversafe using dispersal and the All-or-Nothing Transform would only require 1.6 GB to store the same data. Another difference is that POTSHARDS uses conventional RAID to achieve reliability as opposed to more flexible arbitrary K of N erasure codes. POTSHARDS is what one would get if they had the same design goals as Cleversafe, but were restricted to using well known convention techniques. Cleversafe's advantage is increased reliability and efficiency compared to what was achieved by POTSHARDS. > > ... >> But I've decided not to ignore it, because people who publicly >> spread this kind of misinformation need to be publicly contradicted, >> lest they confuse others. > ... > > The trick is cute, but I argue largely irrelevant. Follows is a > response to this web page that can probably be broadened to be a > criticism of any system that claims security and also claims that key > management of some sort is not a necessary evil. That key management is not a necessary evil was the same conclusion for the author of the POTSHARDS paper you cited. > >> http://dev.cleversafe.org/weblog/?p=111 # Response Part 2: >> Complexities of Key Management > > I agree with many of your points. I would like to make a few of my own. > > 1) If you are already paying the large penalty to Reed Solomon the > encrypted data, the cost of your secret sharing scheme is a small > additional cost to bear, agreed. Using the hash to "prove" you have > all the pieces is cute and does turn Reed Solomon into an AONT. I will > argue that if you were to do a Blakley key split of a random key, and > append each portion to each portion of the encrypted file you would > get similar performance results. I will give you that your scheme is > simpler to describe. Actually what you describe was exactly the system we had considered (using SSSS to split a random key and append to each slice) before learning of the AONT. The difficulty with that approach, however, was that it did not fit well architecturally into our software, as it required special handing of the data both before and after being processed by the IDA. The approach we adopted is simpler in that only requires pre-processing. > > 2) In my opinion, key management is more about process than > cryptography. The whole premise of Shamir and Blakley is that each > share is independently managed. In your case, they are not. All of the > pieces are managed by the same people, possibly in the same data > center, etc. Because of this, some could argue that the encryption has > little value, not because it is bad crypto, but because the shares are > not independently controlled. I agree that if someone steals one > piece, they have nothing. They will argue, that if someone can steal > one piece, it is feasible to steal the rest. Right it all comes down to process. Ideally the sites where slices are stored should be geographically dispersed, not only for increased resistance to attack but for greater failure independence (higher reliability). Nothing about this protocol requires systems at each site be managed by the same individuals or even the same organization. Therefore the process can be adjusted to achieve independence from corrupt insiders. I mention this idea in the third response which deals with disclosure. > > 3) Unless broken drives are degaussed, if they are discarded, they can > be considered lost. Because of this, there will be drive loss all the > time (3% per year according to several papers). As long as all N > pieces are not on the same media, you can actually lose the media, no > problem. This can be expanded to a loss of a server, raid controllers, > NAS box, etc. without problem as long as there is only N-1 pieces, no > problem. What if you lose N chunks (drives, systems, etc.) over time, > are you sure you have not lost control of someone's data? That is an interesting question. The comfort provided by threshold schemes might make people careless when discarding drives. Again in the post on disclosure I note that drives should still be zeroed before being discarded, to prevent just such an attack. > Have you > tracked what was on each and every lost drive? What is your process > when you do a technology refresh and retire a complete configuration? > If media destruction is still necessary, will resulting operational > process really any easier or safer than if the data were just split? Refreshes should be done whenever it is determined that there has been a compromise of one or more slices. One may even adopt a continuous refresh schedule if they have the spare resources, we haven't defined our recommendations on this yet. While zeroing the data or destroying the media is still recommended, geographic dispersion improves the process in cases where media destruction was not used. These are interesting questions to consider. > > 4) What do you do if you believe your system has been compromised by a > hacker? Could they have read N pieces? Could they have erased the logs? If you believe a system has been compromised, the best course of action would be to collect any logs possible to do an investigation and then re-image the machine entirely, as there is no telling what backdoor or root-kit the attacker might have left behind. In fact one may never detect the compromise, and this makes a stronger case for using scheduled data refreshes. Any one system, or site when using geo-dispersion, should only contain slices of the same "slice index" (if we number the slices produced by the IDA 1 - N, the number is the slice index). Therefore the compromise of any single system, or site, will never yield K peer slices, maintaining the confidentiality of the data. Using a periodic refresh, therefore, would force an attacker to compromise K different systems at K different sites within T time. > > 5) I also suggest that there is other prior art out there for this > kind of storage system. I suggest the paper "POTSHARDS: Secure Long- > Term Storage Without Encryption" by Storer, Greenan, and Miller at http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf > because it covers the same space, and has a good set of references > to other systems. Thanks for this reference, we are planning to submit a paper to the usenix conference on this topic so this reference is greatly appreciated. By my comments above, do you agree that dispersal+aont is a new technique with new useful benefits? > > My final comment is that you raised the bar, yes. I will argue that > you did not make the case that key management is not needed. Secrets > are still needed to get past the residual problems described in these > comments. Keys are small secrets that can be secured at lower cost > that securing the entire bulk of the data. Your system requires the > bulk of the data to to be protected, and thus in the long run, does > not offer operational efficiency that simple bulk encryption with a > traditional key management provides. > > Jim > > Thanks Jim. What do you think about the combination of a convention encryption system with dispersal+aont? Would not the combination be perhaps the most confidential storage system created to date? I appreciate your insightful comments and look forward to your reply. Regards, Jason > --------------------------------------------------------------------- > The Cryptography Mailing List > Unsubscribe by sending "unsubscribe cryptography" to majordomo at metzdowd.com > > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev From shawn-tahoe at willden.org Sun Jul 26 11:08:55 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 12:08:55 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907252322.55298.shawn-tahoe@willden.org> Message-ID: <200907261208.55195.shawn-tahoe@willden.org> On Sunday 26 July 2009 12:37:14 am Black Dew wrote: > I can't check right now, but possibly my buildslave has the default > compiler set in distutils config file, maybe the -c isn't passed on > from tahoe setup.py to the dependencies. Do you know if there's anything else you've done to set up your buildslave? Anything do to with library paths? I can't get past this issue with the pyopenssl build process being unable to find the ssleay32 library. Shawn. From zooko at zooko.com Sun Jul 26 14:46:53 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 26 Jul 2009 15:46:53 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907261208.55195.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907252322.55298.shawn-tahoe@willden.org> <200907261208.55195.shawn-tahoe@willden.org> Message-ID: On Sunday,2009-07-26, at 12:08 , Shawn Willden wrote: > Anything do to with library paths? I can't get past this issue > with the pyopenssl build process being unable to find the ssleay32 > library. This sounds like it might be related to https://bugs.launchpad.net/pyopenssl/+bug/238658 # please provide binaries Perhaps you should get a version of pyOpenSSL from after the merge of the "working-windows-build" branch. Regards, Zooko From shawn-tahoe at willden.org Sun Jul 26 16:01:31 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 17:01:31 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907261208.55195.shawn-tahoe@willden.org> Message-ID: <200907261701.31209.shawn-tahoe@willden.org> On Sunday 26 July 2009 03:46:53 pm Zooko Wilcox-O'Hearn wrote: > Perhaps you should get a version of pyOpenSSL from after the merge of > the "working-windows-build" branch. How would I get the tahoe build process to use that version? Shawn. From zooko at zooko.com Sun Jul 26 19:17:08 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 26 Jul 2009 20:17:08 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907261701.31209.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907261208.55195.shawn-tahoe@willden.org> <200907261701.31209.shawn-tahoe@willden.org> Message-ID: On Sunday,2009-07-26, at 17:01 , Shawn Willden wrote: > On Sunday 26 July 2009 03:46:53 pm Zooko Wilcox-O'Hearn wrote: >> Perhaps you should get a version of pyOpenSSL from after the merge >> of the "working-windows-build" branch. > > How would I get the tahoe build process to use that version? The easiest way is to install that version of pyOpenSSL into your system (i.e. "cd pyOpenSSL && python ./setup.py install"). A tidier way is to build a .egg ("cd pyOpenSSL && python ./setup.py bdist_egg") and copy the resulting .egg file or .egg directory into a subdirectory of the tahoe source dir that you create and name "tahoe- deps". (The tahoe setup.cfg specifies that "tahoe-deps" is one of the locations that the build system should look to find dependencies). Regards, Zooko From zooko at zooko.com Sun Jul 26 20:24:04 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Sun, 26 Jul 2009 21:24:04 -0600 Subject: [tahoe-dev] seeking volunteer for a few hours's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907261208.55195.shawn-tahoe@willden.org> <200907261701.31209.shawn-tahoe@willden.org> Message-ID: <24F90148-D241-4576-ACEE-C047BA6F37D2@zooko.com> Thank you very much to Shawn Willden, Black Dew, and Michael Walsh for working on this task. I've now automated the upload of Python binary packages (eggs) for pycryptopp. You can see the process on the buildbot: Linux: http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu- jaunty-yukyuk/builds/26/steps/upload%20egg/logs/stdio Cygwin: http://allmydata.org/buildbot-pycryptopp/builders/windows-cygwin/ builds/76/steps/upload%20egg/logs/stdio Windows: http://allmydata.org/buildbot-pycryptopp/builders/windows-native/ builds/100/steps/upload%20egg/logs/stdio (Oops that one didn't work. http://foolscap.lothar.com/trac/ticket/ 109 ) And the resulting .egg packages in this directory: http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/?C=M;O=D This should relieve builders of Tahoe-LAFS (which doesn't, itself, require a C or C++ compiler) from needing to compile pycryptopp (which requires a C++ compiler), if they are on one of the platforms for which there are eggs in that directory. I intend to add automated uploads of pycryptopp eggs from more buildslaves as time goes by, and probably eventually also automated uploads of zfec eggs. (Currently we already have all the zfec eggs that we need, I think.) Regards, Zooko From shawn-tahoe at willden.org Sun Jul 26 21:27:31 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 22:27:31 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907261701.31209.shawn-tahoe@willden.org> Message-ID: <200907262227.31911.shawn-tahoe@willden.org> On Sunday 26 July 2009 08:17:08 pm Zooko Wilcox-O'Hearn wrote: > On Sunday,2009-07-26, at 17:01 , Shawn Willden wrote: > > On Sunday 26 July 2009 03:46:53 pm Zooko Wilcox-O'Hearn wrote: > >> Perhaps you should get a version of pyOpenSSL from after the merge > >> of the "working-windows-build" branch. > > > > How would I get the tahoe build process to use that version? > > The easiest way is to install that version of pyOpenSSL into your > system (i.e. "cd pyOpenSSL && python ./setup.py install"). This option doesn't work. Even with pyOpenSSL installed on my system, the Tahoe setup insists on trying to build it again. This is after installing it from an MSI package. I have not been able to build it from source because I can't figure out how to tell it how to find the OpenSSL header files. I have them installed in the default place (C:\OpenSSL\include), but it doesn't find them there, and the setup.py build command doesn't seem to allow any parameters to specify include paths, etc. > A tidier > way is to build a .egg ("cd pyOpenSSL && python ./setup.py > bdist_egg") and copy the resulting .egg file or .egg directory into a > subdirectory of the tahoe source dir that you create and name "tahoe- > deps". (The tahoe setup.cfg specifies that "tahoe-deps" is one of > the locations that the build system should look to find dependencies). When I can successfully build pyOpenSSL, I'll try that. BTW... "an hour's labor"... Ha! I'm up to about 12 hours so far and haven't managed a single successful Tahoe build and install with any of a large variety of permutations. Granted, I'm fairly clueless when it comes to both Windows and Python setuptools stuff, and especially both together, but this is far from simple. Shawn. From shawn-tahoe at willden.org Sun Jul 26 21:43:49 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Sun, 26 Jul 2009 22:43:49 -0600 Subject: [tahoe-dev] seeking volunteer for an hour's labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907262227.31911.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907262227.31911.shawn-tahoe@willden.org> Message-ID: <200907262243.49977.shawn-tahoe@willden.org> On Sunday 26 July 2009 10:27:31 pm Shawn Willden wrote: > I have not been able to build it from source > because I can't figure out how to tell it how to find the OpenSSL header > files. Duh. Just found out about build_ext. I've probably seen it a dozen times, but missed the "_ext". Still not building successfully, though. The linker can't find library ssl32. Shawn. From jresch at cleversafe.com Mon Jul 27 00:36:38 2009 From: jresch at cleversafe.com (Jason Resch) Date: Mon, 27 Jul 2009 02:36:38 -0500 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated Message-ID: <404AAC62CDC41F418D5D484FE86D84C001448C9A@csi-01-ex.atlarge.net> On Jul 26, 2009, at 12:11 AM, james hughes wrote: On Jul 24, 2009, at 9:33 PM, Zooko Wilcox-O'Hearn wrote: [cross-posted to tahoe-... at allmydata.org and cryptography at metzdowd.com ] Disclosure: Cleversafe is to some degree a competitor of my Tahoe- LAFS project. ... I am tempted to ignore this idea that they are pushing about encryption being overrated, because they are wrong and it is embarassing. The trick is cute, but I argue largely irrelevant. Follows is a response to this web page that can probably be broadened to be a criticism of any system that claims security and also claims that key management of some sort is not a necessary evil.... > It seems to me there's a much simpler critique. The Cleversafe approach - which is not without its nice points - solves the "key management problem" in exactly the same way that some version of Windows solved the "frequent General Protection Fault crashes" problem (by eliminating the error message). Eliminating the error message amounts to ignoring the problem while sweeping it there under the rug, which I don't think is an accurate representation of how this technique handles key management. This technique provides a genuine method for achieving a high degree of security without the need for a key management system. James Hughes, who posted earlier in this thread, referenced a paper which explores this topic in greater detail: http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf. Cleversafe's method has the same confidentiality advantages as POTSHARDS yet we achieve much greater storage efficiency than is possible using an information theoretically secure secret sharing scheme, as POTSHARDS does. While one may object to the fact that Cleversafe's technique requires that multiple secure locations for slices to be stored, this problem always existed for geo-dispersal. When Cleversafe first began, the design did not have the feature of AONT; the benefits of dispersal alone (increased reliability, availability and efficiency in storage) were strong enough a motivation to pursue this model for data storage. From bogus@does.not.exist.com Wed Jul 1 18:26:42 2009 From: bogus@does.not.exist.com () Date: Thu, 02 Jul 2009 01:26:42 -0000 Subject: No subject Message-ID: additional pre-processiing step was a very minor modification which = yielded very significant results. =20 =20 > The "key management problem" comes down to: I have encrypted data = stored somewhere (where we assume attackers can access it, but not make = use of it without the key). To make that data meaningful, I need to be = able to locate the key appropriate to that data. What's a key? It's some = private information. In Cleversafe's approach, I have data stored in = pieces all over the place. To get at it, I need to know where the pieces = of some data are. That information has to be secret, since anyone who = has access to it can do the same computation and recover the data just = as I can. =20 I think you may be missing one piece of the puzzle. The location of = where the data is stored is not secret. In fact anyone with a packet = sniffer could see the IP addresses of where the slices are being = dispersed to. The slices stored on slice servers are not available for = anyone to download, an authentication system is used to ensure only the = proper party can access the slices, much like any key management system = must authenticate the user in some manner before releasing the key. = What authentication system is used can vary from deployment to = deployment, but the advantage of authentication keys vs. data encryption = keys is that authentication keys can be lost and replaced without any = impact on the reliability of the data. Therefore one need not replicate = their authentication keys to many locations to prevent their loss. =20 > Alternatively, I can rely not on the secrecy of that information, but = on the discretion of those who hold the pieces. OK, but I could have = done that with a simpler technique: Encrypt the data conventionally, = then split the key among the trusted holders. That's a tiny, and more to = the point, *fixed* overhead beyond the size of the data, which will = always beat the cleverest Reed-Solomon or erasure coding. (It also has - = if I use an appropriate mode - such nice features as random access to = small parts of the data without the need to decrypt the whole thing = first.) =20 As noted above, the main point of dispersal is the extremely high levels = of reliability that can be achieved. The security features of AONT when = combined with dispersal is only the icing on the cake. Consider that a = 10 of 16 dispersed storage network can suffer 4 simultaneous failures = and still have the reliability of a RAID 6 system. Security encompasses = not just confidentiality but also availability, if the failure of a = single piece of hardware results in irrecoverable data loss, then such = data is not very secure. Regarding the problem of random access, our = software does have a segmentation layer for exactly this purpose, it = splits large files into smaller segments which can be accessed = individually. =20 > Granted, Cleversafe has other nice features. But other than changing = "the key management problem" to "the secret information needed to get at = the data, which won't be used as a crypto key" problem, I don't see how = they've actually *solved* anything. =20 If you consider any hypothetical conventional system for data encryption = and key management, it will almost surely suffer from one of the = following common problems, problems which are mitigated or eliminated by = dispersal+aont: =20 - Low reliability or availability of key storage system - Ease of physical compromise - Vulnerability to malicious or incompetent insiders - Reliance on asymmetric cryptography - Reliance on passwords which are either easy to forget or easy = to brute force - Complex procedures for cycling and expiring keys - Difficulty with accessing off-line key shares =20 =20 > Further: If I'm only encrypting stuff for myself, there's little = reason to use multiple keys. The key management problem becomes = interesting when there is different encrypted data with different access = rights for different groups of users. It's beyond me how Cleversafe's = approach makes this easier - or harder. =20 It actually becomes entirely about access rights and authentication, as = it should be. The reason for data encryption is to serve as a last line = of defense for attacks that can circumvent authentication mechanisms. = If it were impossible to open up your laptop and take its hard drive = out, full disk encryption wouldn't be needed. Likewise if it were = physically impossible to tap a connection, TLS would be unnecessary. = Using the AONT with Dispersal makes it such that bypassing the = authentication system (by physically stealing a machine holding slices, = or remotely compromising it) is entirely fruitless. Only by getting = access to a threshold number of devices (which we maintain is harder = than accessing a single location where one key is kept) can one get at = the data. =20 I hope this helps to clarify the reason behind why we have adopted this = approach. If you have any further questions don't hesitate to ask. =20 Jason =20 =20 P.S. =20 Zooko cross-posted his original post to several threads. You may wish to = check out what has been said at the tahoe-dev at allmydata.org mailing list = on this topic. ------_=_NextPart_001_01CA0E8C.FBA44185 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

On Jul 26, 2009, at 12:11 AM, james hughes = wrote:

 

 

=A0=A0=A0 On Jul 24, 2009, at 9:33 PM, Zooko = Wilcox-O'Hearn wrote:

 

 

=A0=A0=A0=A0=A0=A0=A0 [cross-posted to = tahoe-... at allmydata.org and cryptography at metzdowd.com ]

 

=A0=A0=A0=A0=A0=A0=A0 Disclosure: Cleversafe is to = some degree a competitor of my Tahoe- LAFS project.

 

=A0=A0=A0 ...

 

=A0=A0=A0=A0=A0=A0=A0 I am tempted to ignore this = idea that they are pushing about encryption being overrated, because they are wrong and it = is embarassing.

 

=A0=A0=A0 The trick is cute, but I argue largely = irrelevant. Follows is a response to this web page that can probably be broadened to be a = criticism of any system that claims security and also claims that key management = of some sort is not a necessary evil....

 

> It seems to me there's a much simpler critique. = The Cleversafe approach - which is not without its nice points - solves the = "key management problem" in exactly the same way that some version of = Windows solved the "frequent General Protection Fault crashes" problem = (by eliminating the error message).

 

Eliminating the error message amounts to ignoring the problem while sweeping it there under the rug, which I don=92t think is = an accurate representation of how this technique handles key management. = =A0This technique provides a genuine method for achieving a high degree of = security without the need for a key management system. =A0James Hughes, who = posted earlier in this thread, referenced a paper which explores this topic in greater = detail: http://www.ssrc.ucsc.edu/Papers/storer-usenix07.pdf.=A0 Cleversafe=92s method has the same confidentiality advantages as = POTSHARDS yet we achieve much greater storage efficiency than is possible using an = information theoretically secure secret sharing scheme, as POTSHARDS does.=A0 While = one may object to the fact that Cleversafe=92s technique requires that multiple secure = locations for slices to be stored, this problem always existed for = geo-dispersal.=A0 When Cleversafe first began, the design did not have the feature of AONT; the benefits of dispersal alone (increased reliability, availability and = efficiency in storage) were strong enough a motivation to pursue this model for = data storage. =A0From this frame of reference, you can see that adding the = AONT as an additional pre-processiing step was a very minor modification which = yielded very significant results.

 

 

> The "key management problem" comes = down to: I have encrypted data stored somewhere (where we assume attackers can = access it, but not make use of it without the key). To make that data meaningful, I = need to be able to locate the key appropriate to that data. What's a key? It's some private information. In Cleversafe's approach, I have data stored in = pieces all over the place. To get at it, I need to know where the pieces of some = data are. That information has to be secret, since anyone who has access to it can = do the same computation and recover the data just as I = can.

 

I think you may be missing one piece of the puzzle. = =A0The location of where the data is stored is not secret. =A0In fact anyone = with a packet sniffer could see the IP addresses of where the slices are being dispersed to. =A0The slices stored on slice servers are not available = for anyone to download, an authentication system is used to ensure only the proper = party can access the slices, much like any key management system must = authenticate the user in some manner before releasing the key.=A0 What authentication = system is used can vary from deployment to deployment, but the advantage of authentication keys vs. data encryption keys is that authentication keys = can be lost and replaced without any impact on the reliability of the data. = =A0Therefore one need not replicate their authentication keys to many locations to = prevent their loss.

 

> Alternatively, I can rely not on the secrecy of = that information, but on the discretion of those who hold the pieces. OK, but = I could have done that with a simpler technique: Encrypt the data = conventionally, then split the key among the trusted holders. That's a tiny, and more to = the point, *fixed* overhead beyond the size of the data, which will always = beat the cleverest Reed-Solomon or erasure coding. (It also has - if I use an appropriate mode - such nice features as random access to small parts of = the data without the need to decrypt the whole thing = first.)

 

As noted above, the main point of dispersal is the = extremely high levels of reliability that can be achieved.=A0 The security = features of AONT when combined with dispersal is only the icing on the cake.=A0 Consider = that a 10 of 16 dispersed storage network can suffer 4 simultaneous failures and = still have the reliability of a RAID 6 system. =A0Security encompasses not = just confidentiality but also availability, if the failure of a single piece = of hardware results in irrecoverable data loss, then such data is not very = secure.=A0 Regarding the problem of random access, our software does have a = segmentation layer for exactly this purpose, it splits large files into smaller = segments which can be accessed individually.

 

> Granted, Cleversafe has other nice features. But = other than changing "the key management problem" to "the secret information needed to get at the data, which won't be used as a crypto = key" problem, I don't see how they've actually *solved* = anything.

 

If you consider any hypothetical conventional system = for data encryption and key management, it will almost surely suffer from one of = the following common problems, problems which are mitigated or eliminated by = dispersal+aont:

 

-          Low = reliability or availability of key storage system

-          Ease of = physical compromise

-          Vulnerability to malicious or incompetent insiders

-          Reliance = on asymmetric cryptography

-          Reliance = on passwords which are either easy to forget or easy to brute = force

-          Complex = procedures for cycling and expiring keys

-          Difficulty with accessing off-line key shares

=A0

 

> Further: If I'm only encrypting stuff for = myself, there's little reason to use multiple keys. The key management problem becomes interesting when there is different encrypted data with different access = rights for different groups of users. It's beyond me how Cleversafe's approach = makes this easier - or harder.

 

It actually becomes entirely about access rights and authentication, as it should be. =A0The reason for data encryption is to = serve as a last line of defense for attacks that can circumvent authentication = mechanisms.=A0 If it were impossible to open up your laptop and take its hard drive = out, full disk encryption wouldn=92t be needed.=A0 Likewise if it were physically = impossible to tap a connection, TLS would be unnecessary.=A0 =A0=A0Using the AONT = with Dispersal makes it such that bypassing the authentication system (by physically = stealing a machine holding slices, or remotely compromising it) is entirely = fruitless. =A0Only by getting access to a threshold number of devices (which we maintain is = harder than accessing a single location where one key is kept) can one get at = the data.

 

I hope this helps to clarify the reason behind why we = have adopted this approach. =A0If you have any further questions don=92t = hesitate to ask.

 

Jason

 

 

P.S.

 

Zooko cross-posted his original post to several = threads. You may wish to check out what has been said at the tahoe-dev at allmydata.org mailing list on this topic.

------_=_NextPart_001_01CA0E8C.FBA44185-- From zooko at zooko.com Mon Jul 27 05:17:06 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 27 Jul 2009 06:17:06 -0600 Subject: [tahoe-dev] seeking volunteer for many hours labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907262227.31911.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907261701.31209.shawn-tahoe@willden.org> <200907262227.31911.shawn-tahoe@willden.org> Message-ID: <79F40FAA-274C-435F-B03E-4BD05051BBF5@zooko.com> On Sunday,2009-07-26, at 22:27 , Shawn Willden wrote: >> The easiest way is to install that version of pyOpenSSL into your >> system (i.e. "cd pyOpenSSL && python ./setup.py install"). > > This option doesn't work. Even with pyOpenSSL installed on my > system, the Tahoe setup insists on trying to build it again. This > is after installing it from an MSI package. Which MSI package? "pyOpenSSL-0.9.win32-py2.6.msi" from https:// bugs.launchpad.net/pyopenssl/+bug/238658/comments/74 ? That would be a bug (whether in pyOpenSSL packaging or in setuptools), Does the .msi install a file or directory named "pyOpenSSL-0.9-py2.6- win32.egg-info"? > BTW... "an hour's labor"... Ha! Ahem. Yes, well in my defense almost none of the problems we've encountered so far have to do with building Tahoe-LAFS itself so much as with building dependencies. Hence the "higher layer" of having prepackaged binaries of the dependencies for people who want to spend less than twelve hours installing. I just now got the python 2.5 win32 package of pycryptopp automatically uploading: http://allmydata.org/buildbot-pycryptopp/builders/windows-native/ builds/103/steps/upload%20egg/logs/stdio Landing here: http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ And I'm now going to get the recent pyOpenSSL binaries attached to this ticket: https://bugs.launchpad.net/pyopenssl/+bug/238658 and copy them into our "tahoe-dep-eggs" directory... There. Okay as visible in the README.html (scroll down to the bottom when viewing the "tahoe-dep-eggs" directory), we are lacking packages for some versions of Python on some operating systems. Hopefully we'll get some volunteer buildbots set up to build those. Regards, Zooko From shawn-tahoe at willden.org Mon Jul 27 06:13:54 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 27 Jul 2009 07:13:54 -0600 Subject: [tahoe-dev] seeking volunteer for many hours labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <79F40FAA-274C-435F-B03E-4BD05051BBF5@zooko.com> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907262227.31911.shawn-tahoe@willden.org> <79F40FAA-274C-435F-B03E-4BD05051BBF5@zooko.com> Message-ID: <200907270713.55072.shawn-tahoe@willden.org> On Monday 27 July 2009 06:17:06 am Zooko Wilcox-O'Hearn wrote: > Which MSI package? "pyOpenSSL-0.9.win32-py2.6.msi" from https:// > bugs.launchpad.net/pyopenssl/+bug/238658/comments/74 ? That would be > a bug (whether in pyOpenSSL packaging or in setuptools), I was using Python 2.5, so I grabbed the one from comment 71. > Does the .msi install a file or directory named "pyOpenSSL-0.9-py2.6- > win32.egg-info"? Not sure. I did finally get pyopenssl to build from bzr source last night, and got Tahoe running that way, but I did so many different things I'm not sure what was necessary. I'm starting over this morning (in and around meetings) with a clean VM which I'll snapshot at appropriate points so I can roll back changes as I go through the various permutations and identify what's really needed. > > BTW... "an hour's labor"... Ha! > > Ahem. Yes, well in my defense almost none of the problems we've > encountered so far have to do with building Tahoe-LAFS itself so much > as with building dependencies. Agreed. There's been no trouble with Tahoe itself. > I just now got the python 2.5 > win32 package of pycryptopp automatically uploading: Cool. I'll see how that impacts the build process. > There. Okay as visible in the README.html (scroll down to the bottom > when viewing the "tahoe-dep-eggs" directory), we are lacking packages > for some versions of Python on some operating systems. Hopefully > we'll get some volunteer buildbots set up to build those. That would be good. I don't have any windows machines, or any Linux machines with enough spare RAM to leave a Windows VM running all the time, or I'd offer. Shawn. From trac at allmydata.org Mon Jul 27 06:13:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 13:13:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.78701f550461d9384e891921aa24748d@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Black Dew's debian-unstable buildslave is now trying to upload .deb's... http://allmydata.org/buildbot-pycryptopp/builders/BlackDew%20debian- unstable-i386 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 06:14:31 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 13:14:31 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.d58cada6a9a318f3ffd0232f7122937f@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Nobody has yet succeeded at installing the current snapshot (release candidate for v1.5.0) Tahoe-LAFS from source on Windows. But we're getting closer! http://allmydata.org/pipermail/tahoe-dev/2009-July/002426.html -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Mon Jul 27 07:19:28 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Mon, 27 Jul 2009 08:19:28 -0600 Subject: [tahoe-dev] seeking volunteer for many hours labor Re: [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <200907270713.55072.shawn-tahoe@willden.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907262227.31911.shawn-tahoe@willden.org> <79F40FAA-274C-435F-B03E-4BD05051BBF5@zooko.com> <200907270713.55072.shawn-tahoe@willden.org> Message-ID: <383C5D4F-8705-4485-BD4C-00827EEBFE96@zooko.com> On Monday,2009-07-27, at 7:13 , Shawn Willden wrote: >> I just now got the python 2.5 >> win32 package of pycryptopp automatically uploading: > > Cool. I'll see how that impacts the build process. Remove the remote URLs from the "find_links" section in the setup.cfg to prevent the build process from looking there. However, it will still look on http://pypi.python.org as well as follow links from there to other sites. Perhaps the easiest way to keep it from depending on remote packages at build time is to turn off your network. :-/ Regards, Zooko From trac at allmydata.org Mon Jul 27 06:58:28 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 13:58:28 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.2424864742977c2343024f1b50d0756f@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): It worked! Now we need someone with Debian to test whether {{{sudo apt- get install allmydata-tahoe}}} works. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From ludo at gnu.org Mon Jul 27 07:22:59 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Mon, 27 Jul 2009 16:22:59 +0200 Subject: [tahoe-dev] seeking volunteer for many hours labor Re: [tahoe-lafs] #773: run through installation on Windows References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> <200907262227.31911.shawn-tahoe@willden.org> <79F40FAA-274C-435F-B03E-4BD05051BBF5@zooko.com> <200907270713.55072.shawn-tahoe@willden.org> <383C5D4F-8705-4485-BD4C-00827EEBFE96@zooko.com> Message-ID: <87hbwykxx8.fsf@gnu.org> Hello, Zooko Wilcox-O'Hearn writes: > Remove the remote URLs from the "find_links" section in the setup.cfg > to prevent the build process from looking there. However, it will > still look on http://pypi.python.org as well as follow links from > there to other sites. Perhaps the easiest way to keep it from > depending on remote packages at build time is to turn off your > network. :-/ If all the dependencies it needs are available in $PYTHONPATH, easy_install(1) will not try to contact pypi.python.org et al. (That's what the NixOS package does.) Thanks, Ludo'. From trac at allmydata.org Mon Jul 27 10:19:05 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 17:19:05 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windowshttp://allmydata.org/source/tahoe/snapshots/ (was: run through installation on Windows) In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.87513fb789105ce5beeaa0200e2e0ce9@allmydata.org> #773: run through installation on Windowshttp://allmydata.org/source/tahoe/snapshots/ -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): Here's a sequence that works: 1. Install Windows XP, Service Pack 2 (other variations should work, untested) 2. Download and install Python 2.5 from http://www.python.org/download/releases/2.5.4/ 3. Download and install pywin32 from http://sourceforge.net/projects/pywin32/ 4. Download and install MinGW from http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/MinGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You don't need any of the optional components, just the base. 5. Configure distutils to use MinGW. To configure distutils to use mingw32, create a file called "distutils.cfg" in C:\Python25\Libs\distutils, and put in that file: {{{ [build] compiler=mingw32 }}} 6. Add the MinGW binary path to your Path variable. To do that, go to the control panel (classic view), double-click "System", click on the "Advanced" tab, click on the "Environment Variables" button, scroll down to the "Path" variable in the "System Variables" list, double-click it, and append ";C:\MinGW\bin" to the path. While you're at it, might as well append ";C:\Python25;C:\Python25\Scripts" also so that you don't have to type full paths to run python or installed python scripts. 7. Download and install the Visual Studio 2008 redistributables from http://www.microsoft.com/downloads/details.aspx?familyid=9B2DA534-3E03-4391 -8A4D-074B9F2BC1BF&displaylang=en. This is needed for OpenSSL. 8. Download and install OpenSSL v.9.8k Light from http://www.slproweb.com/products/Win32OpenSSL.html. 9. Download the latest Tahoe snapshot from http://allmydata.org/source/tahoe/snapshots/. Unpack it. I put it in C:\tahoe 10. Open a command prompt and cd to the top of the Tahoe tree (e.g. cd \tahoe). 11. Run "python setup.py build". Wait a bit until it stops working. Note that you need a working network connection because it will download various dependencies. 12. Run "python setup.py install". Wait a bit until it stops working. Again, you may need a working network connection. 13. Run "tahoe create-client --basedir C:\tahoebase". You can pick a different base directory if you want. 14. Run "notepad C:\tahoebase\tahoe.cfg" to edit your config file. Paste in your introducer FURL. 15. Run "tahoe start --basedir C:\taboebase". Your node will start running and connect to the grid! The Windows firewall may ask whether or not to allow "python" to make network connections. Say yes. Easy peasy! ;-) Actually, all of the MinGW steps (steps 4-6) could be omitted if win32 eggs were available for all dependencies. I think zfec being out of date is currently the only issue requiring installing a compiler. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 12:06:00 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 19:06:00 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows (was: run through installation on Windowshttp://allmydata.org/source/tahoe/snapshots/) In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.9b9e0baf04acd5c52717961ce64c2911@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): And this works with 2.6: 1. Install Windows XP, Service Pack 2 (other variations should work, untested) 2. Download and install Python 2.6.2 from http://www.python.org/ftp/python/2.6.2/python-2.6.2.msi 3. Download and install pywin32 from http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/pywin32-214.win32-py2.6.exe/download 4. Download and install MinGW from http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/MinGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You need the base components plus the C++ compiler 5. Configure distutils to use MinGW. To configure distutils to use mingw32, create a file called "distutils.cfg" in C:\Python25\Libs\distutils, and put in that file: {{{ [build] compiler=mingw32 }}} 6. Add the MinGW binary path to your Path variable. To do that, go to the control panel (classic view), double-click "System", click on the "Advanced" tab, click on the "Environment Variables" button, scroll down to the "Path" variable in the "System Variables" list, double-click it, and append ";C:\MinGW\bin" to the path. While you're at it, might as well append ";C:\Python25;C:\Python25\Scripts" also so that you don't have to type full paths to run python or installed python scripts. 7. Download the latest Tahoe snapshot from http://allmydata.org/source/tahoe/snapshots/. Unpack it. I put it in C:\tahoe 8. Open a command prompt and cd to the top of the Tahoe tree (e.g. cd \tahoe). 9. Run "python setup.py build". Wait a bit until it stops working. Note that you need a working network connection because it will download various dependencies. 10. Run "python setup.py install". Wait a bit until it stops working. Again, you may need a working network connection. 11. Run "tahoe create-client --basedir C:\tahoebase". You can pick a different base directory if you want. 12. Run "notepad C:\tahoebase\tahoe.cfg" to edit your config file. Paste in your introducer FURL. 13. Run "tahoe start --basedir C:\taboebase". Your node will start running and connect to the grid! The Windows firewall may ask whether or not to allow "python" to make network connections. Say yes. Actually, all of the MinGW steps (steps 4-6) could be omitted if win32 eggs were available for all dependencies. I think zfec and pycryptopp being out of date are currently the only issues requiring installing a compiler. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 12:31:42 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 19:31:42 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.52b0530fb5f5bed47734d8adccac8a84@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Let's see... According to http://allmydata.org/source/tahoe/deps/tahoe- dep-eggs/ we have {{{zfec-1.4.4-py2.5-win32.egg}}} and {{{zfec-1.4.4-py2.6-win32.egg}}} . Since [source:_auto_deps.py] specifies {{{zfec >= 1.1.0}}}, then this ought to satisfy. Is zfec being rebuilt from source when you do this process? As for pycryptopp, we currently have {{{pycryptopp-0.5.16-py2.5-win32.egg}}} and {{{pycryptopp-0.5.12-py2.6-win32.egg}}}. The one for py2.6 is too old a version of pycryptopp, so I would expect pycryptopp to get built from source if using Python 2.6 but not if using Python 2.5. Is this consistent with your process? Thanks! -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 13:07:29 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 20:07:29 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.389b70767dc5a76249c4598c1ef823b4@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): zfec is being rebuilt from source for 2.5. I just started the 2.5 + pywin32 snapshot and re-verified. I believe that's the only reason I need a compiler for 2.5. It appears that the reason zfec is being rebuilt is that the tarball is 1.4.5 while the egg is 1.4.4. Both satify the requirement, but setup takes the newer one. That seems like a bug to me, and one that may often require the ability to build from source when it isn't really necessary. pycryptopp is being rebuilt from source for 2.6, but not for 2.5. The other oddity is that the pyOpenSSL for 2.5 needs OpenSSL DLLs to be installed separately, but pyOpenSSL for 2.6 does not. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 13:12:25 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 20:12:25 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.00f1aabced205802295be4de6bebf588@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): Oh, one more thing: Removing the "sdists" URL from the setup.cfg find_list allows the build to complete. The zfec egg is used then. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 14:54:33 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 21:54:33 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.964539be7406668780168ed89b21b7c8@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): Replying to [comment:9 zooko]: > Added a ticket to setuptools: http://bugs.python.org/setuptools/issue83 (I wish setuptools would prefer v1.4.1 in binary egg format over v1.4.5 in sdist format when I'm requiring >= v1.1.0). Until your wish is fulfilled, perhaps it would be good to offer a "binary- only" 1.5.0 release that does not have the sdist line in setup.cfg. For systems without development tools installed, removing the sdist line will make the installation more likely to succeed. Even better (for Windows) is to get a working py2exe installer that "Just Works", of course. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Mon Jul 27 15:04:55 2009 From: trac at allmydata.org (tahoe-lafs) Date: Mon, 27 Jul 2009 22:04:55 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.af0b88486ef5ec63db455c5b1b29d6b9@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): > Even better (for Windows) is to get a working py2exe installer that "Just Works", of course. I hope that these are complementary, not alternative, i.e. I hope that we have a well-tested and documented process of building from source including building dependencies from source (layer 1), as well as a good process of building Tahoe-LAFS from source including installing binary dependencies (layer 2) (this one can be much easier than layer 1 -- no C compiling required!), as well as a prebuilt py2exe or bbfreeze installer for Windows (layer 3). I don't want the existence of the higher layers to mean we allow the lower layers to rot, for example, if people could easily build Tahoe-LAFS by pulling in binary dependencies, but nobody knew how to build the dependencies from source. This was actually the case at one point in the past -- Tahoe-LAFS required pyOpenSSL but nobody -- literally nobody that I was able to find -- knew how to build pyOpenSSL from source for Windows. We had to rely on downloading a prebuilt pyOpenSSL binary from this one web site where it had, as far as we could tell, been abandoned by whoever originally compiled it. I'm glad that's not the case anymore! Anyway, until my wish is fulfilled (http://bugs.python.org/setuptools/issue83 ), I think a reasonable policy for layer 2 is that we don't upload sdists of dependencies without also uploading accompanying bdists, at least for Windows and probably also Mac. I've already automated most of this for pycryptopp (using Brian's excellent new {{{flappserver}}} tool, plus his excellent old {{{buildbot}}} tool), and I guess I'll proceed to do similar automation for zfec next... -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Mon Jul 27 15:21:39 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 27 Jul 2009 16:21:39 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates Message-ID: <1248733299.494025.28646.nullmailer@mordsith.willden.org> Wed Jul 22 12:20:52 MDT 2009 Shawn Willden * Lossmodel updates Various improvements to the lossmodel paper. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/x-darcs-patch Size: 689052 bytes Desc: A darcs patch for your repository! Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090727/e7012731/attachment-0001.bin From shawn-tahoe at willden.org Mon Jul 27 15:30:19 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Mon, 27 Jul 2009 16:30:19 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: <1248733299.494025.28646.nullmailer@mordsith.willden.org> References: <1248733299.494025.28646.nullmailer@mordsith.willden.org> Message-ID: <200907271630.19867.shawn-tahoe@willden.org> On Monday 27 July 2009 04:21:39 pm Shawn Willden wrote: > Wed Jul 22 12:20:52 MDT 2009 Shawn Willden > * Lossmodel updates > > Various improvements to the lossmodel paper. Sorry for the size. It seems to me that there's little value in having the paper in the tree without a PDF of it that people can read. I probably should have separated this into two patches -- one that updates the source document and one that adds the PDF. Shawn. From trac at allmydata.org Mon Jul 27 20:10:07 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 03:10:07 -0000 Subject: [tahoe-dev] [tahoe-lafs] #776: users are confused by "tahoe rm" Message-ID: <037.5d52af220f12376061c9078b9a3fad30@allmydata.org> #776: users are confused by "tahoe rm" -------------------------------+-------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-frontend-cli | Version: 1.4.1 Keywords: | Launchpad_bug: -------------------------------+-------------------------------------------- I found the following conversation log between a new user (!PovAddict) and an experienced user (soul9). They are confused about what "tahoe rm" means. Possible changes that could improve this include renaming it to "tahoe unlink", improving the error messages (see below), and improving the docs. {{{ I just installed a tahoe client and attached it to the "volunteer grid" [19:01] ... also, too many Python exceptions to take tahoe seriously... [19:17] PovAddict: what python exceptions. [19:19] ? python2.6? 2.5.2 hm i didn't have python exceptions, except deprecationwarnings in python 2.6 http://pastebin.com/df3bfddf [19:20] nicolas at ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe ls Traceback (most recent call last): File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')() File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/cli.py", line 380, in list rc = tahoe_ls.list(options) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/tahoe_ls.py", line 18, in list rootcap, path = get_alias(aliases, where, DEFAULT_ALIAS) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/common.py", line 148, in get_alias return aliases[default], path KeyError: 'tahoe' nicolas at ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe ls URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946 Traceback (most recent call last): File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')() File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/cli.py", line 380, in list rc = tahoe_ls.list(options) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/tahoe_ls.py", line 67, in list childtype = child[0] KeyError: 0 nicolas at ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe rm URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946 Traceback (most recent call last): File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')() File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run rc = runner(sys.argv[1:]) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner rc = cli.dispatch[command](so) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/cli.py", line 409, in rm rc = tahoe_rm.rm(options) File "/home/nicolas/src/allmydata- tahoe-1.4.1/src/allmydata/scripts/tahoe_rm.py", line 25, in rm assert path AssertionError I uploaded a file to the VolunteerGrid from the web interface I can download it back from there, or using "tahoe get" yes, well and that's it; is 'rm' supposed to work? i think that has to do with a not defined default rootcap [19:21] yes, it is, of course no matter what the problem really is, the command line tool should give a proper error message, instead of dumping a stacktrace... er [19:22] "KeyError: 0" might only mean something to people who know the code and know the causes of this situation definitely means nothing to me it's not dumping it's core at all it printed a stacktrace on the console [19:23] it's trying to tell you you didn't give it where to ls no, that's warnings if you don't want that do tahoe cmd ... blabla 2> /dev/null "KeyError: 0" and "AssertionError" are not warnings, they are exceptions [19:24] they end in error because there is nothing to ls tahoe has no "root" so you can't do an ls without giving it a default rootcap my point is: why does it say "KeyError: 0" instead of "rootcap required"? which is the tahoe "alias" is it an argument I'm supposed to pass after 'ls'? [19:25] well, that's a good point, and you should definitely open a bug if there isn't one allready "tahoe ls something"? tahoe ls ROOT:cap:thisisasecuretahoeurlwhateverhexblah yeah, it's an tahoe rootcap "tahoe ls --help" doesn't say there are required arguments [19:26] so e.g. you do tahoe mkdir foo gives you a rootcap back then you can save the rootcap as an alias so you don't need to remember the horridly long hash you can give it a name because if you add an tahoe alias then that will be used as default okay, actually, --help doesn't even mention any argument [19:27] just the options which is why it's looking for the tahoe key and "tahoe [command options] mkdir [options]" makes no sense :) mkdir *is* the anyway; what about the rm error? * soul9 ? bugs try it with a rootcap [19:28] it'll work tahoe is well coded for the purpose it serves wich is uploading big files using the web ui because the cmdline client also just uses the webui in the backend [19:29] hell, the fuse client just uses the webui aswell I don't know how I'm supposed to get that 'rootcap' I uploaded a file from the web ui, now I want to delete it *** jsgf (n=jeremy at adsl-69-107-65-92.dsl.pltn13.pacbell.net) has quit: Read error: 110 (Connection timed out) I have the URI I got after uploading [19:30] that's the cap and I can use it to download the file that big hash yup so you do tahoe rm HASH tahoe ls HASH and it throws an AssertionError, as shown on my paste [19:31] tahoe cp foo HASH:dor/you/want hm [19:32] dunno worksforme? also note line 16 of paste was wrapped; I *did* pass a hash to ls and it just throws a different error [19:33] then you have a problem in your setup, or the directory is empty iirc I didn't create a directory [19:34] I uploaded it from here: http://nooxie.zooko.com:9798/ using the "Upload a file" form, I didn't create a directory first [19:35] unfortunately none of the shares ended up on my own node, which is kind of what I wanted to try [19:36] so, now, I want to delete the file from the grid; and I only have the one hash it gave me yeah, tahoe ls file doesn't work [19:38] you can't ls a file use the hash to delete it it's as simple as tahoe rm HASH I did and it throws AssertionError the hash is in the pastebin; can you try rm yourself? is the file still there tho? [19:39] i'm not connected to the volounteer grid i have my own ;) 'check' says the file is healthy Summary: Healthy weird, it should work storage index: xetcmqax6bi5eozzggamxh7gg4 [19:40] good-shares: 10 (encoding is 3-of-10) mmmm [19:41] ? what is the introducer furl to the volounteer grid? [19:42] pb://6cypm5tfsv6ag43g3j5z74qx5nfxa2qq at 207.7.145.200:64228,nooxie.zooko.com:64228/introducer mm [19:45] that's a bug! [19:46] can't rm it :D too bad the contest is about hacking tahoe and not just breaking it :) [19:47] :D include that hash in the bugreport [19:49] what version of tahoe are you using ? 1.4.1 I'm currently running it under a debugger, stepping through get_alias [19:50] ok so, if the path starts with "URI: but doesn't contain ":./", then get_alias returns "" for path [19:51] and then 'assert path' explodes really, this is weird man [19:52] i have put stuff and deleted through the cmdline and the webui and mixes never encountered this I think I got it [19:53] but I'll upload another file and try again 'rm' seems not to be prepared to delete files out of directories... [20:03] hmm looks like you can't delete files actually! [20:06] "Note that this does not actually delete the file or directory that the name points to from the tahoe grid -- it only removes the named reference from this directory." [20:07] the object will just become "unreachable" (but the data still there?) if no reachable directories have the file, and everybody loses the readcap [20:08] so when do shares actually disappear? [20:10] i think the crawxler is the garbage collector [20:14] }}} -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Tue Jul 28 05:46:49 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Tue, 28 Jul 2009 06:46:49 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: <200907271630.19867.shawn-tahoe@willden.org> References: <1248733299.494025.28646.nullmailer@mordsith.willden.org> <200907271630.19867.shawn-tahoe@willden.org> Message-ID: Yes, I don't want to add the 100 KB PDF file into our source tree. In general I much prefer for the source tree to contain only human-written artifacts, not computer-generated artifacts. What about adding a hyperlink from the latex source to a directory on the TestGrid that hosts the PDF? :-) Regards, Zooko From shawn-tahoe at willden.org Tue Jul 28 06:32:04 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 28 Jul 2009 07:32:04 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: References: <1248733299.494025.28646.nullmailer@mordsith.willden.org> <200907271630.19867.shawn-tahoe@willden.org> Message-ID: <200907280732.04481.shawn-tahoe@willden.org> On Tuesday 28 July 2009 06:46:49 am Zooko O'Whielacronx wrote: > What about > adding a hyperlink from the latex source to a directory on the > TestGrid that hosts the PDF? :-) We could do that. The other option I considered was linking to a copy on my web server. Shawn From trac at allmydata.org Tue Jul 28 06:32:13 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 13:32:13 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.38fb6a2c97d97baef6c1ddc08783aab3@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Okay, and thanks to Arthur Lutz we now have a second Debian buildslave uploading .deb's, this one is lenny: http://allmydata.org/buildbot-pycryptopp/builders/Arthur%20debian- lenny-c7-i386/builds/22/steps/upload%20deb/logs/stdio Now we just need someone with Debian-or-Ubuntu/x86 to test whether the {{{pycryptopp}}} .deb's in the allmydata.org apt repo work for them (and whether therefore the {{{allmydata-tahoe}}} .deb's also work). -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 28 07:11:04 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 14:11:04 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.40de3ee2ec949534f5ab0eb5d71cd11a@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): Okay I hooked up allmydata.com's Windows buildslave to upload binary .egg's for Python 2.5 for Win32. Now I guess if you are using Python 2.5 there should be *no* compilation required, so steps 4-6 of Shawn's process in comment:5 should be unnecessary. If we get a Windows buildbot running Python 2.6 working, then this should be true for Python 2.6 as well. (Black Dew has contributed one, but at the moment it is very confused and fails to "download" the setuptools (actually {{{zetuptoolz}}}) .egg from the {{{misc/dependencies/}}} subdirectory. :-( http://allmydata.org/buildbot-zfec/builders/BlackDew%20windows- mingw32-py26 -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 28 07:23:09 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 14:23:09 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.63b5bf302698223ca761136a48db6739@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by arthur): This is what I get when doing a apt-get update : Err http://allmydata.org lenny/main Packages 404 Not Found Err http://allmydata.org lenny/tahoe Packages 404 Not Found Err http://allmydata.org lenny/main Sources 404 Not Found Err http://allmydata.org lenny/tahoe Sources 404 Not Found W: Failed to fetch http://allmydata.org/debian/dists/lenny/main/binary-i386/Packages 404 Not Found W: Failed to fetch http://allmydata.org/debian/dists/lenny/tahoe/binary-i386/Packages 404 Not Found W: Failed to fetch http://allmydata.org/debian/dists/lenny/main/source/Sources 404 Not Found W: Failed to fetch http://allmydata.org/debian/dists/lenny/tahoe/source/Sources 404 Not Found E: Some index files failed to download, they have been ignored, or old ones used instead. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From cyberaxe at hotmail.com Tue Jul 28 10:35:09 2009 From: cyberaxe at hotmail.com (CyberAxe-Hotmail) Date: Tue, 28 Jul 2009 11:35:09 -0600 Subject: [tahoe-dev] tahoe windows test error Message-ID: -----Original Message----- From: CyberAxe-Hotmail [mailto:cyberaxe at hotmail.com] Sent: Tuesday, July 28, 2009 10:33 AM To: 'Zooko Wilcox-O'Hearn' Subject: FW: tahoe windows test error 4. Download and install MinGW from http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/Mi nGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You need the base components plus the C++ compiler Note: this program has a typo that shows c++ as g++ I missed adding c++ compiler because it says g++. -----Original Message----- From: CyberAxe-Hotmail [mailto:cyberaxe at hotmail.com] Sent: Tuesday, July 28, 2009 10:26 AM To: 'Zooko Wilcox-O'Hearn' Subject: tahoe windows test error Steps I took - summary changed from run through installation on Windowshttp://allmydata.org/source/tahoe/snapshots/ to run through installation on Windows And this works with 2.6: -OK 1. Install Windows XP, Service Pack 2 (other variations should work, untested) Mine - Windows 7 -7100 RC 64-bit -OK 2. Download and install Python 2.6.2 from http://www.python.org/ftp/python/2.6.2/python-2.6.2.msi -OK 3. Download and install pywin32 from http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/pywin32-21 4.win32-py2.6.exe/download -OK 4. Download and install MinGW from http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/Mi nGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You need the base components plus the C++ compiler -Ok w/change 5. Configure distutils to use MinGW. To configure distutils to use mingw32, create a file called "distutils.cfg" in C:\Python26(not 25)\Lib(not Libs)\distutils, and put in that file: [build] compiler=mingw32 -OK w/change 6. Add the MinGW binary path to your Path variable. To do that, go to the control panel (classic view), double-click "System", click on the "Advanced" tab, click on the "Environment Variables" button, scroll down to the "Path" variable in the "System Variables" list, double-click it, and append ";C:\MinGW\bin" to the path. While you're at it, might as well append ";C:\Python26(not 25);C:\Python26(not 25)\Scripts" also so that you don't have to type full paths to run python or installed python scripts. -OK 7. Download the latest Tahoe snapshot from http://allmydata.org/source/tahoe/snapshots/. Unpack it. I put it in C:\tahoe NOTE: used "allmydata-tahoe-1.4.1-r4026.zip" -OK 8. Open a command prompt and cd to the top of the Tahoe tree (e.g. cd \tahoe). NOTE: Does Windows Vista and Windows 7 need this "Run as Administrator"? -Failed 9. Run "python setup.py build". Wait a bit until it stops working. Note that you need a working network connection because it will download various dependencies. NOTE: see complete log below " File "C:\tahoe\setuptools-0.6c12dev.egg\setuptools\command\easy_install.py", l ine 446, in easy_install distutils.errors.DistutilsError: Could not find suitable distribution for Requir ement.parse('Twisted>=2.4.0') 10. Run "python setup.py install". Wait a bit until it stops working. Again, you may need a working network connection. 11. Run "tahoe create-client --basedir C:\tahoebase". You can pick a different base directory if you want. 12. Run "notepad C:\tahoebase\tahoe.cfg" to edit your config file. Paste in your introducer FURL. 13. Run "tahoe start --basedir C:\taboebase". Your node will start running and connect to the grid! The Windows firewall may ask whether or not to allow "python" to make network connections. Say yes. Actually, all of the MinGW steps (steps 4-6) could be omitted if win32 eggs were available for all dependencies. I think zfec and pycryptopp being out of date are currently the only issues requiring installing a compiler. --------------------------- Resulted in error C:\tahoe>python setup.py build Downloading file:misc/dependencies/setuptools-0.6c12dev.egg C:\tahoe\ez_setup.py:29: DeprecationWarning: the md5 module is deprecated; use h ashlib instead from md5 import md5 Not found: tahoe-deps Not found: ../tahoe-deps Installed c:\tahoe\setuptools_darcs-1.2.8-py2.6.egg Searching for setuptools-trial>=0.5 Best match: setuptools-trial 0.5.2 Processing setuptools_trial-0.5.2.tar Running setuptools_trial-0.5.2\setup.py -q bdist_egg --dist-dir c:\users\jbenis~ 1.bly\appdata\local\temp\easy_install-dothm6\setuptools_trial-0.5.2\egg-dist -tmp -jakszj Installed c:\tahoe\setuptools_trial-0.5.2-py2.6.egg Searching for darcsver>=1.2.0 Best match: darcsver 1.2.1 Processing darcsver-1.2.1.tar Running darcsver-1.2.1\setup.py -q bdist_egg --dist-dir c:\users\jbenis~1.bly\ap pdata\local\temp\easy_install-kphdgm\darcsver-1.2.1\egg-dist-tmp-2jiruo Installed c:\tahoe\darcsver-1.2.1-py2.6.egg Searching for Twisted>=2.4.0 Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/ Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ Reading http://pypi.python.org/simple/Twisted/ Couldn't find index page for 'Twisted' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://pypi.python.org/simple/ No local packages or download links found for Twisted>=2.4.0 Traceback (most recent call last): File "setup.py", line 363, in zip_safe=False, # We prefer unzipped for easier access. File "C:\Python26\lib\distutils\core.py", line 113, in setup _setup_distribution = dist = klass(attrs) File "C:\tahoe\setuptools-0.6c12dev.egg\setuptools\dist.py", line 219, in __in it__ File "C:\tahoe\setuptools-0.6c12dev.egg\setuptools\dist.py", line 243, in fetc h_build_eggs File "C:\tahoe\setuptools-0.6c12dev.egg\pkg_resources.py", line 522, in resolv e File "C:\tahoe\setuptools-0.6c12dev.egg\pkg_resources.py", line 758, in best_m atch File "C:\tahoe\setuptools-0.6c12dev.egg\pkg_resources.py", line 770, in obtain File "C:\tahoe\setuptools-0.6c12dev.egg\setuptools\dist.py", line 286, in fetc h_build_egg File "C:\tahoe\setuptools-0.6c12dev.egg\setuptools\command\easy_install.py", l ine 446, in easy_install distutils.errors.DistutilsError: Could not find suitable distribution for Requir ement.parse('Twisted>=2.4.0') C:\tahoe> -------------- next part -------------- A non-text attachment was scrubbed... Name: MiniGW-error typo c++.JPG Type: image/jpeg Size: 12823 bytes Desc: not available Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090728/39c2d703/attachment.jpeg From zooko at zooko.com Tue Jul 28 10:51:51 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 28 Jul 2009 11:51:51 -0600 Subject: [tahoe-dev] tahoe windows test error In-Reply-To: References: Message-ID: Could someone please turn this knowledge that is accumulating in the trac ticket and mailing list archives into a wiki page? I nominate Shawn Willden, although if someone else who didn't create most of the information in the first place wants to try editing it into a nice wiki format, that might be an opportunity for them to discover omissions or editing mistakes in Shawn's original recipe. Thank you! Regards, Zooko From zooko at zooko.com Tue Jul 28 11:04:20 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Tue, 28 Jul 2009 12:04:20 -0600 Subject: [tahoe-dev] tahoe windows test error In-Reply-To: References: Message-ID: <600646E8-39DD-442A-8871-2F85052083AE@zooko.com> This is basically both layer 1 (build everything from source) and layer 2 (build Tahoe-LAFS using binary dependencies) that we're talking about here. Perhaps they deserve separate wiki pages? I'll leave that up to whoever volunteers to write a good wiki page about it. By the way, layer 3 already exists! The Windows buildslave operated by allmydata.com uses py2exe to build an installer for Tahoe- LAFS, on every checkin: http://allmydata.org/buildbot/builders/windows/builds/1566/steps/ installer/logs/stdio It then uploads the resulting installer: rsync -av -e /usr/bin/ssh dist/installer/ amduser at dev:/home/amduser/ public_html/dist/tahoe/windows/ Ah, the hostname "dev:" means hanford.allmydata.com, an internal server of allmydata.com that is accessible only to allmydata.com employees: http://dev.allmydata.com/~amduser/dist/tahoe/windows/ If someone wants to try out the Windows installer for Tahoe-LAFS (I personally don't have time right now and my children are always borrowing my Windows laptop to play Battle For Wesnoth), then we should arrange for those installers to get uploaded to http:// allmydata.org . Regards, Zooko From trac at allmydata.org Tue Jul 28 10:59:37 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 17:59:37 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: run through installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.02a12c005b7b9193dd8494f09c223f13@allmydata.org> #773: run through installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: warner Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by swillden): Replying to [comment:12 zooko]: > Okay I hooked up allmydata.com's Windows buildslave to upload binary .egg's for Python 2.5 for Win32. Now I guess if you are using Python 2.5 there should be *no* compilation required, so steps 4-6 of Shawn's process in comment:5 should be unnecessary. I've confirmed this. Installation of OpenSSL is still necessary, but no compilation is needed for 2.5. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From warner at lothar.com Tue Jul 28 13:54:47 2009 From: warner at lothar.com (Brian Warner) Date: Tue, 28 Jul 2009 16:54:47 -0400 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: <200907271630.19867.shawn-tahoe@willden.org> References: <1248733299.494025.28646.nullmailer@mordsith.willden.org> <200907271630.19867.shawn-tahoe@willden.org> Message-ID: <98B2B6F0-1C18-4BC0-A8A6-76B12804F130@lothar.com> On Jul 27, 2009, at 6:30 PM, Shawn Willden wrote: > Sorry for the size. It seems to me that there's little value in > having the > paper in the tree without a PDF of it that people can read. I go back and forth on this sort of thing. One compromise I've tried with other projects is to put the document source (TeX, lyx, SVG, texinfo, etc) under version control, but then arrange for the release process to render it into something readable (PDF, png, info) for inclusion in the release tarball. Folks who are working with a trunk checkout are used to doing more work anyways. I also upload rendered versions to a website upon each release. A "make release" target with generates doc PDFs and then runs "setup.py sdist" (and a suitable MANIFEST) entry to include the generated files) would probably do the trick. cheers, -Brian From shawn-tahoe at willden.org Tue Jul 28 14:29:42 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 28 Jul 2009 15:29:42 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: <98B2B6F0-1C18-4BC0-A8A6-76B12804F130@lothar.com> References: <1248733299.494025.28646.nullmailer@mordsith.willden.org> <200907271630.19867.shawn-tahoe@willden.org> <98B2B6F0-1C18-4BC0-A8A6-76B12804F130@lothar.com> Message-ID: <200907281529.42571.shawn-tahoe@willden.org> On Tuesday 28 July 2009 02:54:47 pm Brian Warner wrote: > On Jul 27, 2009, at 6:30 PM, Shawn Willden > > wrote: > > Sorry for the size. It seems to me that there's little value in > > having the paper in the tree without a PDF of it that people can > > read. > > I go back and forth on this sort of thing. One compromise I've tried > with other projects is to put the document source (TeX, lyx, SVG, > texinfo, etc) under version control, but then arrange for the release > process to render it into something readable (PDF, png, info) for > inclusion in the release tarball. I also generally prefer to avoid including generated files in source control. It's a bigger issue with generated files that are expected to change frequently, which this won't, but it's still an issue. In this case I'm not sure that automatically building the PDF is a good choice, though, because the requirements to build it are pretty heavy. A full TeX installation is around a half a gigabyte, plus a bit more for LyX. I think zooko's suggestion is probably the best one: just include a link to the PDF hosted elsewhere. I'll submit a revised patch with just the source changes plus the addition of a README file that links to the PDF. Shawn. From trac at allmydata.org Tue Jul 28 14:38:32 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 21:38:32 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.ff6e69a6c08b3e09bd68a83f63d5aa43@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by warner): I'm in an airport now, but when I'm back at home tomorrow I'll try to fix up the apt repo update scripts to include the new releases: lenny, jaunty, etc. (I had set up the flappserver receivers ahead of time, for more releases than we had APT automation to handle). I'm a bit worried about the version numbers on the pycryptopp debs: it looks like they're being created with "0.5.16-1", with no "r1234"-type suffix. Is that right? The APT repository (and the debian clients which pull from it) will get confused if two different binaries are uploaded with the same version number, such as if 0.5.16-r1234 and 0.5.16-r1235 are both uploaded under the name "0.5.16-1". Is stdeb using the output of darcsver? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 28 14:53:24 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 21:53:24 -0000 Subject: [tahoe-dev] [tahoe-lafs] #754: merge manually specified tub location with autodetected tub location In-Reply-To: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> References: <037.45a5b58438159cf41598df9b1bab75d4@allmydata.org> Message-ID: <046.a812a242a2162abd54f68d6f0ec919bd@allmydata.org> #754: merge manually specified tub location with autodetected tub location --------------------------+------------------------------------------------- Reporter: zooko | Owner: warner Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-network | Version: 1.4.1 Keywords: | Launchpad_bug: --------------------------+------------------------------------------------- Comment(by warner): maybe {{{tub.location = AUTO,1.2.3.4:1234}}} since Tub locations are supposed to be comma-separated anyways, and "AUTO" doesn't parse as a valid location (no colon). robk: maybe we should have a Trac page that points at source:docs/configuration.txt ? Or, we've kicked around the idea of automatically copying the files in docs/ into wiki pages (since most of them use wikiformatting anyways), so maybe it should point at something like [wiki:docs/configuration.txt] instead. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 28 14:57:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 21:57:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #775: no pycrypto + authorized_keys file = failure to start? In-Reply-To: <037.830fde1bf845c05dd9761f7cec87faec@allmydata.org> References: <037.830fde1bf845c05dd9761f7cec87faec@allmydata.org> Message-ID: <046.a5f9112b4ff751c3ae8437db0249e376@allmydata.org> #775: no pycrypto + authorized_keys file = failure to start? ----------------------------+----------------------------------------------- Reporter: zooko | Owner: Type: defect | Status: new Priority: major | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- Comment(by warner): oh, I'm not sure. Having configuration problems cause a clear startup failure might be better than having the node start but then fail to do something later. We could wrap the import statement and emit a prettier message, but I think I'd prefer an exception at "tahoe start" time to a log message buried in {{{logs/twistd.log}}}. What does the "explosion" look like? -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Tue Jul 28 15:01:16 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 22:01:16 -0000 Subject: [tahoe-dev] [tahoe-lafs] #770: Tahoe should allow a node to listen on multiple interfaces/ports In-Reply-To: <040.17e6ff7ec47b60c717b2a9181652819e@allmydata.org> References: <040.17e6ff7ec47b60c717b2a9181652819e@allmydata.org> Message-ID: <049.2931651ff5b369292ec417808073a6c6@allmydata.org> #770: Tahoe should allow a node to listen on multiple interfaces/ports ----------------------------+----------------------------------------------- Reporter: swillden | Owner: Type: enhancement | Status: new Priority: minor | Milestone: undecided Component: code-nodeadmin | Version: 1.4.1 Keywords: | Launchpad_bug: ----------------------------+----------------------------------------------- Changes (by warner): * component: code-network => code-nodeadmin Comment: As discussed in email, let's use a comma-separated list of strports specifications in {{{web.port=}}}, since I think commas are not normally a part of strports strings. Shawn pointed out that it's not clear what to do with the {{{BASEDIR/node.url}}} file that is written out at node startup when there are multiple web ports. I'd suggest only writing out the URL for the first {{{web.port}}} specification, and documenting the config value to encourage folks to put the cheapest locally-reachable port (e.g. non- encrypted localhost HTTP) as the first value. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From shawn-tahoe at willden.org Tue Jul 28 15:46:44 2009 From: shawn-tahoe at willden.org (Shawn Willden) Date: Tue, 28 Jul 2009 16:46:44 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates Message-ID: <1248821204.971028.9858.nullmailer@mordsith.willden.org> Tue Jul 28 16:44:30 MDT 2009 Shawn Willden * Lossmodel updates Various improvements to the lossmodel, plus addition of README.lossmodel that provides a link to the PDF. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/x-darcs-patch Size: 62479 bytes Desc: A darcs patch for your repository! Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090728/a4b0bb97/attachment-0001.bin From trac at allmydata.org Tue Jul 28 15:51:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Tue, 28 Jul 2009 22:51:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #770: Tahoe should allow a node to listen on multiple interfaces/ports In-Reply-To: <040.17e6ff7ec47b60c717b2a9181652819e@allmydata.org> References: <040.17e6ff7ec47b60c717b2a9181652819e@allmydata.org> Message-ID: <049.fe12571cb2a360d40ad148ac77039442@allmydata.org> #770: Tahoe should allow a node to listen on multiple interfaces/ports ---------------------------+------------------------------------------------ Reporter: swillden | Type: enhancement Status: new | Priority: minor Milestone: undecided | Component: code-nodeadmin Version: 1.4.1 | Keywords: Launchpad_bug: | ---------------------------+------------------------------------------------ Comment(by swillden): I've attached a very simple patch that implements multi-port listening. It still needs unit tests, but as I indicated on the mailing list I'm not sure where to implement those. If someone who knows the test infrastructure could give me some guidance, I'd be glad to write some tests. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zookog at gmail.com Wed Jul 29 06:56:22 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 29 Jul 2009 07:56:22 -0600 Subject: [tahoe-dev] bloggety blog Message-ID: Folks: I've blogged about a few things that might be of interest to tahoe-lafs hackers, including All-Or-Nothing Transform, file-append in HDFS, attacking multiple discrete log public keys at once, and a possible improvement to the theory of Zooko's Triangle. My klog is stored on the Tahoe Test Grid (so it sometimes has crummy availability if there are servers attached to the Test Grid which just silently wait when you ask them for shares). http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html Or http://zooko.com/klog If you want to give a friend a hyperlink to my klog, and you give them the first one, then this empowers your friend to subsequently check whether the klog entries he downloads are signed by my private key. (But it doesn't require him to do so -- he can just rely on http://testgrid.allmydata.com to do that for him.) If you give him the second form, the short form at http://zooko.com, then he has no choice but to rely on http://zooko.com , which means whoever controls that server or domain name gets to choose what file he gets when he makes that request. (That's part of what Zooko's Triangle was all about.) Regards, Zooko From anton.vazir at gmail.com Wed Jul 29 13:15:20 2009 From: anton.vazir at gmail.com (Anton VG) Date: Thu, 30 Jul 2009 01:15:20 +0500 Subject: [tahoe-dev] Tahoe FUSE Message-ID: Hello friends! Looking for creating a redundant distributed file storage, and see TAHOE quite promising. Do you think I can use TAHOE to create a ~10 - 50 TB multi-node, reliable (fault tolerant) storage cluster and use FUSE to access and write the data in NFS-like way? I do not need to operate often-cahged data, like databases or WEB, but I need a local file store, for storing log files, backups, and also mp3s and videos, which is rare-written - but oftenly read, by more than a single person at a time. Do you think TAHOE is good for the described task, or there are some hidden stones, like FUSE is incomplete or anything? Sincerely, Anton. From zookog at gmail.com Wed Jul 29 15:10:26 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Wed, 29 Jul 2009 16:10:26 -0600 Subject: [tahoe-dev] Tahoe FUSE In-Reply-To: References: Message-ID: Welcome, Anton! Tahoe-LAFS sounds like a good match for your needs except for the hidden stone of the FUSE interface being incomplete. It doesn't even attempt to implement writes, and I don't know how will it works for reads. It does have nice tests, so anybody who wanted to fix it would get a nice head-start in that department by already having tests. Everyone who uses Tahoe-LAFS so far uses either the command-line interface (CLI) or the RESTful web API (WAPI) or the web user interface (WUI). Regards, Zooko From jbglaw at lug-owl.de Wed Jul 29 22:57:17 2009 From: jbglaw at lug-owl.de (Jan-Benedict Glaw) Date: Thu, 30 Jul 2009 07:57:17 +0200 Subject: [tahoe-dev] Tahoe FUSE In-Reply-To: References: Message-ID: <20090730055717.GA24150@lug-owl.de> On Wed, 2009-07-29 16:10:26 -0600, Zooko O'Whielacronx wrote: > Tahoe-LAFS sounds like a good match for your needs except for the > hidden stone of the FUSE interface being incomplete. It doesn't even > attempt to implement writes, and I don't know how will it works for > reads. It does have nice tests, so anybody who wanted to fix it would > get a nice head-start in that department by already having tests. > > Everyone who uses Tahoe-LAFS so far uses either the command-line > interface (CLI) or the RESTful web API (WAPI) or the web user > interface (WUI). I haven't dived into accessing Tahoe-LAFS through my OS's native VFS, but my current reading is that there are a number of native FUSE implementations for Tahoe. Additional to that, the ssh/sftp stuff *might* be nearly working. It is worth a test? ISTR that somebody tried that half a year ago and basically found only quite simple buglets that hindered it to work... MfG, JBG -- Jan-Benedict Glaw jbglaw at lug-owl.de +49-172-7608481 Signature of: http://www.eyrie.org/~eagle/faqs/questions.html the second : -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090730/1b34e4c4/attachment.pgp From secorp at allmydata.com Thu Jul 30 00:31:18 2009 From: secorp at allmydata.com (Peter Secor) Date: Thu, 30 Jul 2009 00:31:18 -0700 Subject: [tahoe-dev] Tahoe FUSE In-Reply-To: <20090730055717.GA24150@lug-owl.de> References: <20090730055717.GA24150@lug-owl.de> Message-ID: <4A714C46.8000808@allmydata.com> Also, we are about to open-source the Allmydata.com Windows client which does provide access to a storage grid via an SMB-mounted drive. This is how the Allmydata.com commercial service is built out along with several other small enterprise customers. I know this has been pending for awhile, but we haven't had time to get to it. Maybe somebody on this list might want to help me out? Ps Jan-Benedict Glaw wrote: > On Wed, 2009-07-29 16:10:26 -0600, Zooko O'Whielacronx wrote: >> Tahoe-LAFS sounds like a good match for your needs except for the >> hidden stone of the FUSE interface being incomplete. It doesn't even >> attempt to implement writes, and I don't know how will it works for >> reads. It does have nice tests, so anybody who wanted to fix it would >> get a nice head-start in that department by already having tests. >> >> Everyone who uses Tahoe-LAFS so far uses either the command-line >> interface (CLI) or the RESTful web API (WAPI) or the web user >> interface (WUI). > > I haven't dived into accessing Tahoe-LAFS through my OS's native VFS, > but my current reading is that there are a number of native FUSE > implementations for Tahoe. Additional to that, the ssh/sftp stuff > *might* be nearly working. It is worth a test? ISTR that somebody > tried that half a year ago and basically found only quite simple > buglets that hindered it to work... > > MfG, JBG > > > > ------------------------------------------------------------------------ > > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev From ludo at gnu.org Thu Jul 30 01:27:20 2009 From: ludo at gnu.org (Ludovic =?iso-8859-1?Q?Court=E8s?=) Date: Thu, 30 Jul 2009 10:27:20 +0200 Subject: [tahoe-dev] Tahoe FUSE References: <20090730055717.GA24150@lug-owl.de> Message-ID: <87iqhar2xj.fsf@gnu.org> Hi, Jan-Benedict Glaw writes: > Additional to that, the ssh/sftp stuff *might* be nearly working. It > is worth a test? ISTR that somebody tried that half a year ago and > basically found only quite simple buglets that hindered it to work... The `docs/frontends/FTP-and-SFTP.txt' file in 1.4.1 says that a patch for Twisted is needed, which is not available in Twisted 8.1.0; however it's not in the more recent 8.2.0 either, which makes it impossible to use this front-end with a released version of Twisted. Perhaps the Twisted patch should be pushed more insistently, and a workaround for Tahoe could be found in the meantime? (Unfortunately, I'm not volunteering for any of these...) What do you think? Thanks, Ludo'. From zookog at gmail.com Thu Jul 30 07:41:14 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 30 Jul 2009 08:41:14 -0600 Subject: [tahoe-dev] darcs patch: Lossmodel updates In-Reply-To: <1248821204.971028.9858.nullmailer@mordsith.willden.org> References: <1248821204.971028.9858.nullmailer@mordsith.willden.org> Message-ID: Applied in in [4030] == [20090728224430-e9cad-32750e489e8467e59a4cbe66ab5a22e22630780c]. Thanks! Regards, Zooko changesets mentioned in this message: http://allmydata.org/trac/tahoe/changeset/4030 http://allmydata.org/trac/tahoe/changeset/20090728224430-e9cad-32750e489e8467e59a4cbe66ab5a22e22630780c From trac at allmydata.org Thu Jul 30 07:47:45 2009 From: trac at allmydata.org (tahoe-lafs) Date: Thu, 30 Jul 2009 14:47:45 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: document installation on Windows (was: run through installation on Windows) In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.c160e7f5ec156447a5173856c4dc1c6a@allmydata.org> #773: document installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by zooko): This ticket is now renamed from "run through installation on Windows" (thank you Cyberaxe, Ludo', David-Sarah, Black Dew, Michael Walsh, and especially Shawn Willden!) to "document installation on Windows". To close this ticket, collect the wisdom from this ticket and from these mailing list threads and make a wiki page that will help other people install Tahoe-LAFS on Windows: http://allmydata.org/pipermail/tahoe-dev/2009-July/002392.html http://allmydata.org/pipermail/tahoe-dev/2009-July/002449.html http://allmydata.org/pipermail/tahoe-dev/2009-July/002450.html -- Ticket URL: tahoe-lafs secure decentralized file storage grid From gc20090728 at metcarob.com Thu Jul 30 09:51:48 2009 From: gc20090728 at metcarob.com (gc20090728 at metcarob.com) Date: Thu, 30 Jul 2009 17:51:48 +0100 Subject: [tahoe-dev] Installing on a Ubuntu based machine Message-ID: Hi, I am trying to install on a ubuntu based machine. I have an error in the instalisation occuring. After the process I have no file to run in the bin directory. The problem is I am getting a lot of errors but I am not sure which is the problem. I have put the log of my setup in this email. Can anyone look at it and let me know where it is failing and advise on what I need to do to fix. Thansk Robert allmydata-tahoe-1.4.1 allmydata-tahoe-1.4.1.zip linuxmce at dcerouter:~/Tahoe$ cd allmtyydata-tahoe-1.4.1 linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1$ ls _auto_deps.py CREDITS misc relnotes.txt twisted bin docs NEWS setup.cfg windows contrib ez_setup.py PKG-INFO setup.py COPYING.GPL mac README src COPYING.TGPPL.html Makefile relnotes-short.txt Tahoe.home linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1$ sudo python setup.py build Downloading file:misc/dependencies/setuptools-0.6c12dev.egg Not found: tahoe-deps Not found: ../tahoe-deps Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/setuptools_darcs-1.2.8-py2.5.egg Searching for setuptools-trial>=0.5 Best match: setuptools-trial 0.5.2 Processing setuptools_trial-0.5.2.tar Running setuptools_trial-0.5.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SE_D_l/setuptools_trial-0.5.2/egg-dist-tmp-mVbY0d Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/setuptools_trial-0.5.2-py2.5.egg Searching for darcsver>=1.2.0 Best match: darcsver 1.2.1 Processing darcsver-1.2.1.tar Running darcsver-1.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-beH79A/darcsver-1.2.1/egg-dist-tmp-GRdY01 Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/darcsver-1.2.1-py2.5.egg Searching for Twisted>=2.4.0 Reading http://allmydata.org/source/tahoe/deps/tahoe-deps/ Best match: Twisted 8.2.0 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/Twisted-8.2.0.tar.bz2 Processing Twisted-8.2.0.tar.bz2 Running Twisted-8.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-nJndG1/Twisted-8.2.0/egg-dist-tmp-IAV4LE twisted/protocols/_c_urlarg.c: In function ?unquote?: twisted/protocols/_c_urlarg.c:65: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:75: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:83: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:85: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:93: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:96: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/protocols/_c_urlarg.c:97: warning: pointer targets in passing argument 2 of ?PycStringIO->cwrite? differ in signedness twisted/python/_epoll.c: In function ?__pyx_f_6_epoll_5epoll___dealloc__?: twisted/python/_epoll.c:168: warning: label ?__pyx_L1? defined but not used twisted/python/_epoll.c: In function ?__pyx_f_6_epoll_5epoll_wait?: twisted/python/_epoll.c:432: warning: label ?__pyx_L7? defined but not used twisted/python/_epoll.c:430: warning: label ?__pyx_L6? defined but not used twisted/python/_epoll.c: In function ?__pyx_tp_new_6_epoll_epoll?: twisted/python/_epoll.c:508: warning: unused variable ?p? twisted/python/_epoll.c: In function ?__pyx_tp_dealloc_6_epoll_epoll?: twisted/python/_epoll.c:513: warning: unused variable ?p? twisted/python/_epoll.c: In function ?__pyx_tp_traverse_6_epoll_epoll?: twisted/python/_epoll.c:528: warning: unused variable ?p? twisted/python/_epoll.c:527: warning: unused variable ?e? twisted/python/_epoll.c: In function ?__pyx_tp_clear_6_epoll_epoll?: twisted/python/_epoll.c:533: warning: unused variable ?p? twisted/python/_epoll.c: At top level: twisted/python/_epoll.c:32: warning: ?__Pyx_UnpackItem? declared ?static? but never defined twisted/python/_epoll.c:33: warning: ?__Pyx_EndUnpack? declared ?static? but never defined twisted/python/_epoll.c:34: warning: ?__Pyx_PrintItem? declared ?static? but never defined twisted/python/_epoll.c:35: warning: ?__Pyx_PrintNewline? declared ?static? but never defined twisted/python/_epoll.c:37: warning: ?__Pyx_ReRaise? declared ?static? but never defined twisted/python/_epoll.c:38: warning: ?__Pyx_Import? declared ?static? but never defined twisted/python/_epoll.c:39: warning: ?__Pyx_GetExcValue? declared ?static? but never defined twisted/python/_epoll.c:40: warning: ?__Pyx_ArgTypeTest? declared ?static? but never defined twisted/python/_epoll.c:41: warning: ?__Pyx_TypeTest? declared ?static? but never defined twisted/python/_epoll.c:42: warning: ?__Pyx_GetStarArgs? declared ?static? but never defined twisted/python/_epoll.c:43: warning: ?__Pyx_WriteUnraisable? declared ?static? but never defined twisted/python/_epoll.c:45: warning: ?__Pyx_ImportType? declared ?static? but never defined twisted/python/_epoll.c:46: warning: ?__Pyx_SetVtable? declared ?static? but never defined twisted/python/_epoll.c:47: warning: ?__Pyx_GetVtable? declared ?static? but never defined twisted/python/_epoll.c:48: warning: ?__Pyx_CreateClass? declared ?static? but never defined twisted/python/_epoll.c:50: warning: ?__Pyx_InitStrings? declared ?static? but never defined twisted/python/_epoll.c:51: warning: ?__Pyx_InitCApi? declared ?static? but never defined twisted/python/_epoll.c:52: warning: ?__Pyx_ImportModuleCApi? declared ?static? but never defined Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/Twisted-8.2.0-py2.5-linux-i686.eg g Searching for zope.interface Best match: zope.interface 3.5.2 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/zope.interface-3.5.2.tar.g z Processing zope.interface-3.5.2.tar.gz Running zope.interface-3.5.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-06oLQk/zope.interface-3.5.2/egg-dist-tmp-_NRZ5a Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/zope.interface-3.5.2-py2.5-linux- i686.egg running darcsver setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone. running develop Not found: tahoe-deps Not found: ../tahoe-deps running egg_info writing requirements to src/allmydata_tahoe.egg-info/requires.txt writing src/allmydata_tahoe.egg-info/PKG-INFO writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' running build_ext Creating /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/site.py Processing setuptools-0.6c12dev.egg creating /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/setuptools-0.6c12dev.egg Extracting setuptools-0.6c12dev.egg to /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es Adding setuptools 0.6c12dev to easy-install.pth file Installing easy_install script to support/bin Installing easy_install-2.5 script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/setuptools-0.6c12dev.egg Creating /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/allmydata-tahoe.egg-link (link to src) Adding allmydata-tahoe 1.4.1 to easy-install.pth file Installing tahoe script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/src Processing dependencies for allmydata-tahoe==1.4.1 Searching for Nevow>=0.6.0 Reading http://allmydata.org/source/tahoe/deps/tahoe-deps/ Best match: Nevow 0.9.33 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/Nevow-0.9.33.tar.gz Processing Nevow-0.9.33.tar.gz Running Nevow-0.9.33/setup.py -q bdist_egg --dist-dir /tmp/easy_install-oqrdNQ/Nevow-0.9.33/egg-dist-tmp-Sl3ia- package init file 'twisted/plugins/__init__.py' not found (or not a regular file) Unable to execute darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files. If you are not building a package then you can ignore this warning. warning: no files found matching 'examples/pastebin/data' no previously-included directories found matching '*/.svn' no previously-included directories found matching 'doc/html/*.html' zip_safe flag not set; analyzing archive contents... nevow.athena: module references __file__ nevow.util: module references __file__ nevow.jsutil: module references __file__ nevow.plugins.nevow_package: module references __file__ nevow.plugins.__init__: module references __path__ nevow.livetrial.runner: module references __file__ nevow.test.test_rend: module references __file__ nevow.test.test_testutil: module references __file__ nevow.test.test_howtolistings: module references __file__ nevow.test.test_athena: module references __file__ Adding Nevow 0.9.33-r17222 to easy-install.pth file Installing nit script to support/bin Installing nevow-xmlgettext script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/Nevow-0.9.33_r17222-py2.5.egg Searching for foolscap[secure_connections]>=0.3.1 Best match: foolscap 0.4.2 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/foolscap-0.4.2.tar.gz Processing foolscap-0.4.2.tar.gz Running foolscap-0.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Bq0wen/foolscap-0.4.2/egg-dist-tmp-s60zMz zip_safe flag not set; analyzing archive contents... Adding foolscap 0.4.2 to easy-install.pth file Installing flappclient script to support/bin Installing flogtool script to support/bin Installing flappserver script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/foolscap-0.4.2-py2.5.egg Searching for simplejson>=1.4 Best match: simplejson 2.0.9 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/simplejson-2.0.9.tar.gz Processing simplejson-2.0.9.tar.gz Running simplejson-2.0.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-FaNzqh/simplejson-2.0.9/egg-dist-tmp-VRDK1Y Adding simplejson 2.0.9 to easy-install.pth file Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/simplejson-2.0.9-py2.5-linux-i686.egg Searching for zfec>=1.1.0 Best match: zfec 1.4.5 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/zfec-1.4.5.tar.gz Processing zfec-1.4.5.tar.gz Running zfec-1.4.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-2eTiyM/zfec-1.4.5/egg-dist-tmp-cQfLmG setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and zfec/_version.py already exists, so leaving it alone. Adding zfec 1.4.5 to easy-install.pth file Installing zfec script to support/bin Installing zunfec script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/zfec-1.4.5-py2.5-linux-i686.egg Searching for pycryptopp>=0.5 Best match: pycryptopp 0.5.15 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/pycryptopp-0.5.15.tar.gz Processing pycryptopp-0.5.15.tar.gz Running pycryptopp-0.5.15/setup.py -q bdist_egg --dist-dir /tmp/easy_install-JH5ecf/pycryptopp-0.5.15/egg-dist-tmp-VzGI_k setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and pycryptopp/_version.py already exists, so leaving it alone. cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++ Adding pycryptopp 0.5.15 to easy-install.pth file Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/pycryptopp-0.5.15-py2.5-linux-i686.egg Searching for pyOpenSSL Best match: pyOpenSSL 0.9 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/pyOpenSSL-0.9.tar.gz Processing pyOpenSSL-0.9.tar.gz Running pyOpenSSL-0.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-4V1RBC/pyOpenSSL-0.9/egg-dist-tmp-Ulqm1k warning: no previously-included files matching '*.pyc' found anywhere in distribution In file included from src/crypto/crypto.h:17, from src/crypto/crypto.c:14: src/crypto/x509.h:17:25: error: openssl/ssl.h: No such file or directory In file included from src/crypto/crypto.h:17, from src/crypto/crypto.c:14: src/crypto/x509.h:27: error: expected specifier-qualifier-list before ?X509? In file included from src/crypto/crypto.h:18, from src/crypto/crypto.c:14: src/crypto/x509name.h:27: error: expected specifier-qualifier-list before ?X509_NAME? In file included from src/crypto/crypto.h:19, from src/crypto/crypto.c:14: src/crypto/netscape_spki.h:24: error: expected specifier-qualifier-list before ?NETSCAPE_SPKI? In file included from src/crypto/crypto.h:20, from src/crypto/crypto.c:14: src/crypto/x509store.h:25: error: expected specifier-qualifier-list before ?X509_STORE? In file included from src/crypto/crypto.h:21, from src/crypto/crypto.c:14: src/crypto/x509req.h:25: error: expected specifier-qualifier-list before ?X509_REQ? In file included from src/crypto/crypto.h:22, from src/crypto/crypto.c:14: src/crypto/pkey.h:27: error: expected specifier-qualifier-list before ?EVP_PKEY? In file included from src/crypto/crypto.h:23, from src/crypto/crypto.c:14: src/crypto/x509ext.h:16:28: error: openssl/x509v3.h: No such file or directory In file included from src/crypto/crypto.h:23, from src/crypto/crypto.c:14: src/crypto/x509ext.h:27: error: expected specifier-qualifier-list before ?X509_EXTENSION? In file included from src/crypto/crypto.h:24, from src/crypto/crypto.c:14: src/crypto/pkcs7.h:15:27: error: openssl/pkcs7.h: No such file or directory In file included from src/crypto/crypto.h:24, from src/crypto/crypto.c:14: src/crypto/pkcs7.h:25: error: expected specifier-qualifier-list before ?PKCS7? In file included from src/crypto/crypto.h:25, from src/crypto/crypto.c:14: src/crypto/pkcs12.h:14:28: error: openssl/pkcs12.h: No such file or directory src/crypto/pkcs12.h:15:26: error: openssl/asn1.h: No such file or directory In file included from src/crypto/crypto.h:26, from src/crypto/crypto.c:14: src/crypto/../util.h:17:25: error: openssl/err.h: No such file or directory In file included from src/crypto/crypto.c:14: src/crypto/crypto.h:75: error: expected ?)? before ?*? token src/crypto/crypto.h:76: error: expected ?)? before ?*? token src/crypto/crypto.h:77: error: expected ?)? before ?*? token src/crypto/crypto.h:78: error: expected ?)? before ?*? token src/crypto/crypto.h:79: error: expected ?)? before ?*? token src/crypto/crypto.h:81: error: expected ?)? before ?*? token src/crypto/crypto.h:82: error: expected ?)? before ?*? token src/crypto/crypto.c: In function ?crypto_load_privatekey?: src/crypto/crypto.c:65: error: expected ?)? before ?*? token src/crypto/crypto.c:69: error: ?pem_password_cb? undeclared (first use in this function) src/crypto/crypto.c:69: error: (Each undeclared identifier is reported only once src/crypto/crypto.c:69: error: for each function it appears in.) src/crypto/crypto.c:69: error: ?cb? undeclared (first use in this function) src/crypto/crypto.c:71: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:71: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:72: error: ?EVP_PKEY? undeclared (first use in this function) src/crypto/crypto.c:72: error: ?pkey? undeclared (first use in this function) src/crypto/crypto.c:96: warning: implicit declaration of function ?BIO_new_mem_buf? src/crypto/crypto.c:99: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:100: warning: implicit declaration of function ?PEM_read_bio_PrivateKey? src/crypto/crypto.c:103: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:104: warning: implicit declaration of function ?d2i_PrivateKey_bio? src/crypto/crypto.c:109: warning: implicit declaration of function ?BIO_free? src/crypto/crypto.c:120: warning: implicit declaration of function ?crypto_PKey_New? src/crypto/crypto.c: In function ?crypto_dump_privatekey?: src/crypto/crypto.c:145: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?*? token src/crypto/crypto.c:145: error: ?cipher? undeclared (first use in this function) src/crypto/crypto.c:147: error: ?pem_password_cb? undeclared (first use in this function) src/crypto/crypto.c:147: error: ?cb? undeclared (first use in this function) src/crypto/crypto.c:149: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:149: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:150: error: ?RSA? undeclared (first use in this function) src/crypto/crypto.c:150: error: ?rsa? undeclared (first use in this function) src/crypto/crypto.c:164: warning: implicit declaration of function ?EVP_get_cipherbyname? src/crypto/crypto.c:187: warning: implicit declaration of function ?BIO_new? src/crypto/crypto.c:187: warning: implicit declaration of function ?BIO_s_mem? src/crypto/crypto.c:190: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:191: warning: implicit declaration of function ?PEM_write_bio_PrivateKey? src/crypto/crypto.c:191: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:199: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:200: warning: implicit declaration of function ?i2d_PrivateKey_bio? src/crypto/crypto.c:200: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:204: warning: implicit declaration of function ?EVP_PKEY_get1_RSA? src/crypto/crypto.c:204: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:205: warning: implicit declaration of function ?RSA_print? src/crypto/crypto.c:206: warning: implicit declaration of function ?RSA_free? src/crypto/crypto.c:222: warning: implicit declaration of function ?BIO_get_mem_data? src/crypto/crypto.c: In function ?crypto_load_certificate?: src/crypto/crypto.c:242: error: expected ?)? before ?*? token src/crypto/crypto.c:245: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:245: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:246: error: ?X509? undeclared (first use in this function) src/crypto/crypto.c:246: error: ?cert? undeclared (first use in this function) src/crypto/crypto.c:254: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:255: warning: implicit declaration of function ?PEM_read_bio_X509? src/crypto/crypto.c:258: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:259: warning: implicit declaration of function ?d2i_X509_bio? src/crypto/crypto.c:275: warning: implicit declaration of function ?crypto_X509_New? src/crypto/crypto.c: In function ?crypto_dump_certificate?: src/crypto/crypto.c:294: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:294: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:304: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:305: warning: implicit declaration of function ?PEM_write_bio_X509? src/crypto/crypto.c:305: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c:308: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:309: warning: implicit declaration of function ?i2d_X509_bio? src/crypto/crypto.c:309: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c:313: warning: implicit declaration of function ?X509_print_ex? src/crypto/crypto.c:313: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c: In function ?crypto_load_certificate_request?: src/crypto/crypto.c:349: error: expected ?)? before ?*? token src/crypto/crypto.c:352: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:352: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:353: error: ?X509_REQ? undeclared (first use in this function) src/crypto/crypto.c:353: error: ?req? undeclared (first use in this function) src/crypto/crypto.c:361: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:362: warning: implicit declaration of function ?PEM_read_bio_X509_REQ? src/crypto/crypto.c:365: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:366: warning: implicit declaration of function ?d2i_X509_REQ_bio? src/crypto/crypto.c:382: warning: implicit declaration of function ?crypto_X509Req_New? src/crypto/crypto.c: In function ?crypto_dump_certificate_request?: src/crypto/crypto.c:401: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:401: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:411: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:412: warning: implicit declaration of function ?PEM_write_bio_X509_REQ? src/crypto/crypto.c:412: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c:415: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:416: warning: implicit declaration of function ?i2d_X509_REQ_bio? src/crypto/crypto.c:416: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c:420: warning: implicit declaration of function ?X509_REQ_print_ex? src/crypto/crypto.c:420: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c: In function ?crypto_load_pkcs7_data?: src/crypto/crypto.c:458: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:458: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:459: error: ?PKCS7? undeclared (first use in this function) src/crypto/crypto.c:459: error: ?pkcs7? undeclared (first use in this function) src/crypto/crypto.c:470: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:471: warning: implicit declaration of function ?PEM_read_bio_PKCS7? src/crypto/crypto.c:474: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:475: warning: implicit declaration of function ?d2i_PKCS7_bio? src/crypto/crypto.c:494: warning: implicit declaration of function ?crypto_PKCS7_New? src/crypto/crypto.c: In function ?crypto_load_pkcs12?: src/crypto/crypto.c:510: error: expected ?)? before ?*? token src/crypto/crypto.c:513: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:513: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:514: error: ?PKCS12? undeclared (first use in this function) src/crypto/crypto.c:514: error: ?p12? undeclared (first use in this function) src/crypto/crypto.c:520: warning: implicit declaration of function ?d2i_PKCS12_bio? src/crypto/crypto.c:528: warning: implicit declaration of function ?crypto_PKCS12_New? src/crypto/crypto.c: In function ?crypto_X509?: src/crypto/crypto.c:547: warning: implicit declaration of function ?X509_new? src/crypto/crypto.c: In function ?crypto_X509Name?: src/crypto/crypto.c:568: warning: implicit declaration of function ?crypto_X509Name_New? src/crypto/crypto.c:568: warning: implicit declaration of function ?X509_NAME_dup? src/crypto/crypto.c:568: error: ?crypto_X509NameObj? has no member named ?x509_name? src/crypto/crypto.c: In function ?crypto_X509Req?: src/crypto/crypto.c:586: warning: implicit declaration of function ?X509_REQ_new? src/crypto/crypto.c: In function ?crypto_PKey?: src/crypto/crypto.c:606: warning: implicit declaration of function ?EVP_PKEY_new? src/crypto/crypto.c:606: warning: assignment makes pointer from integer without a cast src/crypto/crypto.c:608: error: ?crypto_PKeyObj? has no member named ?initialized? src/crypto/crypto.c: In function ?crypto_NetscapeSPKI?: src/crypto/crypto.c:653: error: ?NETSCAPE_SPKI? undeclared (first use in this function) src/crypto/crypto.c:653: error: ?spki? undeclared (first use in this function) src/crypto/crypto.c:659: warning: implicit declaration of function ?NETSCAPE_SPKI_b64_decode? src/crypto/crypto.c:661: warning: implicit declaration of function ?NETSCAPE_SPKI_new? src/crypto/crypto.c:667: warning: implicit declaration of function ?crypto_NetscapeSPKI_New? src/crypto/crypto.c: In function ?crypto_X509_verify_cert_error_string?: src/crypto/crypto.c:686: warning: implicit declaration of function ?X509_verify_cert_error_string? src/crypto/crypto.c:686: warning: assignment makes pointer from integer without a cast src/crypto/crypto.c: In function ?locking_function?: src/crypto/crypto.c:728: error: ?CRYPTO_LOCK? undeclared (first use in this function) src/crypto/crypto.c: In function ?init_openssl_threads?: src/crypto/crypto.c:745: warning: implicit declaration of function ?CRYPTO_num_locks? src/crypto/crypto.c:752: warning: implicit declaration of function ?CRYPTO_set_id_callback? src/crypto/crypto.c:753: warning: implicit declaration of function ?CRYPTO_set_locking_callback? src/crypto/crypto.c: In function ?initcrypto?: src/crypto/crypto.c:792: warning: implicit declaration of function ?ERR_load_crypto_strings? src/crypto/crypto.c:793: warning: implicit declaration of function ?OpenSSL_add_all_algorithms? src/crypto/crypto.c:799: error: ?crypto_X509_New? undeclared (first use in this function) src/crypto/crypto.c:800: error: ?crypto_X509Name_New? undeclared (first use in this function) src/crypto/crypto.c:801: error: ?crypto_X509Req_New? undeclared (first use in this function) src/crypto/crypto.c:802: error: ?crypto_X509Store_New? undeclared (first use in this function) src/crypto/crypto.c:803: error: ?crypto_PKey_New? undeclared (first use in this function) src/crypto/crypto.c:805: error: ?crypto_PKCS7_New? undeclared (first use in this function) src/crypto/crypto.c:806: error: ?crypto_NetscapeSPKI_New? undeclared (first use in this function) src/crypto/crypto.c:817: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:818: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:821: error: ?EVP_PKEY_RSA? undeclared (first use in this function) src/crypto/crypto.c:822: error: ?EVP_PKEY_DSA? undeclared (first use in this function) error: Setup script exited with error: command 'gcc' failed with exit status 1 linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1$ python setup.py test running darcsver setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone. running develop error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 13] Permission denied: 'support/lib/python2.5/site-packages/test-easy-install-2287.write-test' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: support/lib/python2.5/site-packages Perhaps your account does not have write access to this directory? If the installation directory is a system-owned directory, you may need to sign in as the administrator or "root" account. If you do not have administrative access to this machine, you may wish to choose a different installation directory, preferably one that is listed in your PYTHONPATH environment variable. For information on other options, you may wish to consult the documentation at: http://peak.telecommunity.com/EasyInstall.html Please make the appropriate changes for your system and try again. linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1$ python setup.py testsudo running darcsver setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone. running develop Not found: tahoe-deps Not found: ../tahoe-deps running egg_info writing requirements to src/allmydata_tahoe.egg-info/requires.txt writing src/allmydata_tahoe.egg-info/PKG-INFO writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' running build_ext Processing setuptools-0.6c12dev.egg removing '/home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packa ges/setuptools-0.6c12dev.egg' (and everything under it) creating /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/setuptools-0.6c12dev.egg Extracting setuptools-0.6c12dev.egg to /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es setuptools 0.6c12dev is already the active version in easy-install.pth Installing easy_install script to support/bin Installing easy_install-2.5 script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/setuptools-0.6c12dev.egg Creating /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/support/lib/python2.5/site-packag es/allmydata-tahoe.egg-link (link to src) allmydata-tahoe 1.4.1 is already the active version in easy-install.pth Installing tahoe script to support/bin Installed /home/linuxmce/Tahoe/allmydata-tahoe-1.4.1/src Processing dependencies for allmydata-tahoe==1.4.1 Searching for pyOpenSSL Reading http://allmydata.org/source/tahoe/deps/tahoe-deps/ Best match: pyOpenSSL 0.9 Downloading http://allmydata.org/source/tahoe/deps/tahoe-deps/pyOpenSSL-0.9.tar.gz Processing pyOpenSSL-0.9.tar.gz Running pyOpenSSL-0.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-qy8Bjn/pyOpenSSL-0.9/egg-dist-tmp-bRYvTH warning: no previously-included files matching '*.pyc' found anywhere in distribution In file included from src/crypto/crypto.h:17, from src/crypto/crypto.c:14: src/crypto/x509.h:17:25: error: openssl/ssl.h: No such file or directory In file included from src/crypto/crypto.h:17, from src/crypto/crypto.c:14: src/crypto/x509.h:27: error: expected specifier-qualifier-list before ?X509? In file included from src/crypto/crypto.h:18, from src/crypto/crypto.c:14: src/crypto/x509name.h:27: error: expected specifier-qualifier-list before ?X509_NAME? In file included from src/crypto/crypto.h:19, from src/crypto/crypto.c:14: src/crypto/netscape_spki.h:24: error: expected specifier-qualifier-list before ?NETSCAPE_SPKI? In file included from src/crypto/crypto.h:20, from src/crypto/crypto.c:14: src/crypto/x509store.h:25: error: expected specifier-qualifier-list before ?X509_STORE? In file included from src/crypto/crypto.h:21, from src/crypto/crypto.c:14: src/crypto/x509req.h:25: error: expected specifier-qualifier-list before ?X509_REQ? In file included from src/crypto/crypto.h:22, from src/crypto/crypto.c:14: src/crypto/pkey.h:27: error: expected specifier-qualifier-list before ?EVP_PKEY? In file included from src/crypto/crypto.h:23, from src/crypto/crypto.c:14: src/crypto/x509ext.h:16:28: error: openssl/x509v3.h: No such file or directory In file included from src/crypto/crypto.h:23, from src/crypto/crypto.c:14: src/crypto/x509ext.h:27: error: expected specifier-qualifier-list before ?X509_EXTENSION? In file included from src/crypto/crypto.h:24, from src/crypto/crypto.c:14: src/crypto/pkcs7.h:15:27: error: openssl/pkcs7.h: No such file or directory In file included from src/crypto/crypto.h:24, from src/crypto/crypto.c:14: src/crypto/pkcs7.h:25: error: expected specifier-qualifier-list before ?PKCS7? In file included from src/crypto/crypto.h:25, from src/crypto/crypto.c:14: src/crypto/pkcs12.h:14:28: error: openssl/pkcs12.h: No such file or directory src/crypto/pkcs12.h:15:26: error: openssl/asn1.h: No such file or directory In file included from src/crypto/crypto.h:26, from src/crypto/crypto.c:14: src/crypto/../util.h:17:25: error: openssl/err.h: No such file or directory In file included from src/crypto/crypto.c:14: src/crypto/crypto.h:75: error: expected ?)? before ?*? token src/crypto/crypto.h:76: error: expected ?)? before ?*? token src/crypto/crypto.h:77: error: expected ?)? before ?*? token src/crypto/crypto.h:78: error: expected ?)? before ?*? token src/crypto/crypto.h:79: error: expected ?)? before ?*? token src/crypto/crypto.h:81: error: expected ?)? before ?*? token src/crypto/crypto.h:82: error: expected ?)? before ?*? token src/crypto/crypto.c: In function ?crypto_load_privatekey?: src/crypto/crypto.c:65: error: expected ?)? before ?*? token src/crypto/crypto.c:69: error: ?pem_password_cb? undeclared (first use in this function) src/crypto/crypto.c:69: error: (Each undeclared identifier is reported only once src/crypto/crypto.c:69: error: for each function it appears in.) src/crypto/crypto.c:69: error: ?cb? undeclared (first use in this function) src/crypto/crypto.c:71: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:71: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:72: error: ?EVP_PKEY? undeclared (first use in this function) src/crypto/crypto.c:72: error: ?pkey? undeclared (first use in this function) src/crypto/crypto.c:96: warning: implicit declaration of function ?BIO_new_mem_buf? src/crypto/crypto.c:99: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:100: warning: implicit declaration of function ?PEM_read_bio_PrivateKey? src/crypto/crypto.c:103: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:104: warning: implicit declaration of function ?d2i_PrivateKey_bio? src/crypto/crypto.c:109: warning: implicit declaration of function ?BIO_free? src/crypto/crypto.c:120: warning: implicit declaration of function ?crypto_PKey_New? src/crypto/crypto.c: In function ?crypto_dump_privatekey?: src/crypto/crypto.c:145: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?*? token src/crypto/crypto.c:145: error: ?cipher? undeclared (first use in this function) src/crypto/crypto.c:147: error: ?pem_password_cb? undeclared (first use in this function) src/crypto/crypto.c:147: error: ?cb? undeclared (first use in this function) src/crypto/crypto.c:149: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:149: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:150: error: ?RSA? undeclared (first use in this function) src/crypto/crypto.c:150: error: ?rsa? undeclared (first use in this function) src/crypto/crypto.c:164: warning: implicit declaration of function ?EVP_get_cipherbyname? src/crypto/crypto.c:187: warning: implicit declaration of function ?BIO_new? src/crypto/crypto.c:187: warning: implicit declaration of function ?BIO_s_mem? src/crypto/crypto.c:190: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:191: warning: implicit declaration of function ?PEM_write_bio_PrivateKey? src/crypto/crypto.c:191: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:199: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:200: warning: implicit declaration of function ?i2d_PrivateKey_bio? src/crypto/crypto.c:200: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:204: warning: implicit declaration of function ?EVP_PKEY_get1_RSA? src/crypto/crypto.c:204: error: ?crypto_PKeyObj? has no member named ?pkey? src/crypto/crypto.c:205: warning: implicit declaration of function ?RSA_print? src/crypto/crypto.c:206: warning: implicit declaration of function ?RSA_free? src/crypto/crypto.c:222: warning: implicit declaration of function ?BIO_get_mem_data? src/crypto/crypto.c: In function ?crypto_load_certificate?: src/crypto/crypto.c:242: error: expected ?)? before ?*? token src/crypto/crypto.c:245: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:245: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:246: error: ?X509? undeclared (first use in this function) src/crypto/crypto.c:246: error: ?cert? undeclared (first use in this function) src/crypto/crypto.c:254: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:255: warning: implicit declaration of function ?PEM_read_bio_X509? src/crypto/crypto.c:258: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:259: warning: implicit declaration of function ?d2i_X509_bio? src/crypto/crypto.c:275: warning: implicit declaration of function ?crypto_X509_New? src/crypto/crypto.c: In function ?crypto_dump_certificate?: src/crypto/crypto.c:294: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:294: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:304: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:305: warning: implicit declaration of function ?PEM_write_bio_X509? src/crypto/crypto.c:305: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c:308: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:309: warning: implicit declaration of function ?i2d_X509_bio? src/crypto/crypto.c:309: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c:313: warning: implicit declaration of function ?X509_print_ex? src/crypto/crypto.c:313: error: ?crypto_X509Obj? has no member named ?x509? src/crypto/crypto.c: In function ?crypto_load_certificate_request?: src/crypto/crypto.c:349: error: expected ?)? before ?*? token src/crypto/crypto.c:352: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:352: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:353: error: ?X509_REQ? undeclared (first use in this function) src/crypto/crypto.c:353: error: ?req? undeclared (first use in this function) src/crypto/crypto.c:361: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:362: warning: implicit declaration of function ?PEM_read_bio_X509_REQ? src/crypto/crypto.c:365: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:366: warning: implicit declaration of function ?d2i_X509_REQ_bio? src/crypto/crypto.c:382: warning: implicit declaration of function ?crypto_X509Req_New? src/crypto/crypto.c: In function ?crypto_dump_certificate_request?: src/crypto/crypto.c:401: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:401: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:411: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:412: warning: implicit declaration of function ?PEM_write_bio_X509_REQ? src/crypto/crypto.c:412: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c:415: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:416: warning: implicit declaration of function ?i2d_X509_REQ_bio? src/crypto/crypto.c:416: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c:420: warning: implicit declaration of function ?X509_REQ_print_ex? src/crypto/crypto.c:420: error: ?crypto_X509ReqObj? has no member named ?x509_req? src/crypto/crypto.c: In function ?crypto_load_pkcs7_data?: src/crypto/crypto.c:458: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:458: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:459: error: ?PKCS7? undeclared (first use in this function) src/crypto/crypto.c:459: error: ?pkcs7? undeclared (first use in this function) src/crypto/crypto.c:470: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:471: warning: implicit declaration of function ?PEM_read_bio_PKCS7? src/crypto/crypto.c:474: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:475: warning: implicit declaration of function ?d2i_PKCS7_bio? src/crypto/crypto.c:494: warning: implicit declaration of function ?crypto_PKCS7_New? src/crypto/crypto.c: In function ?crypto_load_pkcs12?: src/crypto/crypto.c:510: error: expected ?)? before ?*? token src/crypto/crypto.c:513: error: ?BIO? undeclared (first use in this function) src/crypto/crypto.c:513: error: ?bio? undeclared (first use in this function) src/crypto/crypto.c:514: error: ?PKCS12? undeclared (first use in this function) src/crypto/crypto.c:514: error: ?p12? undeclared (first use in this function) src/crypto/crypto.c:520: warning: implicit declaration of function ?d2i_PKCS12_bio? src/crypto/crypto.c:528: warning: implicit declaration of function ?crypto_PKCS12_New? src/crypto/crypto.c: In function ?crypto_X509?: src/crypto/crypto.c:547: warning: implicit declaration of function ?X509_new? src/crypto/crypto.c: In function ?crypto_X509Name?: src/crypto/crypto.c:568: warning: implicit declaration of function ?crypto_X509Name_New? src/crypto/crypto.c:568: warning: implicit declaration of function ?X509_NAME_dup? src/crypto/crypto.c:568: error: ?crypto_X509NameObj? has no member named ?x509_name? src/crypto/crypto.c: In function ?crypto_X509Req?: src/crypto/crypto.c:586: warning: implicit declaration of function ?X509_REQ_new? src/crypto/crypto.c: In function ?crypto_PKey?: src/crypto/crypto.c:606: warning: implicit declaration of function ?EVP_PKEY_new? src/crypto/crypto.c:606: warning: assignment makes pointer from integer without a cast src/crypto/crypto.c:608: error: ?crypto_PKeyObj? has no member named ?initialized? src/crypto/crypto.c: In function ?crypto_NetscapeSPKI?: src/crypto/crypto.c:653: error: ?NETSCAPE_SPKI? undeclared (first use in this function) src/crypto/crypto.c:653: error: ?spki? undeclared (first use in this function) src/crypto/crypto.c:659: warning: implicit declaration of function ?NETSCAPE_SPKI_b64_decode? src/crypto/crypto.c:661: warning: implicit declaration of function ?NETSCAPE_SPKI_new? src/crypto/crypto.c:667: warning: implicit declaration of function ?crypto_NetscapeSPKI_New? src/crypto/crypto.c: In function ?crypto_X509_verify_cert_error_string?: src/crypto/crypto.c:686: warning: implicit declaration of function ?X509_verify_cert_error_string? src/crypto/crypto.c:686: warning: assignment makes pointer from integer without a cast src/crypto/crypto.c: In function ?locking_function?: src/crypto/crypto.c:728: error: ?CRYPTO_LOCK? undeclared (first use in this function) src/crypto/crypto.c: In function ?init_openssl_threads?: src/crypto/crypto.c:745: warning: implicit declaration of function ?CRYPTO_num_locks? src/crypto/crypto.c:752: warning: implicit declaration of function ?CRYPTO_set_id_callback? src/crypto/crypto.c:753: warning: implicit declaration of function ?CRYPTO_set_locking_callback? src/crypto/crypto.c: In function ?initcrypto?: src/crypto/crypto.c:792: warning: implicit declaration of function ?ERR_load_crypto_strings? src/crypto/crypto.c:793: warning: implicit declaration of function ?OpenSSL_add_all_algorithms? src/crypto/crypto.c:799: error: ?crypto_X509_New? undeclared (first use in this function) src/crypto/crypto.c:800: error: ?crypto_X509Name_New? undeclared (first use in this function) src/crypto/crypto.c:801: error: ?crypto_X509Req_New? undeclared (first use in this function) src/crypto/crypto.c:802: error: ?crypto_X509Store_New? undeclared (first use in this function) src/crypto/crypto.c:803: error: ?crypto_PKey_New? undeclared (first use in this function) src/crypto/crypto.c:805: error: ?crypto_PKCS7_New? undeclared (first use in this function) src/crypto/crypto.c:806: error: ?crypto_NetscapeSPKI_New? undeclared (first use in this function) src/crypto/crypto.c:817: error: ?X509_FILETYPE_PEM? undeclared (first use in this function) src/crypto/crypto.c:818: error: ?X509_FILETYPE_ASN1? undeclared (first use in this function) src/crypto/crypto.c:821: error: ?EVP_PKEY_RSA? undeclared (first use in this function) src/crypto/crypto.c:822: error: ?EVP_PKEY_DSA? undeclared (first use in this function) error: Setup script exited with error: command 'gcc' failed with exit status 1 linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1$ cd bin linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1/bin$ ls tahoe-script.template linuxmce at dcerouter:~/Tahoe/allmydata-tahoe-1.4.1/bin$ ------------------------- Robert Metcalf robert at metcarob.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20090730/2eab48e8/attachment-0001.htm From zookog at gmail.com Thu Jul 30 12:13:44 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Thu, 30 Jul 2009 13:13:44 -0600 Subject: [tahoe-dev] Installing on a Ubuntu based machine In-Reply-To: References: Message-ID: The fatal error is this one: On Thu, Jul 30, 2009 at 10:51 AM, wrote: > Searching for pyOpenSSL > Reading http://allmydata.org/source/tahoe/deps/tahoe-deps/ > Best match: pyOpenSSL 0.9 > Downloading > http://allmydata.org/source/tahoe/deps/tahoe-deps/pyOpenSSL-0.9.tar.gz > Processing pyOpenSSL-0.9.tar.gz > Running pyOpenSSL-0.9/setup.py -q bdist_egg --dist-dir > /tmp/easy_install-qy8Bjn/pyOpenSSL-0.9/egg-dist-tmp-bRYvTH > warning: no previously-included files matching '*.pyc' found anywhere in > distribution > In file included from src/crypto/crypto.h:17, > ???????????????? from src/crypto/crypto.c:14: > src/crypto/x509.h:17:25: error: openssl/ssl.h: No such file or directory It is trying to compile pyOpenSSL and failing because the OpenSSL development headers are not installed. You can fix this problem by either installing pyOpenSSL (sudo apt-get install python-pyopenssl) or by installing the OpenSSL development headers (sudo apt-get install libssl-dev). Regards, Zooko From trac at allmydata.org Thu Jul 30 18:32:23 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 31 Jul 2009 01:32:23 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.b073c6f92c6b8b2b382f2bc0dd5b88f6@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.0 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Comment(by warner): yeah, that sounds great. thanks! I failed to get the APT repo updated today. I'll be back online next tuesday or wednesday, and will try to get it done then. Note to self: look into using "mini-dinstall" instead of apt-ftparchive. Also, POX says to look at http://upsilon.cc/~zack/blog/posts/2009/04/howto:_uploading_to_people.d.o_using_dput/ . -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 30 18:43:54 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 31 Jul 2009 01:43:54 -0000 Subject: [tahoe-dev] [tahoe-lafs] #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form In-Reply-To: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> References: <038.900c631d92915b23fb21625e7d9f39c9@allmydata.org> Message-ID: <047.706c47d596202a3729e993af8e401482@allmydata.org> #769: debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form -----------------------+---------------------------------------------------- Reporter: warner | Owner: zooko Type: defect | Status: assigned Priority: critical | Milestone: 1.5.1 Component: packaging | Version: 1.4.1 Keywords: debian | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => 1.5.1 Comment: I think we should announce Tahoe-LAFS v1.5.0 Saturday even if we don't have this ticket ({{{apt-get install allmydata-tahoe}}}) isn't fixed and even if #773 (document installation on Windows) isn't fixed. I highly value those tickets, but this release is just taking way too long and I want to get it over with and move on! We can fix these tickets in the days following the v1.5.0 release. Moving this from "v1.5.0" Milestone to "v1.5.1". -- Ticket URL: tahoe-lafs secure decentralized file storage grid From trac at allmydata.org Thu Jul 30 18:45:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 31 Jul 2009 01:45:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #773: document installation on Windows In-Reply-To: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> References: <037.fc410ded3f694686c217b9dbb13dfc57@allmydata.org> Message-ID: <046.2e8ff4a9c6dd2834d9a57dc8ca375b1d@allmydata.org> #773: document installation on Windows -----------------------+---------------------------------------------------- Reporter: zooko | Owner: somebody Type: task | Status: new Priority: major | Milestone: 1.5.1 Component: packaging | Version: 1.4.1 Keywords: win32 | Launchpad_bug: -----------------------+---------------------------------------------------- Changes (by zooko): * milestone: 1.5.0 => 1.5.1 Comment: I think we should announce Tahoe-LAFS v1.5.0 Saturday even if we don't have this ticket fixed and even if #769 ({{{sudo apt-get install allmydata-tahoe}}}) isn't fixed. I highly value those tickets, but this release is just taking way too long and I want to get it over with and move on! We can fix these tickets in the days following the v1.5.0 release. I'm moving this ticket from "v1.5.0" Milestone to "v1.5.1" Milestone. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From zooko at zooko.com Thu Jul 30 19:05:27 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Thu, 30 Jul 2009 20:05:27 -0600 Subject: [tahoe-dev] v1.5.0 -- now with less apt-get! Re: v1.5.0 is feature-complete! In-Reply-To: References: Message-ID: <13ABAE2A-C79E-42F6-9DA3-F1A03FDA1499@zooko.com> Folks: On 2009-07-20 I thought that Tahoe-LAFS v1.5.0 was ready to roll [1], but then Brian added a ticket into the v1.5.0 Milestone about being able to apt-get the pycryptopp and zfec dependencies and any other dependencies (ticket #769). Then, while we were working on that, I realized that nobody had actually manually tested the install.html process on Windows in a while, and I put out a call for volunteers, and several people stepped up, and sure enough there were several issues with installing Windows, which we have now mostly fixed and/or documented on the mailing list and the trac ticket, but not yet as a user-facing doc on the wiki (ticket #773). It just now turned out that Brian isn't going to have time to fix the allmydata.org-hosted apt repositories until next Tuesday or Wednesday (see his comment on ticket #769). I don't think that I can wait that long to announce Tahoe-LAFS v1.5.0! Let's announce its existence, and people who want to try it can follow the install.html process [2], and then we can continue to improve the issues in these tickets (neither of which actually require any changes to the Tahoe-LAFS source code itself). If this is not horribly objectionable to anyone, then I'll plan to finish the relnotes.txt (i.e. the "press release") and send it out far and wide this coming Saturday, the 1st of August. I'll also update http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages to explain that the current release can't be apt-get installed. In the meantime, if anybody else knows how to build an apt repository (possibly by using Ubuntu PPAs?), please let us know -- our buildbots are providing all the .deb packages that we need (I think) -- we just need an apt repository to serve them. Regards, Zooko [1] http://allmydata.org/pipermail/tahoe-dev/2009-July/002347.html # see also the message quoted below [2] http://allmydata.org/source/tahoe/trunk/docs/install.html tickets mentioned in this message: http://allmydata.org/trac/tahoe/ticket/769 # debian 'sid' package fails to run: pycryptopp-0.5.15 not available in debian form http://allmydata.org/trac/tahoe/ticket/773 # organize the documentation for installation on Windows On Monday,2009-07-20, at 8:17 , Zooko Wilcox-O'Hearn wrote: > Thanks to Kevan Carstensen for implementing and Brian Warner for > reviewing, the last issue, ticket #705, has been resolved. > > There are no more open tickets in the v1.5.0 Milestone, and I plan > to defer any patches which change the code until after the 1.5 > release. Any further patches will just be to docs (e.g. the > relnotes.txt file). > > So, test it out! Upgrade your grids to the current trunk and > report any anomalies! I'll write up Release Notes and do the whole > how_to_make_a_tahoe_release process [1] as my day-job schedule > permits, so probably some morning this week before my children wake > up and need breakfast and help with their chores. :-) > > Regards, > > Zooko > > [1] http://allmydata.org/trac/tahoe/browser/docs/ > how_to_make_a_tahoe_release.txt > > tickets mentioned in this e-mail: > > http://allmydata.org/trac/tahoe/ticket/705 # "tahoe mv" unlinks the > target even when it is a directory From warner at lothar.com Thu Jul 30 23:08:28 2009 From: warner at lothar.com (Brian Warner) Date: Thu, 30 Jul 2009 23:08:28 -0700 Subject: [tahoe-dev] v1.5.0 -- now with less apt-get! Re: v1.5.0 is feature-complete! In-Reply-To: <13ABAE2A-C79E-42F6-9DA3-F1A03FDA1499@zooko.com> References: <13ABAE2A-C79E-42F6-9DA3-F1A03FDA1499@zooko.com> Message-ID: <4A728A5C.3030408@lothar.com> Zooko Wilcox-O'Hearn wrote: > In the meantime, if anybody else knows how to build an apt repository > (possibly by using Ubuntu PPAs?), please let us know -- our buildbots > are providing all the .deb packages that we need (I think) -- we just > need an apt repository to serve them. Unfortunately, I don't think this is quite true. The directory where we're currently uploading debs (our our internal machine named "hanford") has the following recent Tahoe debs (note that r4031 is the current trunk, but for whatever reason the cygwin buildslave is offline and that's preventing our buildmaster from making debs): buildslave at hanford:~/tahoe-debs$ find . -name '*allmydata-tahoe_1.4.1-r4026*' ./dists/etch/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb ./dists/feisty/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb ./dists/edgy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb ./dists/gutsy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb ./dists/hardy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb ./dists/hardy/tahoe/binary-amd64/allmydata-tahoe_1.4.1-r4026_all.deb But only the following suitable pycryptopp debs: buildslave at hanford:~/tahoe-debs$ find . -name '*pycryptopp_0.5.16*' ./dists/sid/main/binary-i386/python-pycryptopp_0.5.16-1_i386.deb ./dists/jaunty/main/binary-amd64/python-pycryptopp_0.5.16-1_amd64.deb ./dists/lenny/main/binary-i386/python-pycryptopp_0.5.16-1_i386.deb (also note that we are not yet indexing the jaunty or lenny debs: this is the task I'll try to finish next week). These sets are entirely disjoint! There is no (debian/ubuntu release) + (architecture) combination for which you can get both a correct tahoe deb and a pycryptopp deb. What releases should we target? Really, I'd like us to offer pycryptopp debs and tahoe debs for the same platforms: e.g. the gutsy tahoe deb is useless without a gutsy pycryptopp deb. But to prioritize just a few: Jaunty is the most recent Ubuntu release, Hardy is the most recent Ubuntu LTS (long-term-support) release, and Lenny is the most recent Debian release, so those three are arguably the most important to support. I'll generally prioritize i386 over amd64, noting that tahoe itself is pure-python (and it's only our poor APT-repo-management setup that causes an "_all.deb" to get placed in a binary-i386/ directory), so amd64 users can install a tahoe deb from an i386 directory. Note that in general, you can probably use an older deb on a newer system, but not vice versa. Using a lenny (debian) deb on a jaunty (ubuntu) system or vice versa are both pretty uncertain: there is not a total ordering between debian releases and ubuntu releases. The best recommendations we can currently make are: Jaunty/amd64: probably workable: use the hardy tahoe deb and the jaunty pycryptopp deb [requires APT-repo indexing to be done] Jaunty/i386: fail: we have a hardy tahoe deb, but no pycryptopp deb (the sid is too new, the lenny is debian-vs-ubuntu) Hardy/i386: fail: we have a hardy tahoe deb, but no pycryptopp deb Hardy/amd64: fail: we have a hardy tahoe deb, but no pycryptopp deb Lenny/i386: dubious: we have the right pycryptopp, but best tahoe deb we can offer is from etch [also requires APT-repo indexing to be done] Lenny/amd64: fail: we don't have good debs for either tahoe or pycryptopp So, none of these platforms will work right now, and even once I finish APT-repo indexing task, we'll only provide marginal support for two out of the six targets (jaunty/amd64 and lenny/i386). To support these platforms properly, we need to create debs for: tahoe: lenny/i386, lenny/amd64, jaunty/i386, jaunty/amd64 pycryptopp: lenny/amd64, hardy/i386, hardy/amd64, jaunty/i386 Since hardy debs should be installable on jaunty (and since hardy is the latest LTS), and since lenny users can probably install etch debs, it would be enough to build: tahoe: lenny/amd64 pycryptopp: lenny/amd64, hardy/i386, hardy/amd64 allmydata.com has a host to provide hardy/i386 (deharo1), but we don't yet have jaunty or lenny boxes, or very many amd64 hosts. If Zooko doesn't do it before I get back, I'll copy a pycryptopp tarball over to deharo1 and manually build a deb and copy it into our APT repo. That will at least give us an answer for our two most important debian-ish platforms: hardy/i386 and jaunty/i386. For the future, I've been experimenting with pbuilder/schroot on an EC2 instance to create multiple buildslaves (one per supported release: from dapper on up to jaunty and maybe the latest "karmic" snapshot) at the same time, but it's likely to be at least a month before I've got that stuff working well enough to be useful. (I'm also hoping to implement automated installation testing with the same framework). When that's in place, if I can find the time, I hope to clean up our debian support significantly and offer real packages for all platforms. At that point, I'll see if this could help with the pycryptopp packages too. Also, we should probably create a second APT repo: one for the per-commit development debs, and a separate one that contains only the release debs. Folks who want to live on the bleeding edge can pull from the dev repo, but folks who just want to easily automatically update to the latest official release can point at the relase-repo. I think the Makefile rules that upload a deb should look at the version number to decide whether a deb gets uploaded to the release-repo or not. cheers, -Brian From zooko at zooko.com Fri Jul 31 06:01:18 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Fri, 31 Jul 2009 07:01:18 -0600 Subject: [tahoe-dev] shared fate policy on buildbot Re: v1.5.0 -- now with less apt-get! Re: v1.5.0 is feature-complete! In-Reply-To: <4A728A5C.3030408@lothar.com> References: <13ABAE2A-C79E-42F6-9DA3-F1A03FDA1499@zooko.com> <4A728A5C.3030408@lothar.com> Message-ID: <5BED90D6-6F5C-46F7-ABC3-9D80742764B7@zooko.com> On Friday,2009-07-31, at 0:08 , Brian Warner wrote: > (note that r4031 is the current trunk, but for whatever reason the > cygwin buildslave is offline and that's preventing our buildmaster > from making debs): I guess I was doing Windows Update or cygwin upgrade or something and got called away and forgot to restart it. I restarted the cygwin buildslave and it passed unit tests and the .deb's were built. By the way, I really like the "shared fate policy" on the automatic building of packages. That policy [1] is that packages don't get built for any platforms unless unit tests pass on all supported platforms. That way when you download a package, you know you are getting a version that passed all tests on all platforms, not just on your platform. It also means developers are forced to pay attention to all supported platforms, and not just their preferred platform. For example, I might not have noticed the cygwin buildslave being off- line if you hadn't mentioned it, and you might not have mentioned it if it wasn't preventing the .deb's from being built. If this causes too much hassle, then that probably means we need to remove some platforms (such as cygwin) from the "Supported" category (or get more developers to contribute to the underloved platforms). As long as the set of platforms marked as "Supported" on the web pages really is the set of platforms that can cause developers to stop doing other things and pay attention when they fail, then I'm satisfied. Regards, Zooko [1] http://allmydata.org/trac/tahoe/wiki/BuildbotPolicy From zookog at gmail.com Fri Jul 31 09:21:06 2009 From: zookog at gmail.com (Zooko O'Whielacronx) Date: Fri, 31 Jul 2009 10:21:06 -0600 Subject: [tahoe-dev] AES-256 is looking worse Message-ID: Folks: I've already been investigating for a long time the possibility of switching from AES-256 to XSalsa20 for future versions of Tahoe-LAFS. Today's announcement that AES-256 is weaker than we previously thought makes the issue more urgent. Here's a blog entry I just posted about this (also appended): http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html Regards, Zooko ------- begin appended blog entry Wow! Cryptographers have devised even more effective ways to crack a weakened variant of ~AES-256: [[Schneier's blog entry|http://www.schneier.com/blog/archives/2009/07/another_new_aes.html]]. This doesn't mean that anyone who is current relying on AES is vulnerable, but it does increase the likelihood that in the future someone will come up with a way to crack the full-strength AES. This means that for long-term storage (such as in [[the Tahoe-LAFS storage system|http://allmydata.org]]), it might be better to encrypt with a stronger cipher such as Salsa20 (actually ~XSalsa20, which is just Salsa20 with a larger initialization vector) or, as Bruce Schneier suggests, AES with extra rounds. It is ironic that ~AES-256 is the only algorithm approved for TOP SECRET by the U.S. government (~AES-128 is approved for SECRET but not for TOP SECRET). ~AES-256 has now been revealed as being weaker than ~AES-128. The other issue is that large-scale practical quantum computers (if they existed) could crack any cipher with a mere 128-bit key, but not a good cipher with a 256-bit key. This might mean that ~AES-256 would be vulnerable if there were a sufficiently powerful quantum computer. That would mean there is now no encryption algorithm which is both secure against quantum computers and approved by the U.S. government for TOP SECRET. I was recently pondering whether the next iteration of ~Tahoe-LAFS should switch from ~AES-256 to ~XSalsa20. The benefits I was considering were that ~XSalsa20 is probably more secure than ~AES-256 (see [[the Tahoe-LAFS Bibliography|http://allmydata.org/trac/tahoe/wiki/Bibliography]], especially the practical issue of side-channel attacks) and is certainly much faster. The drawbacks were that ~XSalsa20 is newer and less widely studied and that it wasn't approved for U.S. government usage. This new attack on ~AES-256 makes my dilemma all the more pointed. From trac at allmydata.org Fri Jul 31 09:21:20 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 31 Jul 2009 16:21:20 -0000 Subject: [tahoe-dev] [tahoe-lafs] #777: Facility to automatically renew leases of root caps Message-ID: <038.3f89dc77b23b17350bb67481c8f41743@allmydata.org> #777: Facility to automatically renew leases of root caps ---------------------------+------------------------------------------------ Reporter: kpreid | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-frontend | Version: unknown Keywords: leases | Launchpad_bug: ---------------------------+------------------------------------------------ There should be a facility to automatically renew the leases of (and repair) every cap in your tahoe aliases, better than this: {{{ for item in `tahoe list-aliases | cut -f 1 -d :`; do echo ---$item---; tahoe deep-check --repair --add-lease $item:; done }}} In particular, there should be a command-line command to do it, suitable for executing from cron, *and* there should be an option to have your tahoe client node persistent process do it on a schedule, for whichever people find convenient. Having easy auto-renew seems to me to be important; many people won't find it simple to set it up on their own, and they may forget and "..hey, where's my files?" -- Ticket URL: tahoe-lafs secure decentralized file storage grid From jbglaw at lug-owl.de Fri Jul 31 09:54:03 2009 From: jbglaw at lug-owl.de (Jan-Benedict Glaw) Date: Fri, 31 Jul 2009 18:54:03 +0200 Subject: [tahoe-dev] AES-256 is looking worse In-Reply-To: References: Message-ID: <20090731165403.GH24150@lug-owl.de> On Fri, 2009-07-31 10:21:06 -0600, Zooko O'Whielacronx wrote: > Folks: > > I've already been investigating for a long time the possibility of > switching from AES-256 to XSalsa20 for future versions of Tahoe-LAFS. > Today's announcement that AES-256 is weaker than we previously thought > makes the issue more urgent. Here's a blog entry I just posted about > this (also appended): > > http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html In case of fire, maybe it would be feasible to extend the a repair with a writecap to do additional re-crypting? Another question is: Will a switch be a real switch? Or just allow an additional crypto alg? MfG, JBG -- Jan-Benedict Glaw jbglaw at lug-owl.de +49-172-7608481 Signature of: The course of history shows that as a government grows, liberty the second : decreases." (Thomas Jefferson) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20090731/2125d4f1/attachment.pgp From warner at lothar.com Fri Jul 31 10:20:03 2009 From: warner at lothar.com (Brian Warner) Date: Fri, 31 Jul 2009 10:20:03 -0700 Subject: [tahoe-dev] AES-256 is looking worse In-Reply-To: <20090731165403.GH24150@lug-owl.de> References: <20090731165403.GH24150@lug-owl.de> Message-ID: <4A7327C3.5060908@lothar.com> Jan-Benedict Glaw wrote: > > In case of fire, maybe it would be feasible to extend the a repair > with a writecap to do additional re-crypting? Hm, yeah, that'd be feasible. Doing "cp -r" would probably be more straightforward, though, maybe with an option to prohibit re-use of immutable filecaps (full byte-wise copy of everything). > Another question is: Will a switch be a real switch? Or just allow > an additional crypto alg? It would probably come in the form of a new filecap format ("URI:CHK2"?) which is defined to either contain an algorithm field (URI:CHK2:AES256: or URI:CHK2:XSALSA20:), or which just always uses the new algorithm. New uploads would use the new format, but existing files in both formats would continue to be readable. Maybe we'd provide a switch to let you intentionally upload files in the old format (if you want your files to be readable by older clients), but more likely not. cheers, -Brian From zooko at zooko.com Fri Jul 31 10:54:47 2009 From: zooko at zooko.com (Zooko Wilcox-O'Hearn) Date: Fri, 31 Jul 2009 11:54:47 -0600 Subject: [tahoe-dev] AES-256 is looking worse In-Reply-To: <4A7327C3.5060908@lothar.com> References: <20090731165403.GH24150@lug-owl.de> <4A7327C3.5060908@lothar.com> Message-ID: <16E164FD-3E2E-4C3F-B040-A022D7DADB08@zooko.com> On Friday,2009-07-31, at 11:20 , Brian Warner wrote: > It would probably come in the form of a new filecap format > ("URI:CHK2"?) which is defined to either contain an algorithm field > (URI:CHK2:AES256: or URI:CHK2:XSALSA20:), or which just always uses > the new algorithm. New uploads would use the new format, but > existing files in both formats would continue to be readable. Maybe > we'd provide a switch to let you intentionally upload files in the > old format (if you want your files to be readable by older > clients), but more likely not. Ah, actually I had the opposite plan -- make it as easy as possible to deploy the new Tahoe-LAFS version while interoperating with users of older versions, which means make it very easy (probably even the default) to produce the old-style caps. I think by the time the new caps are ready (six months from now ?), Tahoe-LAFS will have substantial user bases with substantial amount of stored files, etc., and it will be very useful to be able to tell people "Upgrade to Tahoe-LAFS v2.0, and everything will continue to work, with no configuration or change in behavior on your part, and even though the other people on your grid are still using Tahoe-LAFS v1.6.". I've been planning to write up a document about our compatibility plan for such upgrades. Regardless of the details though, everyone reading this can not-worry about compatibility. The current format is secure, functional, performs well and will be carefully supported for the forseeable future. When we introduce new data formats or other changes, we will be sure that it is easy to upgrade without requiring a lot of effort on your part and without requiring everyone that you work with to upgrade on the same day. We've already done this a few times with Tahoe-LAFS, and you haven't noticed. ;-) Regards, Zooko From warner at lothar.com Fri Jul 31 11:35:49 2009 From: warner at lothar.com (Brian Warner) Date: Fri, 31 Jul 2009 11:35:49 -0700 Subject: [tahoe-dev] Tahoe FUSE In-Reply-To: <87iqhar2xj.fsf@gnu.org> References: <20090730055717.GA24150@lug-owl.de> <87iqhar2xj.fsf@gnu.org> Message-ID: <4A733985.4040505@lothar.com> Ludovic Court?s wrote: > Perhaps the Twisted patch should be pushed more insistently, and a > workaround for Tahoe could be found in the meantime? (Unfortunately, > I'm not volunteering for any of these...) For reference, the patch is in the Twisted bugtracker at http://twistedmatrix.com/trac/ticket/3462 , and has been commented-upon favorably, but lacks specific unit tests that would let it pass review and get committed. When I wrote that code a year ago, I was unable to find a clean workaround from the Tahoe side.. it would have required a great deal of monkeypatching and been very fragile. I'd be surprised if anything has changed to improve that since then. cheers, -Brian From warner at lothar.com Fri Jul 31 13:36:53 2009 From: warner at lothar.com (Brian Warner) Date: Fri, 31 Jul 2009 13:36:53 -0700 Subject: [tahoe-dev] cleversafe says: 3 Reasons Why Encryption is Overrated In-Reply-To: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> References: <5595A87F-5097-462F-BD8B-72B52D4DDD38@zooko.com> Message-ID: <4A7355E5.5040803@lothar.com> Zooko Wilcox-O'Hearn wrote: > > Cleversafe has posted a series of blog entries entitled "3 Reasons > Why Encryption is Overrated". The AONT is a neat bit of crypto, but it seems to me that it's merely moving the problem elsewhere, not avoiding it completely. From what I can see of the Cleversafe architecture (i.e. the picture on http://dev.cleversafe.org/weblog/?p=111), they've got a system in which someone who can retrieve "k" shares can reconstruct the plaintext, so all of the policy around who-gets-to-read-what must be enforced by the servers. They express this policy when they decide whether or not to honor a request for one of those shares. The security of your file depends upon enough (N-k+1) of them deciding to say "no" to the "wrong" people; the availability of your file depends upon at least k of them saying "yes" to the "right" people. The means by which those servers make that decision is not described in the picture, but is a vital part of the access-control scheme (and probably depends upon the same sort of public-key encryption that the blog entry disparages). I'd want to know more about how this undocumented access-control scheme works before I was willing to believe that the Cleversafe approach results in "fewer headaches". It is clear that the access-control policy can be changed without re-encrypting the original data, but depending upon how access is granted, it may be an even bigger hassle to change the access keys. I suspect there is some notion of a "user", represented by a private key, and that each server has a typical (user * file) ACL matrix, and that some superuser retains the ability to manipulate that matrix. The object-capability community has shown (http://waterken.sourceforge.net/aclsdont/) that this approach breaks down when trying to make sharing decisions among multiple users, and quickly becomes vulnerable to things like the Confused Deputy attack. Another possibility is that there is some master node which makes the per-user access-control decisions, and all servers honor its requests. In this case the security of the files depends upon both the servers being able to distinguish the master's requests from an attacker's, and upon the master being able to distinguish one user's requests from another's. The issues with ACLs still apply, and both availability and security depend upon the master (which is now a single point of failure). In Tahoe, we made a deep design decision very early: the security of the system should not depend upon the behavior of the servers. We made that choice so that users could take advantage of storage space on a wider variety of servers, including ones that you don't trust to not peek inside your shares. Tahoe shares (and the ciphertext of every file) are considered to be public knowledge. This fact improves system-wide reliability by enabling the use of additional tools without affecting any security properties: * servers can independently copy shares to new servers when they know they're going to leave the grid * pairs of servers can be "buddies" for a single share (and challenge each other with integrity checks at random intervals) * arbitrary parties can perform repair functions and generate new shares without seeing the plaintext * "helpers" can safely perform the erasure coding/decoding for you, to offload bandwidth and CPU effort to a more suitable machine * third-party "relay servers" can transport and hold shares (e.g. for the benefit of servers stuck behind NAT boxes) * users can upload shares to anyone that seems useful: the only consequence is the bandwidth cost The actual security properties we get in Tahoe are: * a small number of misbehaving servers can do absolutely nothing to hurt you * a larger number of misbehaving servers can cause lazy readers (those who do not do a thorough check) to get old versions of mutable files, known as a rollback attack * a very large number of misbehaving servers can cause unavailability of files, and rollback attacks even against thorough readers where "very large" means N-k or more, and "small" means less than about 2*k (this depends upon the reader: they make a tradeoff between bandwidth expended and vulnerability to rollback). Also, a reader who is somehow able to remember the latest sequence number will never be vulnerable to the rollback attack. And of course the rollback attack is not even applicable to immutable files, which have only a single "version". Note that confidentiality and integrity (the lack of undetected bitflips) are guaranteed even if every single server is colluding against you, assuming the crypto primitives remain unbroken. We decided that this was the most important property to achieve. Anything less means you're vulnerable to server behavior, either that of a single server or of a colluding group. The Sybil attack demonstrates that there is no way to prove that your servers are independent, so it is hard to have confidence in any property that depends upon non-collusion. (Also, if all of your servers are managed by the same company, then by definition they're colluding. We wanted to give allmydata.com users the ability to not share their file's plaintext with allmydata.com) By treating the ciphertext of any given file to be public knowledge, we concentrate all of the security of the system into the encryption key. This drastically simplifies the access-control policy, which can be stated in one sentence: you can read a file's contents if and only if you know or can obtain its readcap. There is no superuser who gets to make access-control decisions, no ACL matrices to be updated or consulted, no deputies to be confused, and sharing files is as easy as sharing the readcap. Finally, in all of this, it's important to be clear about the differences between mutable and immutable files, and the differences between symmetric and asymmetric encryption. Much of the Cleversafe blog posts talk about "encryption" being bad and the AONT being the good alternative, but of course the AONT that they depend upon is based on symmetric AES-256. I think they're really trying to say that asymmetric encryption (RSA/ECDSA) is threatened by quantum computing, and that per-file keys are either weak or hard to manage. Any notion of mutable versus immutable files must be expressed in the (undocumented) server access control mechanisms. In Tahoe, our immutable files use AES and SHA256d, and if you have a secure place to store the filecaps, you don't need anything else. (this layer of Tahoe can be construed as a secure DHT with filecaps as keys and file plaintext as values). As the saying goes, cryptography is a means of turning a large security problem into a small security problem: you still need to keep the filecaps safe, but they're a lot smaller than the original files. If you want to keep track of fewer things, you can store those filecaps in mutable directories, which are based upon mutable files that use AES, SHA256d, and RSA2048. The use of an asymmetric algorithm like RSA makes them vulnerable to more sorts of attacks, but their mutability makes them far more flexible, allowing you to keep track of an arbitrary number of changing files with a single 256ish-bit cryptovalue. cheers, -Brian From trac at allmydata.org Fri Jul 31 13:48:11 2009 From: trac at allmydata.org (tahoe-lafs) Date: Fri, 31 Jul 2009 20:48:11 -0000 Subject: [tahoe-dev] [tahoe-lafs] #777: Facility to automatically renew leases of root caps In-Reply-To: <038.3f89dc77b23b17350bb67481c8f41743@allmydata.org> References: <038.3f89dc77b23b17350bb67481c8f41743@allmydata.org> Message-ID: <047.4bba4a9d70c438805d4472e6c6b346cc@allmydata.org> #777: Facility to automatically renew leases of root caps ---------------------------+------------------------------------------------ Reporter: kpreid | Owner: Type: enhancement | Status: new Priority: major | Milestone: undecided Component: code-frontend | Version: unknown Keywords: leases | Launchpad_bug: ---------------------------+------------------------------------------------ Comment(by warner): How many aliases do you have? What about creating a master directory that contains all of them, and then doing deep-check on an alias that points to that? (*something* has to be the starting point.. I generally only have one alias per grid) I agree that the node should have a built-in renewal mechanism. I've wondered how to configure/manage it: it should be possible to check on the status/progress of the renewal process through the webapi, including finding out which files had problems or needed repair, but these status pages should not be reachable without knowing the secret alias values. Maybe a tahoe.cfg section that lists aliases which should be auto-checked, and then any WUI directory page which matches one of the alias values should include a link that says "a deep-repair process is working on this directory and its children, click here for progress". And/or a page which lists all the deep-repair processes currently running, with alias names (but not dircaps) and general information about their progress (but no access to pages with filecaps or dircaps), and instructions to type "tahoe webopen ALIASNAME:" for more details. -- Ticket URL: tahoe-lafs secure decentralized file storage grid From bogus@does.not.exist.com Wed Jul 1 18:26:42 2009 From: bogus@does.not.exist.com () Date: Thu, 02 Jul 2009 01:26:42 -0000 Subject: No subject Message-ID: same size as the compiler's size_t type, but is signed. It will be a typedef for ssize_t where available." Because there's an assertion in the wrapper code that the size is non-negative, a cast to size_t will never alter the semantic value, *when* that definition is used. However, the second hit [3] discusses the proper way to write extension modules that are backwards compatible with CPython 2.5, and I though I didn't follow it thoroughly, I have the impression that on pre-2.5, there could be problems here and elsewhere. Does Tahoe-LAFS intend to support pre-2.5 CPython? [1] http://www.google.com/search?q=3Dpy_ssize_t [2] http://www.python.org/dev/peps/pep-0353/ [3] http://bugs.python.org/issue1485576 On Tue, Sep 8, 2009 at 11:39 PM, pycryptopp wrote: > #19: Segmentation fault in HashMultipleBlocks > ---------------------+---------------------------------------------------= --- > Reporter: =A0francois =A0| =A0 =A0 =A0 =A0 =A0 Owner: =A0francois > =A0 =A0Type: =A0defect =A0 =A0| =A0 =A0 =A0 =A0 =A0Status: =A0new > Priority: =A0major =A0 =A0 | =A0 =A0 =A0 =A0 Version: =A00.5.1 > Keywords: =A0 =A0 =A0 =A0 =A0 =A0| =A0 Launchpad_bug: > ---------------------+---------------------------------------------------= --- > > Comment(by warner): > > =A0Nice catch! I just pushed [669] with your fix: set PY_SSIZE_T_CLEAN > =A0everywhere, and update the two remaining uses of "int"-sized {{{t#}}} > =A0length variables to be Py_ssize_t instead. The segfaults go away, and > =A0tests pass on linux/32bit, OS-X/10.6/64bit, OS-X/10.5/probably-64-bit. > > =A0I didn't touch that implicit cast. If your analysis points out a probl= em, > =A0give a yell. > > =A0francois: could you play with this patch and see if it makes your segf= ault > =A0go away? I know that it happens infrequently enough that it'll be hard= to > =A0develop confidence by merely running the test suite a few times, but m= aybe > =A0if you observed no crash for three times as long as the usual time it > =A0takes to crash, we could treat this one as done.. > > -- > Ticket URL: > pycryptopp > Python bindings for the Crypto++ library > _______________________________________________ > tahoe-dev mailing list > tahoe-dev at allmydata.org > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev > From bogus@does.not.exist.com Wed Jul 1 18:26:42 2009 From: bogus@does.not.exist.com () Date: Thu, 02 Jul 2009 01:26:42 -0000 Subject: No subject Message-ID: included in the next release of Ubuntu. http://ubuntupodcast.net/2009/09/04/ubuntu-podcast-quickie-11 Regards, Zooko From bogus@does.not.exist.com Wed Jul 1 18:26:42 2009 From: bogus@does.not.exist.com () Date: Thu, 02 Jul 2009 01:26:42 -0000 Subject: No subject Message-ID: application will only accept update bundles which are signed by a DSA privkey that matches a pubkey embedded in the app. It'd be nice if Firefox could do the same. And if Firefox were to establish a quietly-backwards-compatible convention (i.e. the hash-mark trick) for strong URL-based authentication of HTTP resources, then other applications could start using it too, and a significant class of current web security problems (like the mixed-content one where an HTTPS page loads a javascript library via HTTP) could be fixed. cheers, -Brian [1]: http://sparkle.andymatuschak.org/documentation/pmwiki.php/Documentation/BasicSetup