Wed Jan 27 06:44:30 GMT Standard Time 2010 david-sarah@jacaranda.org * Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. Wed Jan 27 07:03:09 GMT Standard Time 2010 david-sarah@jacaranda.org * Miscellaneous documentation, test, and code formatting tweaks. Wed Jan 27 23:06:42 GMT Standard Time 2010 david-sarah@jacaranda.org * Address comments by Kevan on 833 and add test for stripping spaces New patches: [Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. david-sarah@jacaranda.org**20100127064430 Ignore-this: 5ef6a3554cf6bef0bf0712cc7d6c0252 ] { hunk ./contrib/fuse/impl_c/blackmatch.py 4 #!/usr/bin/env python #----------------------------------------------------------------------------------------------- -from allmydata.uri import CHKFileURI, DirectoryURI, LiteralFileURI +from allmydata.uri import CHKFileURI, DirectoryURI, LiteralFileURI, is_literal_file_uri from allmydata.scripts.common_http import do_http as do_http_req from allmydata.util.hashutil import tagged_hash from allmydata.util.assertutil import precondition hunk ./contrib/fuse/impl_c/blackmatch.py 338 self.fname = self.tfs.cache.tmp_file(os.urandom(20)) if self.fnode is None: log('TFF: [%s] open() for write: no file node, creating new File %s' % (self.name, self.fname, )) - self.fnode = File(0, 'URI:LIT:') + self.fnode = File(0, LiteralFileURI.BASE_STRING) self.fnode.tmp_fname = self.fname # XXX kill this self.parent.add_child(self.name, self.fnode, {}) elif hasattr(self.fnode, 'tmp_fname'): hunk ./contrib/fuse/impl_c/blackmatch.py 365 self.fname = self.fnode.tmp_fname log('TFF: reopening(%s) for reading' % self.fname) else: - if uri.startswith("URI:LIT") or not self.tfs.async: + if is_literal_file_uri(uri) or not self.tfs.async: log('TFF: synchronously fetching file from cache for reading') self.fname = self.tfs.cache.get_file(uri) else: hunk ./contrib/fuse/impl_c/blackmatch.py 1240 def get_file(self, uri): self.log('get_file(%s)' % (uri,)) - if uri.startswith("URI:LIT"): + if is_literal_file_uri(uri): return self.get_literal(uri) else: return self.get_chk(uri, async=False) hunk ./docs/frontends/webapi.txt 153 === Child Lookup === -Tahoe directories contain named children, just like directories in a regular -local filesystem. These children can be either files or subdirectories. +Tahoe directories contain named child entries, just like directories in a regular +local filesystem. These child entries, called "dirnodes", consist of a name, +metadata, a write slot, and a read slot. The write and read slots normally contain +a write-cap and read-cap referring to the same object, which can be either a file +or a subdirectory. The write slot may be empty (actually, both may be empty, +but that is unusual). If you have a Tahoe URL that refers to a directory, and want to reference a named child inside it, just append the child name to the URL. For example, if hunk ./docs/frontends/webapi.txt 397 } } } ] } + For forward-compatibility, a mutable directory can also contain caps in + a format that is unknown to the webapi server. When such caps are retrieved + from a mutable directory in a "ro_uri" field, they will be prefixed with + the string "ro.", indicating that they must not be decoded without + checking that they are read-only. The "ro." prefix must not be stripped + off without performing this check. (Future versions of the webapi server + will perform it where necessary.) + + If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT, + and the webapi server recognizes the rw_uri as a write cap, then it will + reset the ro_uri to the corresponding read cap and discard the original + contents of ro_uri (in order to ensure that the two caps correspond to the + same object and that the ro_uri is in fact read-only). However this may not + happen for caps in a format unknown to the webapi server. Therefore, when + writing a directory the webapi client should ensure that the contents + of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent + (write cap, read cap) pair if possible. If the webapi client only has + one cap and does not know whether it is a write cap or read cap, then + it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The + client must not put a write cap into a "ro_uri" field. + Note that the webapi-using client application must not provide the "Content-Type: multipart/form-data" header that usually accompanies HTML form submissions, since the body is not formatted this way. Doing so will hunk ./docs/frontends/webapi.txt 432 Like t=mkdir-with-children above, but the new directory will be deep-immutable. This means that the directory itself is immutable, and that - it can only contain deep-immutable objects, like immutable files, literal - files, and deep-immutable directories. A non-empty request body is - mandatory, since after the directory is created, it will not be possible to - add more children to it. + it can only contain objects that are treated as being deep-immutable, like + immutable files, literal files, and deep-immutable directories. + + For forward-compatibility, a deep-immutable directory can also contain caps + in a format that is unknown to the webapi server. When such caps are retrieved + from a deep-immutable directory in a "ro_uri" field, they will be prefixed + with the string "imm.", indicating that they must not be decoded without + checking that they are immutable. The "imm." prefix must not be stripped + off without performing this check. (Future versions of the webapi server + will perform it where necessary.) + + The cap for each child may be given either in the "rw_uri" or "ro_uri" + field of the PROPDICT (not both). If a cap is given in the "rw_uri" field, + then the webapi server will check that it is an immutable read-cap of a + *known* format, and give an error if it is not. If a cap is given in the + "ro_uri" field, then the webapi server will still check whether known + caps are immutable, but for unknown caps it will simply assume that the + cap can be stored, as described above. Note that an attacker would be + able to store any cap in an immutable directory, so this check when + creating the directory is only to help non-malicious clients to avoid + accidentally giving away more authority than intended. + + A non-empty request body is mandatory, since after the directory is created, + it will not be possible to add more children to it. POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir hunk ./docs/frontends/webapi.txt 462 Create new directories as necessary to make sure that the named target ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional - intermediate directories as necessary. If the named target directory already - exists, this will make no changes to it. + intermediate mutable directories as necessary. If the named target directory + already exists, this will make no changes to it. If the final directory is created, it will be empty. hunk ./docs/frontends/webapi.txt 467 - This will return an error if a blocking file is present at any of the parent - names, preventing the server from creating the necessary parent directory. + This operation will return an error if a blocking file is present at any of + the parent names, preventing the server from creating the necessary parent + directory; or if it would require changing an immutable directory. The write-cap of the new directory will be returned as the HTTP response body. hunk ./docs/frontends/webapi.txt 476 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-with-children - Like above, but if the final directory is created, it will be populated with - initial children from the POST request body, as described above in the - /uri?t=mkdir-with-children operation. + Like /uri?t=mkdir-with-children, but the final directory is created as a + child of an existing mutable directory. This will create additional + intermediate mutable directories as necessary. If the final directory is + created, it will be populated with initial children from the POST request + body, as described above. + + This operation will return an error if a blocking file is present at any of + the parent names, preventing the server from creating the necessary parent + directory; or if it would require changing an immutable directory; or if + the immediate parent directory already has a a child named SUBDIR. POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-immutable hunk ./docs/frontends/webapi.txt 489 - Like above, but the final directory will be deep-immutable, with the - children specified as a JSON dictionary in the POST request body. + Like /uri?t=mkdir-immutable, but the final directory is created as a child + of an existing mutable directory. The final directory will be deep-immutable, + and will be populated with the children specified as a JSON dictionary in + the POST request body. + + In Tahoe 1.6 this operation creates intermediate mutable directories if + necessary, but that behaviour should not be relied on; see ticket #920. + + This operation will return an error if the parent directory is immutable, + or already has a child named SUBDIR. POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME hunk ./docs/frontends/webapi.txt 502 - Create a new empty directory and attach it to the given existing directory. - This will create additional intermediate directories as necessary. + Create a new empty mutable directory and attach it to the given existing + directory. This will create additional intermediate directories as necessary. hunk ./docs/frontends/webapi.txt 505 - The URL of this form points to the parent of the bottom-most new directory, - whereas the previous form has a URL that points directly to the bottom-most - new directory. + This operation will return an error if a blocking file is present at any of + the parent names, preventing the server from creating the necessary parent + directory, or if it would require changing any immutable directory. + + The URL of this operation points to the parent of the bottommost new directory, + whereas the /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir operation above has a URL + that points directly to the bottommost new directory. POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME hunk ./docs/frontends/webapi.txt 515 - As above, but the new directory will be populated with initial children via - the POST request body, as described in /uri?t=mkdir-with-children above. + Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME, but the new directory will + be populated with initial children via the POST request body. This command + will create additional intermediate mutable directories as necessary. + + This operation will return an error if a blocking file is present at any of + the parent names, preventing the server from creating the necessary parent + directory; or if it would require changing an immutable directory; or if + the immediate parent directory already has a a child named NAME. + Note that the name= argument must be passed as a queryarg, because the POST request body is used for the initial children JSON. hunk ./docs/frontends/webapi.txt 529 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME - As above, but the new directory will be deep-immutable, with the children - specified as a JSON dictionary in the POST request body. Again, the name= - argument must be passed as a queryarg. + Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME, but the + final directory will be deep-immutable. The children are specified as a + JSON dictionary in the POST request body. Again, the name= argument must be + passed as a queryarg. + + In Tahoe 1.6 this operation creates intermediate mutable directories if + necessary, but that behaviour should not be relied on; see ticket #920. + + This operation will return an error if the parent directory is immutable, + or already has a child named NAME. === Get Information About A File Or Directory (as JSON) === hunk ./docs/frontends/webapi.txt 761 "childinfo" is a dictionary that contains "rw_uri", "ro_uri", and "metadata" keys. You can take the output of "GET /uri/$DIRCAP1?t=json" and use it as the input to "POST /uri/$DIRCAP2?t=set_children" to make DIR2 - look very much like DIR1. + look very much like DIR1 (except for any existing children of DIR2 that + were not overwritten, and any existing "tahoe" metadata keys as described + below). When the set_children request contains a child name that already exists in the target directory, this command defaults to overwriting that child with hunk ./docs/frontends/webapi.txt 964 POST /uri/$DIRCAP/[SUBDIRS../]?t=upload - This uploads a file, and attaches it as a new child of the given directory. - The file must be provided as the "file" field of an HTML encoded form body, - produced in response to an HTML form like this: + This uploads a file, and attaches it as a new child of the given directory, + which must be mutable. The file must be provided as the "file" field of an + HTML-encoded form body, produced in response to an HTML form like this:
hunk ./docs/frontends/webapi.txt 1009 POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload This also uploads a file and attaches it as a new child of the given - directory. It is a slight variant of the previous operation, as the URL - refers to the target file rather than the parent directory. It is otherwise - identical: this accepts mutable= and when_done= arguments too. + directory, which must be mutable. It is a slight variant of the previous + operation, as the URL refers to the target file rather than the parent + directory. It is otherwise identical: this accepts mutable= and when_done= + arguments too. POST /uri/$FILECAP?t=upload hunk ./docs/frontends/webapi.txt 1040 POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME - This instructs the node to delete a child object (file or subdirectory) from - the given directory. Note that the entire subtree is removed. This is - somewhat like "rm -rf" (from the point of view of the parent), but other - references into the subtree will see that the child subdirectories are not - modified by this operation. Only the link from the given directory to its - child is severed. + This instructs the node to remove a child object (file or subdirectory) from + the given directory, which must be mutable. Note that the entire subtree is + unlinked from the parent. Unlike deleting a subdirectory in a UNIX local + filesystem, the subtree need not be empty; if it isn't, then other references + into the subtree will see that the child subdirectories are not modified by + this operation. Only the link from the given directory to its child is severed. === Renaming A Child === hunk ./docs/frontends/webapi.txt 1051 POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW - This instructs the node to rename a child of the given directory. This is - exactly the same as removing the child, then adding the same child-cap under - the new name. This operation cannot move the child to a different directory. + This instructs the node to rename a child of the given directory, which must + be mutable. This has a similar effect to removing the child, then adding the + same child-cap under the new name, except that it preserves metadata. This + operation cannot move the child to a different directory. This operation will replace any existing child of the new name, making it behave like the UNIX "mv -f" command. hunk ./docs/frontends/webapi.txt 1250 "path": a list of strings, with the path that is traversed to reach the object - "cap": a writecap for the file or directory, if available, else a readcap - "verifycap": a verifycap for the file or directory - "repaircap": the weakest cap which can still be used to repair the object + "cap": a write-cap URI for the file or directory, if available, else a + read-cap URI + "verifycap": a verify-cap URI for the file or directory + "repaircap": an URI for the weakest cap that can still be used to repair + the object "storage-index": a base32 storage index for the object "check-results": a copy of the dictionary which would be returned by t=check&output=json, with three top-level keys: hunk ./docs/frontends/webapi.txt 1496 "path": a list of strings, with the path that is traversed to reach the object - "cap": a writecap for the file or directory, if available, else a readcap - "verifycap": a verifycap for the file or directory - "repaircap": the weakest cap which can still be used to repair the object + "cap": a write-cap URI for the file or directory, if available, else a + read-cap URI + "verifycap": a verify-cap URI for the file or directory + "repaircap": an URI for the weakest cap that can still be used to repair + the object "storage-index": a base32 storage index for the object Note that non-distributed files (i.e. LIT files) will have values of None hunk ./docs/frontends/webapi.txt 1731 child's name and the child's URI are included in the results of listing the parent directory, so it isn't any harder to use the URI for this purpose. +The read and write caps in a given directory node are separate URIs, and +can't be assumed to point to the same object even if they were retrieved in +the same operation (although the webapi server attempts to ensure this +in most cases). If you need to rely on that property, you should explicitly +verify it. More generally, you should not make assumptions about the +internal consistency of the contents of mutable directories. As a result +of the signatures on mutable object versions, it is guaranteed that a given +version was written in a single update, but -- as in the case of a file -- +the contents may have been chosen by a malicious writer in a way that is +designed to confuse applications that rely on their consistency. + In general, use names if you want "whatever object (whether file or directory) is found by following this name (or sequence of names) when my request reaches the server". Use URIs if you want "this particular object". hunk ./src/allmydata/client.py 474 # dirnodes. The first takes a URI and produces a filenode or (new-style) # dirnode. The other three create brand-new filenodes/dirnodes. - def create_node_from_uri(self, writecap, readcap=None): - # this returns synchronously. - return self.nodemaker.create_from_cap(writecap, readcap) + def create_node_from_uri(self, write_uri, read_uri=None, deep_immutable=False, name=""): + # This returns synchronously. + # Note that it does *not* validate the write_uri and read_uri; instead we + # may get an opaque node if there were any problems. + return self.nodemaker.create_from_cap(write_uri, read_uri, deep_immutable=deep_immutable, name=name) def create_dirnode(self, initial_children={}): d = self.nodemaker.create_new_mutable_directory(initial_children) hunk ./src/allmydata/client.py 483 return d + def create_immutable_dirnode(self, children, convergence=None): return self.nodemaker.create_immutable_directory(children, convergence) hunk ./src/allmydata/control.py 8 from twisted.internet import defer from twisted.internet.interfaces import IConsumer from foolscap.api import Referenceable -from allmydata.interfaces import RIControlClient +from allmydata.interfaces import RIControlClient, IFileNode from allmydata.util import fileutil, mathutil from allmydata.immutable import upload from twisted.python import log hunk ./src/allmydata/control.py 70 return d def remote_download_from_uri_to_file(self, uri, filename): - filenode = self.parent.create_node_from_uri(uri) + filenode = self.parent.create_node_from_uri(uri, name=filename) + if not IFileNode.providedBy(filenode): + raise AssertionError("The URI does not reference a file.") c = FileWritingConsumer(filename) d = filenode.read(c) d.addCallback(lambda res: filename) hunk ./src/allmydata/control.py 204 if i >= self.count: return n = self.parent.create_node_from_uri(self.uris[i]) + if not IFileNode.providedBy(n): + raise AssertionError("The URI does not reference a file.") if n.is_mutable(): d1 = n.download_best_version() else: hunk ./src/allmydata/dirnode.py 8 from twisted.internet import defer from foolscap.api import fireEventually import simplejson -from allmydata.mutable.common import NotMutableError +from allmydata.mutable.common import NotWriteableError from allmydata.mutable.filenode import MutableFileNode hunk ./src/allmydata/dirnode.py 10 -from allmydata.unknown import UnknownNode +from allmydata.unknown import UnknownNode, strip_prefix_for_ro from allmydata.interfaces import IFilesystemNode, IDirectoryNode, IFileNode, \ IImmutableFileNode, IMutableFileNode, \ ExistingChildError, NoSuchChildError, ICheckable, IDeepCheckable, \ hunk ./src/allmydata/dirnode.py 14 - CannotPackUnknownNodeError + MustBeDeepImmutableError, CapConstraintError from allmydata.check_results import DeepCheckResults, \ DeepCheckAndRepairResults from allmydata.monitor import Monitor hunk ./src/allmydata/dirnode.py 43 new_contents = self.node._pack_contents(children) return new_contents + class MetadataSetter: def __init__(self, node, name, metadata): self.node = node hunk ./src/allmydata/dirnode.py 79 for (name, (child, new_metadata)) in self.entries.iteritems(): precondition(isinstance(name, unicode), name) precondition(IFilesystemNode.providedBy(child), child) + + # Strictly speaking this is redundant because we would raise the + # error again in pack_children. + child.raise_error() + if name in children: if not self.overwrite: raise ExistingChildError("child '%s' already exists" % name) hunk ./src/allmydata/dirnode.py 132 new_contents = self.node._pack_contents(children) return new_contents -def _encrypt_rwcap(filenode, rwcap): - assert isinstance(rwcap, str) +def _encrypt_rw_uri(filenode, rw_uri): + assert isinstance(rw_uri, str) writekey = filenode.get_writekey() if not writekey: return "" hunk ./src/allmydata/dirnode.py 137 - salt = hashutil.mutable_rwcap_salt_hash(rwcap) + salt = hashutil.mutable_rwcap_salt_hash(rw_uri) key = hashutil.mutable_rwcap_key_hash(salt, writekey) cryptor = AES(key) hunk ./src/allmydata/dirnode.py 140 - crypttext = cryptor.process(rwcap) + crypttext = cryptor.process(rw_uri) mac = hashutil.hmac(key, salt + crypttext) assert len(mac) == 32 return salt + crypttext + mac hunk ./src/allmydata/dirnode.py 147 # The MAC is not checked by readers in Tahoe >= 1.3.0, but we still # produce it for the sake of older readers. -class MustBeDeepImmutable(Exception): - """You tried to add a non-deep-immutable node to a deep-immutable - directory.""" - def pack_children(filenode, children, deep_immutable=False): """Take a dict that maps: children[unicode_name] = (IFileSystemNode, metadata_dict) hunk ./src/allmydata/dirnode.py 157 time. If deep_immutable is True, I will require that all my children are deeply - immutable, and will raise a MustBeDeepImmutable exception if not. + immutable, and will raise a MustBeDeepImmutableError if not. """ has_aux = isinstance(children, AuxValueDict) hunk ./src/allmydata/dirnode.py 166 assert isinstance(name, unicode) entry = None (child, metadata) = children[name] - if deep_immutable and child.is_mutable(): - # TODO: consider adding IFileSystemNode.is_deep_immutable() - raise MustBeDeepImmutable("child '%s' is mutable" % (name,)) + child.raise_error() + if deep_immutable and not child.is_allowed_in_immutable_directory(): + raise MustBeDeepImmutableError("child '%s' is not allowed in an immutable directory" % (name,), name) if has_aux: entry = children.get_aux(name) if not entry: hunk ./src/allmydata/dirnode.py 174 assert IFilesystemNode.providedBy(child), (name,child) assert isinstance(metadata, dict) - rwcap = child.get_uri() # might be RO if the child is not writeable - if rwcap is None: - rwcap = "" - assert isinstance(rwcap, str), rwcap - rocap = child.get_readonly_uri() - if rocap is None: - rocap = "" - assert isinstance(rocap, str), rocap + rw_uri = child.get_write_uri() + if rw_uri is None: + rw_uri = "" + assert isinstance(rw_uri, str), rw_uri + + # should be prevented by MustBeDeepImmutableError check above + assert not (rw_uri and deep_immutable) + + ro_uri = child.get_readonly_uri() + if ro_uri is None: + ro_uri = "" + assert isinstance(ro_uri, str), ro_uri entry = "".join([netstring(name.encode("utf-8")), hunk ./src/allmydata/dirnode.py 187 - netstring(rocap), - netstring(_encrypt_rwcap(filenode, rwcap)), + netstring(strip_prefix_for_ro(ro_uri, deep_immutable)), + netstring(_encrypt_rw_uri(filenode, rw_uri)), netstring(simplejson.dumps(metadata))]) entries.append(netstring(entry)) return "".join(entries) hunk ./src/allmydata/dirnode.py 239 plaintext = cryptor.process(crypttext) return plaintext - def _create_node(self, rwcap, rocap): - return self._nodemaker.create_from_cap(rwcap, rocap) + def _create_and_validate_node(self, rw_uri, ro_uri, name): + node = self._nodemaker.create_from_cap(rw_uri, ro_uri, + deep_immutable=not self.is_mutable(), + name=name) + node.raise_error() + return node def _unpack_contents(self, data): # the directory is serialized as a list of netstrings, one per child. hunk ./src/allmydata/dirnode.py 248 - # Each child is serialized as a list of four netstrings: (name, - # rocap, rwcap, metadata), in which the name,rocap,metadata are in - # cleartext. The 'name' is UTF-8 encoded. The rwcap is formatted as: - # pack("16ss32s", iv, AES(H(writekey+iv), plaintextrwcap), mac) + # Each child is serialized as a list of four netstrings: (name, ro_uri, + # rwcapdata, metadata), in which the name, ro_uri, metadata are in + # cleartext. The 'name' is UTF-8 encoded. The rwcapdata is formatted as: + # pack("16ss32s", iv, AES(H(writekey+iv), plaintext_rw_uri), mac) assert isinstance(data, str), (repr(data), type(data)) # an empty directory is serialized as an empty string if data == "": hunk ./src/allmydata/dirnode.py 257 return AuxValueDict() writeable = not self.is_readonly() + mutable = self.is_mutable() children = AuxValueDict() position = 0 while position < len(data): hunk ./src/allmydata/dirnode.py 263 entries, position = split_netstring(data, 1, position) entry = entries[0] - (name, rocap, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + (name, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + if not mutable and len(rwcapdata) > 0: + raise ValueError("the rwcapdata field of a dirnode in an immutable directory was not empty") name = name.decode("utf-8") hunk ./src/allmydata/dirnode.py 267 - rwcap = None + rw_uri = "" if writeable: hunk ./src/allmydata/dirnode.py 269 - rwcap = self._decrypt_rwcapdata(rwcapdata) - if not rwcap: - rwcap = None # rwcap is None or a non-empty string - if not rocap: - rocap = None # rocap is None or a non-empty string - child = self._create_node(rwcap, rocap) - metadata = simplejson.loads(metadata_s) - assert isinstance(metadata, dict) - children.set_with_aux(name, (child, metadata), auxilliary=entry) + rw_uri = self._decrypt_rwcapdata(rwcapdata) + + # Since the encryption uses CTR mode, it currently leaks the length of the + # plaintext rw_uri -- and therefore whether it is present, i.e. whether the + # dirnode is writeable (ticket #925). By stripping spaces in Tahoe >= 1.6.0, + # we may make it easier for future versions to plug this leak. + # ro_uri is treated in the same way for consistency. + # rw_uri and ro_uri will be either None or a non-empty string. + + rw_uri = rw_uri.strip(' ') or None + ro_uri = ro_uri.strip(' ') or None + + try: + child = self._create_and_validate_node(rw_uri, ro_uri, name) + if mutable or child.is_allowed_in_immutable_directory(): + metadata = simplejson.loads(metadata_s) + assert isinstance(metadata, dict) + children[name] = (child, metadata) + children.set_with_aux(name, (child, metadata), auxilliary=entry) + else: + log.msg(format="mutable cap for child '%(name)s' unpacked from an immutable directory", + name=name.encode("utf-8"), + facility="tahoe.webish", level=log.UNUSUAL) + except CapConstraintError, e: + log.msg(format="unmet constraint on cap for child '%(name)s' unpacked from a directory:\n" + "%(message)s", message=e.args[0], name=name.encode("utf-8"), + facility="tahoe.webish", level=log.UNUSUAL) + return children def _pack_contents(self, children): hunk ./src/allmydata/dirnode.py 305 def is_readonly(self): return self._node.is_readonly() + def is_mutable(self): return self._node.is_mutable() hunk ./src/allmydata/dirnode.py 309 + def is_unknown(self): + return False + + def is_allowed_in_immutable_directory(self): + return not self._node.is_mutable() + + def raise_error(self): + pass + def get_uri(self): return self._uri.to_string() hunk ./src/allmydata/dirnode.py 321 + def get_write_uri(self): + if self.is_readonly(): + return None + return self._uri.to_string() + def get_readonly_uri(self): return self._uri.get_readonly().to_string() hunk ./src/allmydata/dirnode.py 331 def get_cap(self): return self._uri + def get_readcap(self): return self._uri.get_readonly() hunk ./src/allmydata/dirnode.py 334 + def get_verify_cap(self): return self._uri.get_verify_cap() hunk ./src/allmydata/dirnode.py 337 + def get_repair_cap(self): if self._node.is_readonly(): return None # readonly (mutable) dirnodes are not yet repairable hunk ./src/allmydata/dirnode.py 403 def set_metadata_for(self, name, metadata): assert isinstance(name, unicode) if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) assert isinstance(metadata, dict) s = MetadataSetter(self, name, metadata) d = self._node.modify(s.modify) hunk ./src/allmydata/dirnode.py 451 precondition(isinstance(name, unicode), name) precondition(isinstance(writecap, (str,type(None))), writecap) precondition(isinstance(readcap, (str,type(None))), readcap) - child_node = self._create_node(writecap, readcap) - if isinstance(child_node, UnknownNode): - # don't be willing to pack unknown nodes: we might accidentally - # put some write-authority into the rocap slot because we don't - # know how to diminish the URI they gave us. We don't even know - # if they gave us a readcap or a writecap. - msg = "cannot pack unknown node as child %s" % str(name) - raise CannotPackUnknownNodeError(msg) + + # We now allow packing unknown nodes, provided they are valid + # for this type of directory. + child_node = self._create_and_validate_node(writecap, readcap, name) d = self.set_node(name, child_node, metadata, overwrite) d.addCallback(lambda res: child_node) return d hunk ./src/allmydata/dirnode.py 472 writecap, readcap, metadata = e precondition(isinstance(writecap, (str,type(None))), writecap) precondition(isinstance(readcap, (str,type(None))), readcap) - child_node = self._create_node(writecap, readcap) - if isinstance(child_node, UnknownNode): - msg = "cannot pack unknown node as child %s" % str(name) - raise CannotPackUnknownNodeError(msg) + + # We now allow packing unknown nodes, provided they are valid + # for this type of directory. + child_node = self._create_and_validate_node(writecap, readcap, name) a.set_node(name, child_node, metadata) d = self._node.modify(a.modify) d.addCallback(lambda ign: self) hunk ./src/allmydata/dirnode.py 488 same name. If this directory node is read-only, the Deferred will errback with a - NotMutableError.""" + NotWriteableError.""" precondition(IFilesystemNode.providedBy(child), child) hunk ./src/allmydata/dirnode.py 493 if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) assert isinstance(name, unicode) assert IFilesystemNode.providedBy(child), child a = Adder(self, overwrite=overwrite) hunk ./src/allmydata/dirnode.py 505 def set_nodes(self, entries, overwrite=True): precondition(isinstance(entries, dict), entries) if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) a = Adder(self, entries, overwrite=overwrite) d = self._node.modify(a.modify) d.addCallback(lambda res: self) hunk ./src/allmydata/dirnode.py 519 the operation completes.""" assert isinstance(name, unicode) if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) d = self._uploader.upload(uploadable) hunk ./src/allmydata/dirnode.py 521 - d.addCallback(lambda results: results.uri) - d.addCallback(self._nodemaker.create_from_cap) + d.addCallback(lambda results: + self._create_and_validate_node(results.uri, None, name)) d.addCallback(lambda node: self.set_node(name, node, metadata, overwrite)) return d hunk ./src/allmydata/dirnode.py 532 fires (with the node just removed) when the operation finishes.""" assert isinstance(name, unicode) if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) deleter = Deleter(self, name) d = self._node.modify(deleter.modify) d.addCallback(lambda res: deleter.old_child) hunk ./src/allmydata/dirnode.py 542 mutable=True): assert isinstance(name, unicode) if self.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) if mutable: d = self._nodemaker.create_new_mutable_directory(initial_children) else: hunk ./src/allmydata/dirnode.py 564 Deferred that fires when the operation finishes.""" assert isinstance(current_child_name, unicode) if self.is_readonly() or new_parent.is_readonly(): - return defer.fail(NotMutableError()) + return defer.fail(NotWriteableError()) if new_child_name is None: new_child_name = current_child_name assert isinstance(new_child_name, unicode) hunk ./src/allmydata/immutable/filenode.py 20 class _ImmutableFileNodeBase(object): implements(IImmutableFileNode, ICheckable) + def get_write_uri(self): + return None + def get_readonly_uri(self): return self.get_uri() hunk ./src/allmydata/immutable/filenode.py 32 def is_readonly(self): return True + def is_unknown(self): + return False + + def is_allowed_in_immutable_directory(self): + return True + + def raise_error(self): + pass + def __hash__(self): return self.u.__hash__() def __eq__(self, other): hunk ./src/allmydata/interfaces.py 459 class IDirnodeURI(Interface): """I am a URI which represents a dirnode.""" - class IFileURI(Interface): """I am a URI which represents a filenode.""" def get_size(): hunk ./src/allmydata/interfaces.py 469 class IMutableFileURI(Interface): """I am a URI which represents a mutable filenode.""" + class IDirectoryURI(Interface): pass hunk ./src/allmydata/interfaces.py 472 + class IReadonlyDirectoryURI(Interface): pass hunk ./src/allmydata/interfaces.py 476 -class CannotPackUnknownNodeError(Exception): - """UnknownNodes (using filecaps from the future that we don't understand) - cannot yet be copied safely, so I refuse to copy them.""" +class CapConstraintError(Exception): + """A constraint on a cap was violated.""" hunk ./src/allmydata/interfaces.py 479 -class UnhandledCapTypeError(Exception): - """I recognize the cap/URI, but I cannot create an IFilesystemNode for - it.""" +class MustBeDeepImmutableError(CapConstraintError): + """Mutable children cannot be added to an immutable directory. + Also, caps obtained from an immutable directory can trigger this error + if they are later found to refer to a mutable object and then used.""" hunk ./src/allmydata/interfaces.py 484 -class NotDeepImmutableError(Exception): - """Deep-immutable directories can only contain deep-immutable children""" +class MustBeReadonlyError(CapConstraintError): + """Known write caps cannot be specified in a ro_uri field. Also, + caps obtained from a ro_uri field can trigger this error if they + are later found to be write caps and then used.""" + +class MustNotBeUnknownRWError(CapConstraintError): + """Cannot add an unknown child cap specified in a rw_uri field.""" # The hierarchy looks like this: # IFilesystemNode hunk ./src/allmydata/interfaces.py 527 """ def get_uri(): - """ - Return the URI string that can be used by others to get access to - this node. If this node is read-only, the URI will only offer + """Return the URI string corresponding to the strongest cap associated + with this node. If this node is read-only, the URI will only offer read-only access. If this node is read-write, the URI will offer read-write access. hunk ./src/allmydata/interfaces.py 536 read-only access with others, use get_readonly_uri(). """ + def get_write_uri(n): + """Return the URI string that can be used by others to get write + access to this node, if it is writeable. If this is a read-only node, + return None.""" + def get_readonly_uri(): """Return the URI string that can be used by others to get read-only access to this node. The result is a read-only URI, regardless of hunk ./src/allmydata/interfaces.py 570 file. """ + def is_unknown(): + """Return True if this is an unknown node.""" + + def is_allowed_in_immutable_directory(): + """Return True if this node is allowed as a child of a deep-immutable + directory. This is true if either the node is of a known-immutable type, + or it is unknown and read-only. + """ + + def raise_error(): + """Raise any error associated with this node.""" + def get_size(): """Return the length (in bytes) of the data this node represents. For directory nodes, I return the size of the backing store. I return hunk ./src/allmydata/interfaces.py 927 ctime/mtime semantics of traditional filesystems. If this directory node is read-only, the Deferred will errback with a - NotMutableError.""" + NotWriteableError.""" def set_children(entries, overwrite=True): """Add multiple children (by writecap+readcap) to a directory node. hunk ./src/allmydata/interfaces.py 953 ctime/mtime semantics of traditional filesystems. If this directory node is read-only, the Deferred will errback with a - NotMutableError.""" + NotWriteableError.""" def set_nodes(entries, overwrite=True): """Add multiple children to a directory node. Takes a dict mapping hunk ./src/allmydata/interfaces.py 2099 Tahoe process will typically have a single NodeMaker, but unit tests may create simplified/mocked forms for testing purposes. """ - def create_from_cap(writecap, readcap=None): + def create_from_cap(writecap, readcap=None, **kwargs): """I create an IFilesystemNode from the given writecap/readcap. I can only provide nodes for existing file/directory objects: use my other methods to create new objects. I return synchronously.""" hunk ./src/allmydata/mutable/common.py 11 # creation MODE_READ = "MODE_READ" -class NotMutableError(Exception): +class NotWriteableError(Exception): pass class NeedMoreDataError(Exception): hunk ./src/allmydata/mutable/filenode.py 217 def get_uri(self): return self._uri.to_string() + + def get_write_uri(self): + if self.is_readonly(): + return None + return self._uri.to_string() + def get_readonly_uri(self): return self._uri.get_readonly().to_string() hunk ./src/allmydata/mutable/filenode.py 236 def is_mutable(self): return self._uri.is_mutable() + def is_readonly(self): return self._uri.is_readonly() hunk ./src/allmydata/mutable/filenode.py 240 + def is_unknown(self): + return False + + def is_allowed_in_immutable_directory(self): + return not self._uri.is_mutable() + + def raise_error(self): + pass + def __hash__(self): return hash((self.__class__, self._uri)) def __cmp__(self, them): hunk ./src/allmydata/nodemaker.py 4 import weakref from zope.interface import implements from allmydata.util.assertutil import precondition -from allmydata.interfaces import INodeMaker, NotDeepImmutableError +from allmydata.interfaces import INodeMaker, MustBeDeepImmutableError from allmydata.immutable.filenode import ImmutableFileNode, LiteralFileNode from allmydata.immutable.upload import Data from allmydata.mutable.filenode import MutableFileNode hunk ./src/allmydata/nodemaker.py 47 def _create_dirnode(self, filenode): return DirectoryNode(filenode, self, self.uploader) - def create_from_cap(self, writecap, readcap=None): + def create_from_cap(self, writecap, readcap=None, deep_immutable=False, name=u""): # this returns synchronously. It starts with a "cap string". assert isinstance(writecap, (str, type(None))), type(writecap) assert isinstance(readcap, (str, type(None))), type(readcap) hunk ./src/allmydata/nodemaker.py 51 + bigcap = writecap or readcap if not bigcap: # maybe the writecap was hidden because we're in a readonly hunk ./src/allmydata/nodemaker.py 57 # directory, and the future cap format doesn't have a readcap, or # something. - return UnknownNode(writecap, readcap) - if bigcap in self._node_cache: - return self._node_cache[bigcap] - cap = uri.from_string(bigcap) - node = self._create_from_cap(cap) + return UnknownNode(None, None) # deep_immutable and name not needed + + # The name doesn't matter for caching since it's only used in the error + # attribute of an UnknownNode, and we don't cache those. + memokey = ("I" if deep_immutable else "M") + bigcap + if memokey in self._node_cache: + return self._node_cache[memokey] + cap = uri.from_string(bigcap, deep_immutable=deep_immutable, name=name) + node = self._create_from_single_cap(cap) if node: hunk ./src/allmydata/nodemaker.py 67 - self._node_cache[bigcap] = node # note: WeakValueDictionary + self._node_cache[memokey] = node # note: WeakValueDictionary else: hunk ./src/allmydata/nodemaker.py 69 - node = UnknownNode(writecap, readcap) # don't cache UnknownNode + # don't cache UnknownNode + node = UnknownNode(writecap, readcap, deep_immutable=deep_immutable, name=name) return node hunk ./src/allmydata/nodemaker.py 73 - def _create_from_cap(self, cap): - # This starts with a "cap instance" + def _create_from_single_cap(self, cap): if isinstance(cap, uri.LiteralFileURI): return self._create_lit(cap) if isinstance(cap, uri.CHKFileURI): hunk ./src/allmydata/nodemaker.py 84 uri.ReadonlyDirectoryURI, uri.ImmutableDirectoryURI, uri.LiteralDirectoryURI)): - filenode = self._create_from_cap(cap.get_filenode_cap()) + filenode = self._create_from_single_cap(cap.get_filenode_cap()) return self._create_dirnode(filenode) return None hunk ./src/allmydata/nodemaker.py 97 return d def create_new_mutable_directory(self, initial_children={}): - # initial_children must have metadata (i.e. {} instead of None), and - # should not contain UnknownNodes + # initial_children must have metadata (i.e. {} instead of None) for (name, (node, metadata)) in initial_children.iteritems(): hunk ./src/allmydata/nodemaker.py 99 - precondition(not isinstance(node, UnknownNode), - "create_new_mutable_directory does not accept UnknownNode", node) precondition(isinstance(metadata, dict), "create_new_mutable_directory requires metadata to be a dict, not None", metadata) hunk ./src/allmydata/nodemaker.py 101 + node.raise_error() d = self.create_mutable_file(lambda n: pack_children(n, initial_children)) d.addCallback(self._create_dirnode) hunk ./src/allmydata/nodemaker.py 111 if convergence is None: convergence = self.secret_holder.get_convergence_secret() for (name, (node, metadata)) in children.iteritems(): - precondition(not isinstance(node, UnknownNode), - "create_immutable_directory does not accept UnknownNode", node) precondition(isinstance(metadata, dict), "create_immutable_directory requires metadata to be a dict, not None", metadata) hunk ./src/allmydata/nodemaker.py 113 - if node.is_mutable(): - raise NotDeepImmutableError("%s is not immutable" % (node,)) + node.raise_error() + if not node.is_allowed_in_immutable_directory(): + raise MustBeDeepImmutableError("%s is not immutable" % (node,), name) n = DummyImmutableFileNode() # writekey=None packed = pack_children(n, children) uploadable = Data(packed, convergence) hunk ./src/allmydata/nodemaker.py 120 d = self.uploader.upload(uploadable, history=self.history) - def _uploaded(results): - filecap = self.create_from_cap(results.uri) - return filecap - d.addCallback(_uploaded) + d.addCallback(lambda results: self.create_from_cap(None, results.uri)) d.addCallback(self._create_dirnode) return d hunk ./src/allmydata/scripts/common.py 131 pass def get_alias(aliases, path, default): + from allmydata import uri # transform "work:path/filename" into (aliases["work"], "path/filename"). # If default=None, then an empty alias is indicated by returning hunk ./src/allmydata/scripts/common.py 134 - # DefaultAliasMarker. We special-case "URI:" to make it easy to access - # specific files/directories by their read-cap. + # DefaultAliasMarker. We special-case strings with a recognized cap URI + # prefix, to make it easy to access specific files/directories by their + # caps. path = path.strip() hunk ./src/allmydata/scripts/common.py 138 - if path.startswith("URI:"): + if uri.has_uri_prefix(path): # The only way to get a sub-path is to use URI:blah:./foo, and we # strip out the :./ sequence. sep = path.find(":./") hunk ./src/allmydata/scripts/tahoe_cp.py 261 readcap = ascii_or_none(data[1].get("ro_uri")) self.children[name] = TahoeFileSource(self.nodeurl, mutable, writecap, readcap) - else: - assert data[0] == "dirnode" + elif data[0] == "dirnode": writecap = ascii_or_none(data[1].get("rw_uri")) readcap = ascii_or_none(data[1].get("ro_uri")) if writecap and writecap in self.cache: hunk ./src/allmydata/scripts/tahoe_cp.py 279 if recurse: child.populate(True) self.children[name] = child + else: + # TODO: there should be an option to skip unknown nodes. + raise TahoeError("Cannot copy unknown nodes (ticket #839). " + "You probably need to use a later version of " + "Tahoe-LAFS to copy this directory.") class TahoeMissingTarget: def __init__(self, url): hunk ./src/allmydata/scripts/tahoe_cp.py 360 urllib.quote(name.encode('utf-8'))]) self.children[name] = TahoeFileTarget(self.nodeurl, mutable, writecap, readcap, url) - else: - assert data[0] == "dirnode" + elif data[0] == "dirnode": writecap = ascii_or_none(data[1].get("rw_uri")) readcap = ascii_or_none(data[1].get("ro_uri")) if writecap and writecap in self.cache: hunk ./src/allmydata/scripts/tahoe_cp.py 378 if recurse: child.populate(True) self.children[name] = child + else: + # TODO: there should be an option to skip unknown nodes. + raise TahoeError("Cannot copy unknown nodes (ticket #839). " + "You probably need to use a later version of " + "Tahoe-LAFS to copy this directory.") def get_child_target(self, name): # return a new target for a named subdirectory of this dir hunk ./src/allmydata/scripts/tahoe_cp.py 418 set_data = {} for (name, filecap) in self.new_children.items(): # it just so happens that ?t=set_children will accept both file - # read-caps and write-caps as ['rw_uri'], and will handle eithe + # read-caps and write-caps as ['rw_uri'], and will handle either # correctly. So don't bother trying to figure out whether the one # we have is read-only or read-write. hunk ./src/allmydata/scripts/tahoe_cp.py 421 + # TODO: think about how this affects forward-compatibility for + # unknown caps set_data[name] = ["filenode", {"rw_uri": filecap}] body = simplejson.dumps(set_data) POST(url, body) hunk ./src/allmydata/scripts/tahoe_cp.py 783 # local-file-in-the-way # touch proposed # tahoe cp -r my:docs/proposed/denver.txt proposed/denver.txt +# handling of unknown nodes # things that maybe should be errors but aren't # local-dir-in-the-way hunk ./src/allmydata/scripts/tahoe_put.py 43 # DIRCAP:./subdir/foo : DIRCAP/subdir/foo # MUTABLE-FILE-WRITECAP : filecap + # FIXME: this shouldn't rely on a particular prefix. if to_file.startswith("URI:SSK:"): url = nodeurl + "uri/%s" % urllib.quote(to_file) else: hunk ./src/allmydata/test/common.py 54 def get_uri(self): return self.my_uri.to_string() + def get_write_uri(self): + return None def get_readonly_uri(self): return self.my_uri.to_string() def get_cap(self): hunk ./src/allmydata/test/common.py 108 return False def is_readonly(self): return True + def is_unknown(self): + return False + def is_allowed_in_immutable_directory(self): + return True + def raise_error(self): + pass def get_size(self): try: hunk ./src/allmydata/test/common.py 201 return self.my_uri.get_readonly() def get_uri(self): return self.my_uri.to_string() + def get_write_uri(self): + if self.is_readonly(): + return None + return self.my_uri.to_string() def get_readonly(self): return self.my_uri.get_readonly() def get_readonly_uri(self): hunk ./src/allmydata/test/common.py 215 return self.my_uri.is_readonly() def is_mutable(self): return self.my_uri.is_mutable() + def is_unknown(self): + return False + def is_allowed_in_immutable_directory(self): + return not self.my_uri.is_mutable() + def raise_error(self): + pass def get_writekey(self): return "\x00"*16 def get_size(self): hunk ./src/allmydata/test/test_client.py 291 self.failUnless(n.is_readonly()) self.failUnless(n.is_mutable()) - future = "x-tahoe-crazy://future_cap_format." - n = c.create_node_from_uri(future) + unknown_rw = "lafs://from_the_future" + unknown_ro = "lafs://readonly_from_the_future" + n = c.create_node_from_uri(unknown_rw, unknown_ro) self.failUnless(IFilesystemNode.providedBy(n)) self.failIf(IFileNode.providedBy(n)) self.failIf(IImmutableFileNode.providedBy(n)) hunk ./src/allmydata/test/test_client.py 299 self.failIf(IMutableFileNode.providedBy(n)) self.failIf(IDirectoryNode.providedBy(n)) - self.failUnlessEqual(n.get_uri(), future) + self.failUnless(n.is_unknown()) + self.failUnlessEqual(n.get_uri(), unknown_rw) + self.failUnlessEqual(n.get_write_uri(), unknown_rw) + self.failUnlessEqual(n.get_readonly_uri(), "ro." + unknown_ro) hunk ./src/allmydata/test/test_dirnode.py 10 from allmydata.client import Client from allmydata.immutable import upload from allmydata.interfaces import IImmutableFileNode, IMutableFileNode, \ - ExistingChildError, NoSuchChildError, NotDeepImmutableError, \ - IDeepCheckResults, IDeepCheckAndRepairResults, CannotPackUnknownNodeError + ExistingChildError, NoSuchChildError, MustNotBeUnknownRWError, \ + MustBeDeepImmutableError, MustBeReadonlyError, \ + IDeepCheckResults, IDeepCheckAndRepairResults from allmydata.mutable.filenode import MutableFileNode from allmydata.mutable.common import UncoordinatedWriteError from allmydata.util import hashutil, base32 hunk ./src/allmydata/test/test_dirnode.py 20 from allmydata.test.common import make_chk_file_uri, make_mutable_file_uri, \ ErrorMixin from allmydata.test.no_network import GridTestMixin -from allmydata.unknown import UnknownNode +from allmydata.unknown import UnknownNode, strip_prefix_for_ro from allmydata.nodemaker import NodeMaker from base64 import b32decode import common_util as testutil hunk ./src/allmydata/test/test_dirnode.py 36 d = c.create_dirnode() def _done(res): self.failUnless(isinstance(res, dirnode.DirectoryNode)) + self.failUnless(res.is_mutable()) + self.failIf(res.is_readonly()) + self.failIf(res.is_unknown()) + self.failIf(res.is_allowed_in_immutable_directory()) + res.raise_error() rep = str(res) self.failUnless("RW-MUT" in rep) d.addCallback(_done) hunk ./src/allmydata/test/test_dirnode.py 53 nm = c.nodemaker setup_py_uri = "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" one_uri = "URI:LIT:n5xgk" # LIT for "one" + mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + mut_read_uri = "URI:SSK-RO:jf6wkflosyvntwxqcdo7a54jvm:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + future_write_uri = "x-tahoe-crazy://I_am_from_the_future." + future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." kids = {u"one": (nm.create_from_cap(one_uri), {}), u"two": (nm.create_from_cap(setup_py_uri), {"metakey": "metavalue"}), hunk ./src/allmydata/test/test_dirnode.py 60 + u"mut": (nm.create_from_cap(mut_write_uri, mut_read_uri), {}), + u"fut": (nm.create_from_cap(future_write_uri, future_read_uri), {}), + u"fro": (nm.create_from_cap(None, future_read_uri), {}), } d = c.create_dirnode(kids) hunk ./src/allmydata/test/test_dirnode.py 65 + def _created(dn): self.failUnless(isinstance(dn, dirnode.DirectoryNode)) hunk ./src/allmydata/test/test_dirnode.py 68 + self.failUnless(dn.is_mutable()) + self.failIf(dn.is_readonly()) + self.failIf(dn.is_unknown()) + self.failIf(dn.is_allowed_in_immutable_directory()) + dn.raise_error() rep = str(dn) self.failUnless("RW-MUT" in rep) return dn.list() hunk ./src/allmydata/test/test_dirnode.py 77 d.addCallback(_created) + def _check_kids(children): hunk ./src/allmydata/test/test_dirnode.py 79 - self.failUnlessEqual(sorted(children.keys()), [u"one", u"two"]) + self.failUnlessEqual(sorted(children.keys()), + [u"fro", u"fut", u"mut", u"one", u"two"]) one_node, one_metadata = children[u"one"] two_node, two_metadata = children[u"two"] hunk ./src/allmydata/test/test_dirnode.py 83 + mut_node, mut_metadata = children[u"mut"] + fut_node, fut_metadata = children[u"fut"] + fro_node, fro_metadata = children[u"fro"] + self.failUnlessEqual(one_node.get_size(), 3) hunk ./src/allmydata/test/test_dirnode.py 88 - self.failUnlessEqual(two_node.get_size(), 14861) + self.failUnlessEqual(one_node.get_uri(), one_uri) + self.failUnlessEqual(one_node.get_readonly_uri(), one_uri) self.failUnless(isinstance(one_metadata, dict), one_metadata) hunk ./src/allmydata/test/test_dirnode.py 91 + + self.failUnlessEqual(two_node.get_size(), 14861) + self.failUnlessEqual(two_node.get_uri(), setup_py_uri) + self.failUnlessEqual(two_node.get_readonly_uri(), setup_py_uri) self.failUnlessEqual(two_metadata["metakey"], "metavalue") hunk ./src/allmydata/test/test_dirnode.py 96 + + self.failUnlessEqual(mut_node.get_uri(), mut_write_uri) + self.failUnlessEqual(mut_node.get_readonly_uri(), mut_read_uri) + self.failUnless(isinstance(mut_metadata, dict), mut_metadata) + + self.failUnless(fut_node.is_unknown()) + self.failUnlessEqual(fut_node.get_uri(), future_write_uri) + self.failUnlessEqual(fut_node.get_readonly_uri(), "ro." + future_read_uri) + self.failUnless(isinstance(fut_metadata, dict), fut_metadata) + + self.failUnless(fro_node.is_unknown()) + self.failUnlessEqual(fro_node.get_uri(), "ro." + future_read_uri) + self.failUnlessEqual(fut_node.get_readonly_uri(), "ro." + future_read_uri) + self.failUnless(isinstance(fro_metadata, dict), fro_metadata) d.addCallback(_check_kids) hunk ./src/allmydata/test/test_dirnode.py 111 + d.addCallback(lambda ign: nm.create_new_mutable_directory(kids)) d.addCallback(lambda dn: dn.list()) d.addCallback(_check_kids) hunk ./src/allmydata/test/test_dirnode.py 115 - future_writecap = "x-tahoe-crazy://I_am_from_the_future." - future_readcap = "x-tahoe-crazy-readonly://I_am_from_the_future." - future_node = UnknownNode(future_writecap, future_readcap) - bad_kids1 = {u"one": (future_node, {})} + + bad_future_node = UnknownNode(future_write_uri, None) + bad_kids1 = {u"one": (bad_future_node, {})} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 119 - self.shouldFail(AssertionError, "bad_kids1", - "does not accept UnknownNode", + self.shouldFail(MustNotBeUnknownRWError, "bad_kids1", + "cannot attach unknown", nm.create_new_mutable_directory, bad_kids1)) bad_kids2 = {u"one": (nm.create_from_cap(one_uri), None)} hunk ./src/allmydata/test/test_dirnode.py 138 nm = c.nodemaker setup_py_uri = "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" one_uri = "URI:LIT:n5xgk" # LIT for "one" - mut_readcap = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" - mut_writecap = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + mut_read_uri = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + future_write_uri = "x-tahoe-crazy://I_am_from_the_future." + future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." kids = {u"one": (nm.create_from_cap(one_uri), {}), u"two": (nm.create_from_cap(setup_py_uri), {"metakey": "metavalue"}), hunk ./src/allmydata/test/test_dirnode.py 145 + u"fut": (nm.create_from_cap(None, future_read_uri), {}), } d = c.create_immutable_dirnode(kids) hunk ./src/allmydata/test/test_dirnode.py 148 + def _created(dn): self.failUnless(isinstance(dn, dirnode.DirectoryNode)) self.failIf(dn.is_mutable()) hunk ./src/allmydata/test/test_dirnode.py 153 self.failUnless(dn.is_readonly()) + self.failIf(dn.is_unknown()) + self.failUnless(dn.is_allowed_in_immutable_directory()) + dn.raise_error() rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() hunk ./src/allmydata/test/test_dirnode.py 163 self.cap = cap return dn.list() d.addCallback(_created) + def _check_kids(children): hunk ./src/allmydata/test/test_dirnode.py 165 - self.failUnlessEqual(sorted(children.keys()), [u"one", u"two"]) + self.failUnlessEqual(sorted(children.keys()), [u"fut", u"one", u"two"]) one_node, one_metadata = children[u"one"] two_node, two_metadata = children[u"two"] hunk ./src/allmydata/test/test_dirnode.py 168 + fut_node, fut_metadata = children[u"fut"] + self.failUnlessEqual(one_node.get_size(), 3) hunk ./src/allmydata/test/test_dirnode.py 171 - self.failUnlessEqual(two_node.get_size(), 14861) + self.failUnlessEqual(one_node.get_uri(), one_uri) + self.failUnlessEqual(one_node.get_readonly_uri(), one_uri) self.failUnless(isinstance(one_metadata, dict), one_metadata) hunk ./src/allmydata/test/test_dirnode.py 174 + + self.failUnlessEqual(two_node.get_size(), 14861) + self.failUnlessEqual(two_node.get_uri(), setup_py_uri) + self.failUnlessEqual(two_node.get_readonly_uri(), setup_py_uri) self.failUnlessEqual(two_metadata["metakey"], "metavalue") hunk ./src/allmydata/test/test_dirnode.py 179 + + self.failUnless(fut_node.is_unknown()) + self.failUnlessEqual(fut_node.get_uri(), "imm." + future_read_uri) + self.failUnlessEqual(fut_node.get_readonly_uri(), "imm." + future_read_uri) + self.failUnless(isinstance(fut_metadata, dict), fut_metadata) d.addCallback(_check_kids) hunk ./src/allmydata/test/test_dirnode.py 185 + d.addCallback(lambda ign: nm.create_from_cap(self.cap.to_string())) d.addCallback(lambda dn: dn.list()) d.addCallback(_check_kids) hunk ./src/allmydata/test/test_dirnode.py 189 - future_writecap = "x-tahoe-crazy://I_am_from_the_future." - future_readcap = "x-tahoe-crazy-readonly://I_am_from_the_future." - future_node = UnknownNode(future_writecap, future_readcap) - bad_kids1 = {u"one": (future_node, {})} + + bad_future_node1 = UnknownNode(future_write_uri, None) + bad_kids1 = {u"one": (bad_future_node1, {})} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 193 - self.shouldFail(AssertionError, "bad_kids1", - "does not accept UnknownNode", + self.shouldFail(MustNotBeUnknownRWError, "bad_kids1", + "cannot attach unknown", c.create_immutable_dirnode, bad_kids1)) hunk ./src/allmydata/test/test_dirnode.py 197 - bad_kids2 = {u"one": (nm.create_from_cap(one_uri), None)} + bad_future_node2 = UnknownNode(future_write_uri, future_read_uri) + bad_kids2 = {u"one": (bad_future_node2, {})} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 200 - self.shouldFail(AssertionError, "bad_kids2", - "requires metadata to be a dict", + self.shouldFail(MustBeDeepImmutableError, "bad_kids2", + "is not immutable", c.create_immutable_dirnode, bad_kids2)) hunk ./src/allmydata/test/test_dirnode.py 204 - bad_kids3 = {u"one": (nm.create_from_cap(mut_writecap), {})} + bad_kids3 = {u"one": (nm.create_from_cap(one_uri), None)} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 206 - self.shouldFail(NotDeepImmutableError, "bad_kids3", - "is not immutable", + self.shouldFail(AssertionError, "bad_kids3", + "requires metadata to be a dict", c.create_immutable_dirnode, bad_kids3)) hunk ./src/allmydata/test/test_dirnode.py 210 - bad_kids4 = {u"one": (nm.create_from_cap(mut_readcap), {})} + bad_kids4 = {u"one": (nm.create_from_cap(mut_write_uri), {})} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 212 - self.shouldFail(NotDeepImmutableError, "bad_kids4", + self.shouldFail(MustBeDeepImmutableError, "bad_kids4", "is not immutable", c.create_immutable_dirnode, bad_kids4)) hunk ./src/allmydata/test/test_dirnode.py 216 + bad_kids5 = {u"one": (nm.create_from_cap(mut_read_uri), {})} + d.addCallback(lambda ign: + self.shouldFail(MustBeDeepImmutableError, "bad_kids5", + "is not immutable", + c.create_immutable_dirnode, + bad_kids5)) d.addCallback(lambda ign: c.create_immutable_dirnode({})) def _created_empty(dn): self.failUnless(isinstance(dn, dirnode.DirectoryNode)) hunk ./src/allmydata/test/test_dirnode.py 227 self.failIf(dn.is_mutable()) self.failUnless(dn.is_readonly()) + self.failIf(dn.is_unknown()) + self.failUnless(dn.is_allowed_in_immutable_directory()) + dn.raise_error() rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() hunk ./src/allmydata/test/test_dirnode.py 245 self.failUnless(isinstance(dn, dirnode.DirectoryNode)) self.failIf(dn.is_mutable()) self.failUnless(dn.is_readonly()) + self.failIf(dn.is_unknown()) + self.failUnless(dn.is_allowed_in_immutable_directory()) + dn.raise_error() rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() hunk ./src/allmydata/test/test_dirnode.py 273 d.addCallback(_check_kids) d.addCallback(lambda ign: n.get(u"subdir")) d.addCallback(lambda sd: self.failIf(sd.is_mutable())) - bad_kids = {u"one": (nm.create_from_cap(mut_writecap), {})} + bad_kids = {u"one": (nm.create_from_cap(mut_write_uri), {})} d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 275 - self.shouldFail(NotDeepImmutableError, "YZ", + self.shouldFail(MustBeDeepImmutableError, "YZ", "is not immutable", n.create_subdirectory, u"sub2", bad_kids, mutable=False)) hunk ./src/allmydata/test/test_dirnode.py 283 d.addCallback(_made_parent) return d - def test_check(self): self.basedir = "dirnode/Dirnode/test_check" self.set_up_grid() hunk ./src/allmydata/test/test_dirnode.py 416 ro_dn = c.create_node_from_uri(ro_uri) self.failUnless(ro_dn.is_readonly()) self.failUnless(ro_dn.is_mutable()) + self.failIf(ro_dn.is_unknown()) + self.failIf(ro_dn.is_allowed_in_immutable_directory()) + ro_dn.raise_error() hunk ./src/allmydata/test/test_dirnode.py 420 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.set_uri, u"newchild", filecap, filecap) hunk ./src/allmydata/test/test_dirnode.py 422 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.set_node, u"newchild", filenode) hunk ./src/allmydata/test/test_dirnode.py 424 - self.shouldFail(dirnode.NotMutableError, "set_nodes ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_nodes ro", None, ro_dn.set_nodes, { u"newchild": (filenode, None) }) hunk ./src/allmydata/test/test_dirnode.py 426 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.add_file, u"newchild", uploadable) hunk ./src/allmydata/test/test_dirnode.py 428 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.delete, u"child") hunk ./src/allmydata/test/test_dirnode.py 430 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.create_subdirectory, u"newchild") hunk ./src/allmydata/test/test_dirnode.py 432 - self.shouldFail(dirnode.NotMutableError, "set_metadata_for ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_metadata_for ro", None, ro_dn.set_metadata_for, u"child", {}) hunk ./src/allmydata/test/test_dirnode.py 434 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, ro_dn.move_child_to, u"child", rw_dn) hunk ./src/allmydata/test/test_dirnode.py 436 - self.shouldFail(dirnode.NotMutableError, "set_uri ro", None, + self.shouldFail(dirnode.NotWriteableError, "set_uri ro", None, rw_dn.move_child_to, u"child", ro_dn) return ro_dn.list() d.addCallback(_ready) hunk ./src/allmydata/test/test_dirnode.py 983 nodemaker = NodeMaker(None, None, None, None, None, None, {"k": 3, "n": 10}, None) - writecap = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" - filenode = nodemaker.create_from_cap(writecap) + write_uri = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + filenode = nodemaker.create_from_cap(write_uri) node = dirnode.DirectoryNode(filenode, nodemaker, None) children = node._unpack_contents(known_tree) self._check_children(children) hunk ./src/allmydata/test/test_dirnode.py 1057 self.failUnlessIn("lit", packed) kids = self._make_kids(nm, ["imm", "lit", "write"]) - self.failUnlessRaises(dirnode.MustBeDeepImmutable, + self.failUnlessRaises(dirnode.MustBeDeepImmutableError, dirnode.pack_children, fn, kids, deep_immutable=True) hunk ./src/allmydata/test/test_dirnode.py 1063 # read-only is not enough: all children must be immutable kids = self._make_kids(nm, ["imm", "lit", "read"]) - self.failUnlessRaises(dirnode.MustBeDeepImmutable, + self.failUnlessRaises(dirnode.MustBeDeepImmutableError, dirnode.pack_children, fn, kids, deep_immutable=True) hunk ./src/allmydata/test/test_dirnode.py 1068 kids = self._make_kids(nm, ["imm", "lit", "dirwrite"]) - self.failUnlessRaises(dirnode.MustBeDeepImmutable, + self.failUnlessRaises(dirnode.MustBeDeepImmutableError, dirnode.pack_children, fn, kids, deep_immutable=True) hunk ./src/allmydata/test/test_dirnode.py 1073 kids = self._make_kids(nm, ["imm", "lit", "dirread"]) - self.failUnlessRaises(dirnode.MustBeDeepImmutable, + self.failUnlessRaises(dirnode.MustBeDeepImmutableError, dirnode.pack_children, fn, kids, deep_immutable=True) hunk ./src/allmydata/test/test_dirnode.py 1099 def get_cap(self): return self.uri + def get_uri(self): return self.uri.to_string() hunk ./src/allmydata/test/test_dirnode.py 1102 + + def get_write_uri(self): + return self.uri.to_string() + def download_best_version(self): return defer.succeed(self.data) hunk ./src/allmydata/test/test_dirnode.py 1108 + def get_writekey(self): return "writekey" hunk ./src/allmydata/test/test_dirnode.py 1111 + def is_readonly(self): return False hunk ./src/allmydata/test/test_dirnode.py 1114 + def is_mutable(self): return True hunk ./src/allmydata/test/test_dirnode.py 1117 + + def is_unknown(self): + return False + + def is_allowed_in_immutable_directory(self): + return False + def modify(self, modifier): self.data = modifier(self.data, None, True) return defer.succeed(None) hunk ./src/allmydata/test/test_dirnode.py 1146 self.nodemaker = client.nodemaker def test_from_future(self): - # create a dirnode that contains unknown URI types, and make sure we - # tolerate them properly. Since dirnodes aren't allowed to add - # unknown node types, we have to be tricky. + # Create a mutable directory that contains unknown URI types, and make sure + # we tolerate them properly. d = self.nodemaker.create_new_mutable_directory() hunk ./src/allmydata/test/test_dirnode.py 1149 - future_writecap = "x-tahoe-crazy://I_am_from_the_future." - future_readcap = "x-tahoe-crazy-readonly://I_am_from_the_future." - future_node = UnknownNode(future_writecap, future_readcap) + future_write_uri = "x-tahoe-crazy://I_am_from_the_future." + future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." + future_imm_uri = "x-tahoe-crazy-immutable://I_am_from_the_future." + future_node = UnknownNode(future_write_uri, future_read_uri) def _then(n): self._node = n return n.set_node(u"future", future_node) hunk ./src/allmydata/test/test_dirnode.py 1158 d.addCallback(_then) - # we should be prohibited from adding an unknown URI to a directory, - # since we don't know how to diminish the cap to a readcap (for the - # dirnode's rocap slot), and we don't want to accidentally grant - # write access to a holder of the dirnode's readcap. + # We should be prohibited from adding an unknown URI to a directory + # just in the rw_uri slot, since we don't know how to diminish the cap + # to a readcap (for the ro_uri slot). d.addCallback(lambda ign: hunk ./src/allmydata/test/test_dirnode.py 1162 - self.shouldFail(CannotPackUnknownNodeError, + self.shouldFail(MustNotBeUnknownRWError, "copy unknown", hunk ./src/allmydata/test/test_dirnode.py 1164 - "cannot pack unknown node as child add", + "cannot attach unknown rw cap as child", self._node.set_uri, u"add", hunk ./src/allmydata/test/test_dirnode.py 1166 - future_writecap, future_readcap)) + future_write_uri, None)) + + # However, we should be able to add both rw_uri and ro_uri as a pair of + # unknown URIs. + d.addCallback(lambda ign: self._node.set_uri(u"add-pair", + future_write_uri, future_read_uri)) + + # and to add an URI prefixed with "ro." or "imm." when it is given in a + # write slot (or URL parameter). + d.addCallback(lambda ign: self._node.set_uri(u"add-ro", + "ro." + future_read_uri, None)) + d.addCallback(lambda ign: self._node.set_uri(u"add-imm", + "imm." + future_imm_uri, None)) + d.addCallback(lambda ign: self._node.list()) def _check(children): hunk ./src/allmydata/test/test_dirnode.py 1182 - self.failUnlessEqual(len(children), 1) + self.failUnlessEqual(len(children), 4) (fn, metadata) = children[u"future"] self.failUnless(isinstance(fn, UnknownNode), fn) hunk ./src/allmydata/test/test_dirnode.py 1185 - self.failUnlessEqual(fn.get_uri(), future_writecap) - self.failUnlessEqual(fn.get_readonly_uri(), future_readcap) - # but we *should* be allowed to copy this node, because the - # UnknownNode contains all the information that was in the - # original directory (readcap and writecap), so we're preserving - # everything. + self.failUnlessEqual(fn.get_uri(), future_write_uri) + self.failUnlessEqual(fn.get_write_uri(), future_write_uri) + self.failUnlessEqual(fn.get_readonly_uri(), "ro." + future_read_uri) + + (fn2, metadata2) = children[u"add-pair"] + self.failUnless(isinstance(fn2, UnknownNode), fn2) + self.failUnlessEqual(fn2.get_uri(), future_write_uri) + self.failUnlessEqual(fn2.get_write_uri(), future_write_uri) + self.failUnlessEqual(fn2.get_readonly_uri(), "ro." + future_read_uri) + + (fn3, metadata3) = children[u"add-ro"] + self.failUnless(isinstance(fn3, UnknownNode), fn3) + self.failUnlessEqual(fn3.get_uri(), "ro." + future_read_uri) + self.failUnlessEqual(fn3.get_write_uri(), None) + self.failUnlessEqual(fn3.get_readonly_uri(), "ro." + future_read_uri) + + (fn4, metadata4) = children[u"add-imm"] + self.failUnless(isinstance(fn4, UnknownNode), fn4) + self.failUnlessEqual(fn4.get_uri(), "imm." + future_imm_uri) + self.failUnlessEqual(fn4.get_write_uri(), None) + self.failUnlessEqual(fn4.get_readonly_uri(), "imm." + future_imm_uri) + + # We should also be allowed to copy the "future" UnknownNode, because + # it contains all the information that was in the original directory + # (readcap and writecap), so we're preserving everything. return self._node.set_node(u"copy", fn) d.addCallback(_check) hunk ./src/allmydata/test/test_dirnode.py 1212 + d.addCallback(lambda ign: self._node.list()) def _check2(children): hunk ./src/allmydata/test/test_dirnode.py 1215 - self.failUnlessEqual(len(children), 2) + self.failUnlessEqual(len(children), 5) (fn, metadata) = children[u"copy"] self.failUnless(isinstance(fn, UnknownNode), fn) hunk ./src/allmydata/test/test_dirnode.py 1218 - self.failUnlessEqual(fn.get_uri(), future_writecap) - self.failUnlessEqual(fn.get_readonly_uri(), future_readcap) + self.failUnlessEqual(fn.get_uri(), future_write_uri) + self.failUnlessEqual(fn.get_write_uri(), future_write_uri) + self.failUnlessEqual(fn.get_readonly_uri(), "ro." + future_read_uri) + d.addCallback(_check2) return d hunk ./src/allmydata/test/test_dirnode.py 1224 + def test_unknown_strip_prefix_for_ro(self): + self.failUnlessEqual(strip_prefix_for_ro("foo", False), "foo") + self.failUnlessEqual(strip_prefix_for_ro("ro.foo", False), "foo") + self.failUnlessEqual(strip_prefix_for_ro("imm.foo", False), "imm.foo") + self.failUnlessEqual(strip_prefix_for_ro("foo", True), "foo") + self.failUnlessEqual(strip_prefix_for_ro("ro.foo", True), "foo") + self.failUnlessEqual(strip_prefix_for_ro("imm.foo", True), "foo") + + def test_unknownnode(self): + mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + mut_read_uri = "URI:SSK-RO:jf6wkflosyvntwxqcdo7a54jvm:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + lit_uri = "URI:LIT:n5xgk" + + # This does not attempt to be exhaustive. + no_no = [# Opaque node, but not an error. + ( 0, UnknownNode(None, None)), + ( 1, UnknownNode(None, None, deep_immutable=True)), + ] + unknown_rw = [# These are errors because we're only given a rw_uri, and we can't + # diminish it. + ( 2, UnknownNode("foo", None)), + ( 3, UnknownNode("foo", None, deep_immutable=True)), + ( 4, UnknownNode("ro.foo", None, deep_immutable=True)), + ( 5, UnknownNode("ro." + mut_read_uri, None, deep_immutable=True)), + ( 6, UnknownNode("URI:SSK-RO:foo", None, deep_immutable=True)), + ( 7, UnknownNode("URI:SSK:foo", None)), + ] + must_be_ro = [# These are errors because a readonly constraint is not met. + ( 8, UnknownNode("ro." + mut_write_uri, None)), + ( 9, UnknownNode(None, "ro." + mut_write_uri)), + ] + must_be_imm = [# These are errors because an immutable constraint is not met. + (10, UnknownNode(None, "ro.URI:SSK-RO:foo", deep_immutable=True)), + (11, UnknownNode(None, "imm.URI:SSK:foo")), + (12, UnknownNode(None, "imm.URI:SSK-RO:foo")), + (13, UnknownNode("bar", "ro.foo", deep_immutable=True)), + (14, UnknownNode("bar", "imm.foo", deep_immutable=True)), + (15, UnknownNode("bar", "imm." + lit_uri, deep_immutable=True)), + (16, UnknownNode("imm." + mut_write_uri, None)), + (17, UnknownNode("imm." + mut_read_uri, None)), + (18, UnknownNode("bar", "imm.foo")), + ] + bad_uri = [# These are errors because the URI is bad once we've stripped the prefix. + (19, UnknownNode("ro.URI:SSK-RO:foo", None)), + (20, UnknownNode("imm.URI:CHK:foo", None, deep_immutable=True)), + ] + ro_prefixed = [# These are valid, and the readcap should end up with a ro. prefix. + (21, UnknownNode(None, "foo")), + (22, UnknownNode(None, "ro.foo")), + (32, UnknownNode(None, "ro." + lit_uri)), + (23, UnknownNode("bar", "foo")), + (24, UnknownNode("bar", "ro.foo")), + (32, UnknownNode("bar", "ro." + lit_uri)), + (25, UnknownNode("ro.foo", None)), + (30, UnknownNode("ro." + lit_uri, None)), + ] + imm_prefixed = [# These are valid, and the readcap should end up with an imm. prefix. + (26, UnknownNode(None, "foo", deep_immutable=True)), + (27, UnknownNode(None, "ro.foo", deep_immutable=True)), + (28, UnknownNode(None, "imm.foo")), + (29, UnknownNode(None, "imm.foo", deep_immutable=True)), + (31, UnknownNode("imm." + lit_uri, None)), + (31, UnknownNode("imm." + lit_uri, None, deep_immutable=True)), + (33, UnknownNode(None, "imm." + lit_uri)), + (33, UnknownNode(None, "imm." + lit_uri, deep_immutable=True)), + ] + error = unknown_rw + must_be_ro + must_be_imm + bad_uri + ok = ro_prefixed + imm_prefixed + + for (i, n) in no_no + error + ok: + self.failUnless(n.is_unknown(), i) + + for (i, n) in no_no + error: + self.failUnless(n.get_uri() is None, i) + self.failUnless(n.get_write_uri() is None, i) + self.failUnless(n.get_readonly_uri() is None, i) + + for (i, n) in no_no + ok: + n.raise_error() + + for (i, n) in unknown_rw: + self.failUnlessRaises(MustNotBeUnknownRWError, lambda: n.raise_error()) + + for (i, n) in must_be_ro: + self.failUnlessRaises(MustBeReadonlyError, lambda: n.raise_error()) + + for (i, n) in must_be_imm: + self.failUnlessRaises(MustBeDeepImmutableError, lambda: n.raise_error()) + + for (i, n) in bad_uri: + self.failUnlessRaises(uri.BadURIError, lambda: n.raise_error()) + + for (i, n) in ok: + self.failIf(n.get_readonly_uri() is None, i) + + for (i, n) in ro_prefixed: + self.failUnless(n.get_readonly_uri().startswith("ro."), i) + + for (i, n) in imm_prefixed: + self.failUnless(n.get_readonly_uri().startswith("imm."), i) + + class DeepStats(unittest.TestCase): timeout = 240 # It takes longer than 120 seconds on Francois's arm box. def test_stats(self): hunk ./src/allmydata/test/test_filenode.py 44 self.failUnlessEqual(fn1.get_readcap(), u) self.failUnlessEqual(fn1.is_readonly(), True) self.failUnlessEqual(fn1.is_mutable(), False) + self.failUnlessEqual(fn1.is_unknown(), False) + self.failUnlessEqual(fn1.is_allowed_in_immutable_directory(), True) + self.failUnlessEqual(fn1.get_write_uri(), None) self.failUnlessEqual(fn1.get_readonly_uri(), u.to_string()) self.failUnlessEqual(fn1.get_size(), 1000) self.failUnlessEqual(fn1.get_storage_index(), u.storage_index) hunk ./src/allmydata/test/test_filenode.py 50 + fn1.raise_error() + fn2.raise_error() d = {} d[fn1] = 1 # exercise __hash__ v = fn1.get_verify_cap() hunk ./src/allmydata/test/test_filenode.py 57 self.failUnless(isinstance(v, uri.CHKFileVerifierURI)) self.failUnlessEqual(fn1.get_repair_cap(), v) + self.failUnlessEqual(v.is_readonly(), True) + self.failUnlessEqual(v.is_mutable(), False) def test_literal_filenode(self): hunk ./src/allmydata/test/test_filenode.py 74 self.failUnlessEqual(fn1.get_readcap(), u) self.failUnlessEqual(fn1.is_readonly(), True) self.failUnlessEqual(fn1.is_mutable(), False) + self.failUnlessEqual(fn1.is_unknown(), False) + self.failUnlessEqual(fn1.is_allowed_in_immutable_directory(), True) + self.failUnlessEqual(fn1.get_write_uri(), None) self.failUnlessEqual(fn1.get_readonly_uri(), u.to_string()) self.failUnlessEqual(fn1.get_size(), len(DATA)) self.failUnlessEqual(fn1.get_storage_index(), None) hunk ./src/allmydata/test/test_filenode.py 80 + fn1.raise_error() + fn2.raise_error() d = {} d[fn1] = 1 # exercise __hash__ hunk ./src/allmydata/test/test_filenode.py 114 self.failUnlessEqual(n.get_writekey(), wk) self.failUnlessEqual(n.get_readkey(), rk) self.failUnlessEqual(n.get_storage_index(), si) - # these itmes are populated on first read (or create), so until that + # these items are populated on first read (or create), so until that # happens they'll be None self.failUnlessEqual(n.get_privkey(), None) self.failUnlessEqual(n.get_encprivkey(), None) hunk ./src/allmydata/test/test_filenode.py 121 self.failUnlessEqual(n.get_pubkey(), None) self.failUnlessEqual(n.get_uri(), u.to_string()) + self.failUnlessEqual(n.get_write_uri(), u.to_string()) self.failUnlessEqual(n.get_readonly_uri(), u.get_readonly().to_string()) self.failUnlessEqual(n.get_cap(), u) self.failUnlessEqual(n.get_readcap(), u.get_readonly()) hunk ./src/allmydata/test/test_filenode.py 127 self.failUnlessEqual(n.is_mutable(), True) self.failUnlessEqual(n.is_readonly(), False) + self.failUnlessEqual(n.is_unknown(), False) + self.failUnlessEqual(n.is_allowed_in_immutable_directory(), False) + n.raise_error() n2 = MutableFileNode(None, None, client.get_encoding_parameters(), None).init_from_cap(u) hunk ./src/allmydata/test/test_filenode.py 136 self.failUnlessEqual(n, n2) self.failIfEqual(n, "not even the right type") self.failIfEqual(n, u) # not the right class + n.raise_error() d = {n: "can these be used as dictionary keys?"} d[n2] = "replace the old one" self.failUnlessEqual(len(d), 1) hunk ./src/allmydata/test/test_filenode.py 147 self.failUnlessEqual(nro.get_readonly(), nro) self.failUnlessEqual(nro.get_cap(), u.get_readonly()) self.failUnlessEqual(nro.get_readcap(), u.get_readonly()) + self.failUnlessEqual(nro.is_mutable(), True) + self.failUnlessEqual(nro.is_readonly(), True) + self.failUnlessEqual(nro.is_unknown(), False) + self.failUnlessEqual(nro.is_allowed_in_immutable_directory(), False) nro_u = nro.get_uri() self.failUnlessEqual(nro_u, nro.get_readonly_uri()) self.failUnlessEqual(nro_u, u.get_readonly().to_string()) hunk ./src/allmydata/test/test_filenode.py 154 - self.failUnlessEqual(nro.is_mutable(), True) - self.failUnlessEqual(nro.is_readonly(), True) + self.failUnlessEqual(nro.get_write_uri(), None) self.failUnlessEqual(nro.get_repair_cap(), None) # RSAmut needs writecap hunk ./src/allmydata/test/test_filenode.py 156 + nro.raise_error() v = n.get_verify_cap() self.failUnless(isinstance(v, uri.SSKVerifierURI)) hunk ./src/allmydata/test/test_system.py 20 from allmydata.interfaces import IDirectoryNode, IFileNode, \ NoSuchChildError, NoSharesError from allmydata.monitor import Monitor -from allmydata.mutable.common import NotMutableError +from allmydata.mutable.common import NotWriteableError from allmydata.mutable import layout as mutable_layout from foolscap.api import DeadReferenceError from twisted.python.failure import Failure hunk ./src/allmydata/test/test_system.py 893 d1.addCallback(lambda res: dirnode.list()) d1.addCallback(self.log, "dirnode.list") - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mkdir(nope)", None, dirnode.create_subdirectory, u"nope")) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "mkdir(nope)", None, dirnode.create_subdirectory, u"nope")) d1.addCallback(self.log, "doing add_file(ro)") ut = upload.Data("I will disappear, unrecorded and unobserved. The tragedy of my demise is made more poignant by its silence, but this beauty is not for you to ever know.", convergence="99i-p1x4-xd4-18yc-ywt-87uu-msu-zo -- completely and totally unguessable string (unless you read this)") hunk ./src/allmydata/test/test_system.py 897 - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "add_file(nope)", None, dirnode.add_file, u"hope", ut)) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "add_file(nope)", None, dirnode.add_file, u"hope", ut)) d1.addCallback(self.log, "doing get(ro)") d1.addCallback(lambda res: dirnode.get(u"mydata992")) hunk ./src/allmydata/test/test_system.py 905 self.failUnless(IFileNode.providedBy(filenode))) d1.addCallback(self.log, "doing delete(ro)") - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "delete(nope)", None, dirnode.delete, u"mydata992")) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "delete(nope)", None, dirnode.delete, u"mydata992")) hunk ./src/allmydata/test/test_system.py 907 - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "set_uri(nope)", None, dirnode.set_uri, u"hopeless", self.uri, self.uri)) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "set_uri(nope)", None, dirnode.set_uri, u"hopeless", self.uri, self.uri)) d1.addCallback(lambda res: self.shouldFail2(NoSuchChildError, "get(missing)", "missing", dirnode.get, u"missing")) hunk ./src/allmydata/test/test_system.py 912 personal = self._personal_node - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mv from readonly", None, dirnode.move_child_to, u"mydata992", personal, u"nope")) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "mv from readonly", None, dirnode.move_child_to, u"mydata992", personal, u"nope")) d1.addCallback(self.log, "doing move_child_to(ro)2") hunk ./src/allmydata/test/test_system.py 915 - d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mv to readonly", None, personal.move_child_to, u"sekrit data", dirnode, u"nope")) + d1.addCallback(lambda res: self.shouldFail2(NotWriteableError, "mv to readonly", None, personal.move_child_to, u"sekrit data", dirnode, u"nope")) d1.addCallback(self.log, "finished with _got_s2ro") return d1 hunk ./src/allmydata/test/test_uri.py 6 from allmydata import uri from allmydata.util import hashutil, base32 from allmydata.interfaces import IURI, IFileURI, IDirnodeURI, IMutableFileURI, \ - IVerifierURI + IVerifierURI, CapConstraintError class Literal(unittest.TestCase): def _help_test(self, data): hunk ./src/allmydata/test/test_uri.py 25 self.failIf(IDirnodeURI.providedBy(u2)) self.failUnlessEqual(u2.data, data) self.failUnlessEqual(u2.get_size(), len(data)) - self.failUnless(u.is_readonly()) - self.failIf(u.is_mutable()) + self.failUnless(u2.is_readonly()) + self.failIf(u2.is_mutable()) + + u2i = uri.from_string(u.to_string(), deep_immutable=True) + self.failUnless(IFileURI.providedBy(u2i)) + self.failIf(IDirnodeURI.providedBy(u2i)) + self.failUnlessEqual(u2i.data, data) + self.failUnlessEqual(u2i.get_size(), len(data)) + self.failUnless(u2i.is_readonly()) + self.failIf(u2i.is_mutable()) u3 = u.get_readonly() self.failUnlessIdentical(u, u3) hunk ./src/allmydata/test/test_uri.py 62 fileURI = 'URI:CHK:f5ahxa25t4qkktywz6teyfvcx4:opuioq7tj2y6idzfp6cazehtmgs5fdcebcz3cygrxyydvcozrmeq:3:10:345834' chk1 = uri.CHKFileURI.init_from_string(fileURI) chk2 = uri.CHKFileURI.init_from_string(fileURI) + unk = uri.UnknownURI("lafs://from_the_future") self.failIfEqual(lit1, chk1) self.failUnlessEqual(chk1, chk2) self.failIfEqual(chk1, "not actually a URI") hunk ./src/allmydata/test/test_uri.py 67 # these should be hashable too - s = set([lit1, chk1, chk2]) - self.failUnlessEqual(len(s), 2) # since chk1==chk2 + s = set([lit1, chk1, chk2, unk]) + self.failUnlessEqual(len(s), 3) # since chk1==chk2 def test_is_uri(self): lit1 = uri.LiteralFileURI("some data").to_string() hunk ./src/allmydata/test/test_uri.py 75 self.failUnless(uri.is_uri(lit1)) self.failIf(uri.is_uri(None)) + def test_is_literal_file_uri(self): + lit1 = uri.LiteralFileURI("some data").to_string() + self.failUnless(uri.is_literal_file_uri(lit1)) + self.failIf(uri.is_literal_file_uri(None)) + self.failIf(uri.is_literal_file_uri("foo")) + self.failIf(uri.is_literal_file_uri("ro.foo")) + self.failIf(uri.is_literal_file_uri("URI:LITfoo")) + self.failUnless(uri.is_literal_file_uri("ro.URI:LIT:foo")) + self.failUnless(uri.is_literal_file_uri("imm.URI:LIT:foo")) + + def test_has_uri_prefix(self): + self.failUnless(uri.has_uri_prefix("URI:foo")) + self.failUnless(uri.has_uri_prefix("ro.URI:foo")) + self.failUnless(uri.has_uri_prefix("imm.URI:foo")) + self.failIf(uri.has_uri_prefix(None)) + self.failIf(uri.has_uri_prefix("foo")) + class CHKFile(unittest.TestCase): def test_pack(self): key = "\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" hunk ./src/allmydata/test/test_uri.py 117 self.failUnless(IFileURI.providedBy(u)) self.failIf(IDirnodeURI.providedBy(u)) self.failUnlessEqual(u.get_size(), 1234) - self.failUnless(u.is_readonly()) - self.failIf(u.is_mutable()) + u_ro = u.get_readonly() self.failUnlessIdentical(u, u_ro) he = u.to_human_encoding() hunk ./src/allmydata/test/test_uri.py 137 self.failUnless(IFileURI.providedBy(u2)) self.failIf(IDirnodeURI.providedBy(u2)) self.failUnlessEqual(u2.get_size(), 1234) - self.failUnless(u2.is_readonly()) - self.failIf(u2.is_mutable()) + + u2i = uri.from_string(u.to_string(), deep_immutable=True) + self.failUnlessEqual(u.to_string(), u2i.to_string()) + u2ro = uri.from_string(uri.ALLEGED_READONLY_PREFIX + u.to_string()) + self.failUnlessEqual(u.to_string(), u2ro.to_string()) + u2imm = uri.from_string(uri.ALLEGED_IMMUTABLE_PREFIX + u.to_string()) + self.failUnlessEqual(u.to_string(), u2imm.to_string()) v = u.get_verify_cap() self.failUnless(isinstance(v.to_string(), str)) hunk ./src/allmydata/test/test_uri.py 147 + self.failUnless(v.is_readonly()) + self.failIf(v.is_mutable()) + v2 = uri.from_string(v.to_string()) self.failUnlessEqual(v, v2) he = v.to_human_encoding() hunk ./src/allmydata/test/test_uri.py 162 total_shares=10, size=1234) self.failUnless(isinstance(v3.to_string(), str)) + self.failUnless(v3.is_readonly()) + self.failIf(v3.is_mutable()) def test_pack_badly(self): key = "\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" hunk ./src/allmydata/test/test_uri.py 217 self.failUnlessEqual(readable["UEB_hash"], base32.b2a(hashutil.uri_extension_hash(ext))) -class Invalid(unittest.TestCase): +class Unknown(unittest.TestCase): def test_from_future(self): # any URI type that we don't recognize should be treated as unknown future_uri = "I am a URI from the future. Whatever you do, don't " hunk ./src/allmydata/test/test_uri.py 224 u = uri.from_string(future_uri) self.failUnless(isinstance(u, uri.UnknownURI)) self.failUnlessEqual(u.to_string(), future_uri) + self.failUnless(u.get_readonly() is None) + self.failUnless(u.get_error() is None) + + u2 = uri.UnknownURI(future_uri, error=CapConstraintError("...")) + self.failUnlessEqual(u.to_string(), future_uri) + self.failUnless(u2.get_readonly() is None) + self.failUnless(isinstance(u2.get_error(), CapConstraintError)) class Constraint(unittest.TestCase): def test_constraint(self): hunk ./src/allmydata/test/test_uri.py 271 self.failUnless(IMutableFileURI.providedBy(u2)) self.failIf(IDirnodeURI.providedBy(u2)) + u2i = uri.from_string(u.to_string(), deep_immutable=True) + self.failUnless(isinstance(u2i, uri.UnknownURI), u2i) + u2ro = uri.from_string(uri.ALLEGED_READONLY_PREFIX + u.to_string()) + self.failUnless(isinstance(u2ro, uri.UnknownURI), u2ro) + u2imm = uri.from_string(uri.ALLEGED_IMMUTABLE_PREFIX + u.to_string()) + self.failUnless(isinstance(u2imm, uri.UnknownURI), u2imm) + u3 = u2.get_readonly() readkey = hashutil.ssk_readkey_hash(writekey) self.failUnlessEqual(u3.fingerprint, fingerprint) hunk ./src/allmydata/test/test_uri.py 288 self.failUnless(IMutableFileURI.providedBy(u3)) self.failIf(IDirnodeURI.providedBy(u3)) + u3i = uri.from_string(u3.to_string(), deep_immutable=True) + self.failUnless(isinstance(u3i, uri.UnknownURI), u3i) + u3ro = uri.from_string(uri.ALLEGED_READONLY_PREFIX + u3.to_string()) + self.failUnlessEqual(u3.to_string(), u3ro.to_string()) + u3imm = uri.from_string(uri.ALLEGED_IMMUTABLE_PREFIX + u3.to_string()) + self.failUnless(isinstance(u3imm, uri.UnknownURI), u3imm) + he = u3.to_human_encoding() u3_h = uri.ReadonlySSKFileURI.init_from_human_encoding(he) self.failUnlessEqual(u3, u3_h) hunk ./src/allmydata/test/test_uri.py 308 self.failUnless(IMutableFileURI.providedBy(u4)) self.failIf(IDirnodeURI.providedBy(u4)) + u4i = uri.from_string(u4.to_string(), deep_immutable=True) + self.failUnless(isinstance(u4i, uri.UnknownURI), u4i) + u4ro = uri.from_string(uri.ALLEGED_READONLY_PREFIX + u4.to_string()) + self.failUnlessEqual(u4.to_string(), u4ro.to_string()) + u4imm = uri.from_string(uri.ALLEGED_IMMUTABLE_PREFIX + u4.to_string()) + self.failUnless(isinstance(u4imm, uri.UnknownURI), u4imm) + u4a = uri.from_string(u4.to_string()) self.failUnlessEqual(u4a, u4) self.failUnless("ReadonlySSKFileURI" in str(u4a)) hunk ./src/allmydata/test/test_uri.py 357 self.failIf(IFileURI.providedBy(u2)) self.failUnless(IDirnodeURI.providedBy(u2)) + u2i = uri.from_string(u1.to_string(), deep_immutable=True) + self.failUnless(isinstance(u2i, uri.UnknownURI)) + u3 = u2.get_readonly() self.failUnless(u3.is_readonly()) self.failUnless(u3.is_mutable()) hunk ./src/allmydata/test/test_uri.py 366 self.failUnless(IURI.providedBy(u3)) self.failIf(IFileURI.providedBy(u3)) self.failUnless(IDirnodeURI.providedBy(u3)) + + u3i = uri.from_string(u2.to_string(), deep_immutable=True) + self.failUnless(isinstance(u3i, uri.UnknownURI)) + u3n = u3._filenode_uri self.failUnless(u3n.is_readonly()) self.failUnless(u3n.is_mutable()) hunk ./src/allmydata/test/test_uri.py 436 self.failIf(IFileURI.providedBy(u2)) self.failUnless(IDirnodeURI.providedBy(u2)) + u2i = uri.from_string(u1.to_string(), deep_immutable=True) + self.failUnlessEqual(u1.to_string(), u2i.to_string()) + u3 = u2.get_readonly() self.failUnlessEqual(u3.to_string(), u2.to_string()) self.failUnless(str(u3)) hunk ./src/allmydata/test/test_uri.py 443 + u3i = uri.from_string(u2.to_string(), deep_immutable=True) + self.failUnlessEqual(u2.to_string(), u3i.to_string()) + u2_verifier = u2.get_verify_cap() self.failUnless(isinstance(u2_verifier, uri.ImmutableDirectoryURIVerifier), hunk ./src/allmydata/test/test_web.py 10 from twisted.web import client, error, http from twisted.python import failure, log from nevow import rend -from allmydata import interfaces, uri, webish +from allmydata import interfaces, uri, webish, dirnode from allmydata.storage.shares import get_share_file from allmydata.storage_client import StorageFarmBroker from allmydata.immutable import upload, download hunk ./src/allmydata/test/test_web.py 21 from allmydata.scripts.debug import CorruptShareOptions, corrupt_share from allmydata.util import fileutil, base32 from allmydata.util.consumer import download_to_data +from allmydata.util.netstring import split_netstring from allmydata.test.common import FakeCHKFileNode, FakeMutableFileNode, \ create_chk_filenode, WebErrorMixin, ShouldFailMixin, make_mutable_file_uri from allmydata.interfaces import IMutableFileNode hunk ./src/allmydata/test/test_web.py 739 d = self.PUT(self.public_url + "/foo/new.txt", self.NEWFILE_CONTENTS) # TODO: we lose the response code, so we can't check this #self.failUnlessEqual(responsecode, 201) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"new.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(self._foo_node, u"new.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 750 self.NEWFILE_CONTENTS) # TODO: we lose the response code, so we can't check this #self.failUnlessEqual(responsecode, 201) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"new.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(self._foo_node, u"new.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 780 self.failIf(u.is_readonly()) return res d.addCallback(_check_uri) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"new.txt") + d.addCallback(self.failUnlessURIMatchesRWChild, self._foo_node, u"new.txt") d.addCallback(lambda res: self.failUnlessMutableChildContentsAre(self._foo_node, u"new.txt", hunk ./src/allmydata/test/test_web.py 800 d = self.PUT(self.public_url + "/foo/bar.txt", self.NEWFILE_CONTENTS) # TODO: we lose the response code, so we can't check this #self.failUnlessEqual(responsecode, 200) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"bar.txt") + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"bar.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(self._foo_node, u"bar.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 825 def test_PUT_NEWFILEURL_mkdirs(self): d = self.PUT(self.public_url + "/foo/newdir/new.txt", self.NEWFILE_CONTENTS) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, u"newdir/new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, fn, u"newdir/new.txt") d.addCallback(lambda res: self.failIfNodeHasChild(fn, u"new.txt")) d.addCallback(lambda res: self.failUnlessNodeHasChild(fn, u"newdir")) d.addCallback(lambda res: hunk ./src/allmydata/test/test_web.py 958 self.failUnless(re.search(get_sub, res), res) d.addCallback(_check) - # look at a directory which is readonly + # look at a readonly directory d.addCallback(lambda res: self.GET(self.public_url + "/reedownlee", followRedirect=True)) def _check2(res): hunk ./src/allmydata/test/test_web.py 1171 return d def test_POST_NEWDIRURL_initial_children(self): - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.POST2(self.public_url + "/foo/newdir?t=mkdir-with-children", simplejson.dumps(newkids)) def _check(uri): hunk ./src/allmydata/test/test_web.py 1178 n = self.s.create_node_from_uri(uri.strip()) d2 = self.failUnlessNodeKeysAre(n, newkids.keys()) d2.addCallback(lambda ign: - self.failUnlessChildURIIs(n, u"child-imm", filecap1)) + self.failUnlessROChildURIIs(n, u"child-imm", + caps['filecap1'])) + d2.addCallback(lambda ign: + self.failUnlessRWChildURIIs(n, u"child-mutable", + caps['filecap2'])) + d2.addCallback(lambda ign: + self.failUnlessROChildURIIs(n, u"child-mutable-ro", + caps['filecap3'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 1187 - self.failUnlessChildURIIs(n, u"child-mutable", - filecap2)) + self.failUnlessROChildURIIs(n, u"unknownchild-ro", + caps['unknown_rocap'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 1190 - self.failUnlessChildURIIs(n, u"child-mutable-ro", - filecap3)) + self.failUnlessRWChildURIIs(n, u"unknownchild-rw", + caps['unknown_rwcap'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 1193 - self.failUnlessChildURIIs(n, u"dirchild", dircap)) + self.failUnlessROChildURIIs(n, u"unknownchild-imm", + caps['unknown_immcap'])) + d2.addCallback(lambda ign: + self.failUnlessRWChildURIIs(n, u"dirchild", + caps['dircap'])) return d2 d.addCallback(_check) d.addCallback(lambda res: hunk ./src/allmydata/test/test_web.py 1205 d.addCallback(lambda res: self._foo_node.get(u"newdir")) d.addCallback(self.failUnlessNodeKeysAre, newkids.keys()) d.addCallback(lambda res: self._foo_node.get(u"newdir")) - d.addCallback(self.failUnlessChildURIIs, u"child-imm", filecap1) + d.addCallback(self.failUnlessROChildURIIs, u"child-imm", caps['filecap1']) return d def test_POST_NEWDIRURL_immutable(self): hunk ./src/allmydata/test/test_web.py 1209 - (newkids, filecap1, immdircap) = self._create_immutable_children() + (newkids, caps) = self._create_immutable_children() d = self.POST2(self.public_url + "/foo/newdir?t=mkdir-immutable", simplejson.dumps(newkids)) def _check(uri): hunk ./src/allmydata/test/test_web.py 1216 n = self.s.create_node_from_uri(uri.strip()) d2 = self.failUnlessNodeKeysAre(n, newkids.keys()) d2.addCallback(lambda ign: - self.failUnlessChildURIIs(n, u"child-imm", filecap1)) + self.failUnlessROChildURIIs(n, u"child-imm", + caps['filecap1'])) + d2.addCallback(lambda ign: + self.failUnlessROChildURIIs(n, u"unknownchild-imm", + caps['unknown_immcap'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 1222 - self.failUnlessChildURIIs(n, u"dirchild-imm", - immdircap)) + self.failUnlessROChildURIIs(n, u"dirchild-imm", + caps['immdircap'])) return d2 d.addCallback(_check) d.addCallback(lambda res: hunk ./src/allmydata/test/test_web.py 1231 d.addCallback(lambda res: self._foo_node.get(u"newdir")) d.addCallback(self.failUnlessNodeKeysAre, newkids.keys()) d.addCallback(lambda res: self._foo_node.get(u"newdir")) - d.addCallback(self.failUnlessChildURIIs, u"child-imm", filecap1) + d.addCallback(self.failUnlessROChildURIIs, u"child-imm", caps['filecap1']) + d.addCallback(lambda res: self._foo_node.get(u"newdir")) + d.addCallback(self.failUnlessROChildURIIs, u"unknownchild-imm", caps['unknown_immcap']) d.addCallback(lambda res: self._foo_node.get(u"newdir")) hunk ./src/allmydata/test/test_web.py 1235 - d.addCallback(self.failUnlessChildURIIs, u"dirchild-imm", immdircap) + d.addCallback(self.failUnlessROChildURIIs, u"dirchild-imm", caps['immdircap']) d.addErrback(self.explain_web_error) return d hunk ./src/allmydata/test/test_web.py 1240 def test_POST_NEWDIRURL_immutable_bad(self): - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.shouldFail2(error.Error, "test_POST_NEWDIRURL_immutable_bad", "400 Bad Request", hunk ./src/allmydata/test/test_web.py 1243 - "a mkdir-immutable operation was given a child that was not itself immutable", + "needed to be immutable but was not", self.POST2, self.public_url + "/foo/newdir?t=mkdir-immutable", simplejson.dumps(newkids)) hunk ./src/allmydata/test/test_web.py 1365 d.addCallback(_check) return d - def failUnlessChildURIIs(self, node, name, expected_uri): + def failUnlessRWChildURIIs(self, node, name, expected_uri): assert isinstance(name, unicode) d = node.get_child_at_path(name) def _check(child): hunk ./src/allmydata/test/test_web.py 1369 + self.failUnless(child.is_unknown() or not child.is_readonly()) self.failUnlessEqual(child.get_uri(), expected_uri.strip()) hunk ./src/allmydata/test/test_web.py 1371 + self.failUnlessEqual(child.get_write_uri(), expected_uri.strip()) + expected_ro_uri = self._make_readonly(expected_uri) + if expected_ro_uri: + self.failUnlessEqual(child.get_readonly_uri(), expected_ro_uri.strip()) + d.addCallback(_check) + return d + + def failUnlessROChildURIIs(self, node, name, expected_uri): + assert isinstance(name, unicode) + d = node.get_child_at_path(name) + def _check(child): + self.failUnless(child.is_unknown() or child.is_readonly()) + self.failUnlessEqual(child.get_write_uri(), None) + self.failUnlessEqual(child.get_uri(), expected_uri.strip()) + self.failUnlessEqual(child.get_readonly_uri(), expected_uri.strip()) + d.addCallback(_check) + return d + + def failUnlessURIMatchesRWChild(self, got_uri, node, name): + assert isinstance(name, unicode) + d = node.get_child_at_path(name) + def _check(child): + self.failUnless(child.is_unknown() or not child.is_readonly()) + self.failUnlessEqual(child.get_uri(), got_uri.strip()) + self.failUnlessEqual(child.get_write_uri(), got_uri.strip()) + expected_ro_uri = self._make_readonly(got_uri) + if expected_ro_uri: + self.failUnlessEqual(child.get_readonly_uri(), expected_ro_uri.strip()) d.addCallback(_check) return d hunk ./src/allmydata/test/test_web.py 1402 - def failUnlessURIMatchesChild(self, got_uri, node, name): + def failUnlessURIMatchesROChild(self, got_uri, node, name): assert isinstance(name, unicode) d = node.get_child_at_path(name) def _check(child): hunk ./src/allmydata/test/test_web.py 1406 + self.failUnless(child.is_unknown() or child.is_readonly()) + self.failUnlessEqual(child.get_write_uri(), None) self.failUnlessEqual(got_uri.strip(), child.get_uri()) hunk ./src/allmydata/test/test_web.py 1409 + self.failUnlessEqual(got_uri.strip(), child.get_readonly_uri()) d.addCallback(_check) return d hunk ./src/allmydata/test/test_web.py 1420 d = self.POST(self.public_url + "/foo", t="upload", file=("new.txt", self.NEWFILE_CONTENTS)) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, u"new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, fn, u"new.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(fn, u"new.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1431 d = self.POST(self.public_url + "/foo", t="upload", file=(filename, self.NEWFILE_CONTENTS)) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, filename) + d.addCallback(self.failUnlessURIMatchesROChild, fn, filename) d.addCallback(lambda res: self.failUnlessChildContentsAre(fn, filename, self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1448 name=filename, file=("overridden", self.NEWFILE_CONTENTS)) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, filename) + d.addCallback(self.failUnlessURIMatchesROChild, fn, filename) d.addCallback(lambda res: self.failUnlessChildContentsAre(fn, filename, self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1550 d = self.POST(self.public_url + "/foo", t="upload", mutable="true", file=("new.txt", self.NEWFILE_CONTENTS)) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, u"new.txt") + d.addCallback(self.failUnlessURIMatchesRWChild, fn, u"new.txt") d.addCallback(lambda res: self.failUnlessMutableChildContentsAre(fn, u"new.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1569 self.POST(self.public_url + "/foo", t="upload", mutable="true", file=("new.txt", NEWER_CONTENTS))) - d.addCallback(self.failUnlessURIMatchesChild, fn, u"new.txt") + d.addCallback(self.failUnlessURIMatchesRWChild, fn, u"new.txt") d.addCallback(lambda res: self.failUnlessMutableChildContentsAre(fn, u"new.txt", NEWER_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1585 NEW2_CONTENTS = NEWER_CONTENTS + "overwrite with PUT\n" d.addCallback(lambda res: self.PUT(self.public_url + "/foo/new.txt", NEW2_CONTENTS)) - d.addCallback(self.failUnlessURIMatchesChild, fn, u"new.txt") + d.addCallback(self.failUnlessURIMatchesRWChild, fn, u"new.txt") d.addCallback(lambda res: self.failUnlessMutableChildContentsAre(fn, u"new.txt", NEW2_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1714 d = self.POST(self.public_url + "/foo", t="upload", file=("bar.txt", self.NEWFILE_CONTENTS)) fn = self._foo_node - d.addCallback(self.failUnlessURIMatchesChild, fn, u"bar.txt") + d.addCallback(self.failUnlessURIMatchesROChild, fn, u"bar.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(fn, u"bar.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 1765 fn = self._foo_node d = self.POST(self.public_url + "/foo", t="upload", name="new.txt", file=self.NEWFILE_CONTENTS) - d.addCallback(self.failUnlessURIMatchesChild, fn, u"new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, fn, u"new.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(fn, u"new.txt", self.NEWFILE_CONTENTS)) hunk ./src/allmydata/test/test_web.py 2028 return d def test_POST_mkdir_initial_children(self): - newkids, filecap1, ign, ign, ign = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.POST2(self.public_url + "/foo?t=mkdir-with-children&name=newdir", simplejson.dumps(newkids)) hunk ./src/allmydata/test/test_web.py 2037 d.addCallback(lambda res: self._foo_node.get(u"newdir")) d.addCallback(self.failUnlessNodeKeysAre, newkids.keys()) d.addCallback(lambda res: self._foo_node.get(u"newdir")) - d.addCallback(self.failUnlessChildURIIs, u"child-imm", filecap1) + d.addCallback(self.failUnlessROChildURIIs, u"child-imm", caps['filecap1']) return d def test_POST_mkdir_immutable(self): hunk ./src/allmydata/test/test_web.py 2041 - (newkids, filecap1, immdircap) = self._create_immutable_children() + (newkids, caps) = self._create_immutable_children() d = self.POST2(self.public_url + "/foo?t=mkdir-immutable&name=newdir", simplejson.dumps(newkids)) hunk ./src/allmydata/test/test_web.py 2050 d.addCallback(lambda res: self._foo_node.get(u"newdir")) d.addCallback(self.failUnlessNodeKeysAre, newkids.keys()) d.addCallback(lambda res: self._foo_node.get(u"newdir")) - d.addCallback(self.failUnlessChildURIIs, u"child-imm", filecap1) + d.addCallback(self.failUnlessROChildURIIs, u"child-imm", caps['filecap1']) + d.addCallback(lambda res: self._foo_node.get(u"newdir")) + d.addCallback(self.failUnlessROChildURIIs, u"unknownchild-imm", caps['unknown_immcap']) d.addCallback(lambda res: self._foo_node.get(u"newdir")) hunk ./src/allmydata/test/test_web.py 2054 - d.addCallback(self.failUnlessChildURIIs, u"dirchild-imm", immdircap) + d.addCallback(self.failUnlessROChildURIIs, u"dirchild-imm", caps['immdircap']) return d def test_POST_mkdir_immutable_bad(self): hunk ./src/allmydata/test/test_web.py 2058 - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.shouldFail2(error.Error, "test_POST_mkdir_immutable_bad", "400 Bad Request", hunk ./src/allmydata/test/test_web.py 2061 - "a mkdir-immutable operation was given a child that was not itself immutable", + "needed to be immutable but was not", self.POST2, self.public_url + "/foo?t=mkdir-immutable&name=newdir", hunk ./src/allmydata/test/test_web.py 2120 d.addErrback(self.explain_web_error) return d + def _make_readonly(self, u): + ro_uri = uri.from_string(u).get_readonly() + if ro_uri is None: + return None + return ro_uri.to_string() + def _create_initial_children(self): contents, n, filecap1 = self.makefile(12) md1 = {"metakey1": "metavalue1"} hunk ./src/allmydata/test/test_web.py 2132 filecap2 = make_mutable_file_uri() node3 = self.s.create_node_from_uri(make_mutable_file_uri()) filecap3 = node3.get_readonly_uri() + unknown_rwcap = "lafs://from_the_future" + unknown_rocap = "ro.lafs://readonly_from_the_future" + unknown_immcap = "imm.lafs://immutable_from_the_future" node4 = self.s.create_node_from_uri(make_mutable_file_uri()) dircap = DirectoryNode(node4, None, None).get_uri() hunk ./src/allmydata/test/test_web.py 2137 - newkids = {u"child-imm": ["filenode", {"ro_uri": filecap1, - "metadata": md1, }], - u"child-mutable": ["filenode", {"rw_uri": filecap2}], + newkids = {u"child-imm": ["filenode", {"rw_uri": filecap1, + "ro_uri": self._make_readonly(filecap1), + "metadata": md1, }], + u"child-mutable": ["filenode", {"rw_uri": filecap2, + "ro_uri": self._make_readonly(filecap2)}], u"child-mutable-ro": ["filenode", {"ro_uri": filecap3}], hunk ./src/allmydata/test/test_web.py 2143 - u"dirchild": ["dirnode", {"rw_uri": dircap}], + u"unknownchild-rw": ["unknown", {"rw_uri": unknown_rwcap, + "ro_uri": unknown_rocap}], + u"unknownchild-ro": ["unknown", {"ro_uri": unknown_rocap}], + u"unknownchild-imm": ["unknown", {"ro_uri": unknown_immcap}], + u"dirchild": ["dirnode", {"rw_uri": dircap, + "ro_uri": self._make_readonly(dircap)}], } hunk ./src/allmydata/test/test_web.py 2150 - return newkids, filecap1, filecap2, filecap3, dircap + return newkids, {'filecap1': filecap1, + 'filecap2': filecap2, + 'filecap3': filecap3, + 'unknown_rwcap': unknown_rwcap, + 'unknown_rocap': unknown_rocap, + 'unknown_immcap': unknown_immcap, + 'dircap': dircap} def _create_immutable_children(self): contents, n, filecap1 = self.makefile(12) hunk ./src/allmydata/test/test_web.py 2164 tnode = create_chk_filenode("immutable directory contents\n"*10) dnode = DirectoryNode(tnode, None, None) assert not dnode.is_mutable() + unknown_immcap = "imm.lafs://immutable_from_the_future" immdircap = dnode.get_uri() hunk ./src/allmydata/test/test_web.py 2166 - newkids = {u"child-imm": ["filenode", {"ro_uri": filecap1, - "metadata": md1, }], - u"dirchild-imm": ["dirnode", {"ro_uri": immdircap}], + newkids = {u"child-imm": ["filenode", {"ro_uri": filecap1, + "metadata": md1, }], + u"unknownchild-imm": ["unknown", {"ro_uri": unknown_immcap}], + u"dirchild-imm": ["dirnode", {"ro_uri": immdircap}], } hunk ./src/allmydata/test/test_web.py 2171 - return newkids, filecap1, immdircap + return newkids, {'filecap1': filecap1, + 'unknown_immcap': unknown_immcap, + 'immdircap': immdircap} def test_POST_mkdir_no_parentdir_initial_children(self): hunk ./src/allmydata/test/test_web.py 2176 - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.POST2("/uri?t=mkdir-with-children", simplejson.dumps(newkids)) def _after_mkdir(res): self.failUnless(res.startswith("URI:DIR"), res) hunk ./src/allmydata/test/test_web.py 2183 n = self.s.create_node_from_uri(res) d2 = self.failUnlessNodeKeysAre(n, newkids.keys()) d2.addCallback(lambda ign: - self.failUnlessChildURIIs(n, u"child-imm", filecap1)) + self.failUnlessROChildURIIs(n, u"child-imm", + caps['filecap1'])) + d2.addCallback(lambda ign: + self.failUnlessRWChildURIIs(n, u"child-mutable", + caps['filecap2'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 2189 - self.failUnlessChildURIIs(n, u"child-mutable", - filecap2)) + self.failUnlessROChildURIIs(n, u"child-mutable-ro", + caps['filecap3'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 2192 - self.failUnlessChildURIIs(n, u"child-mutable-ro", - filecap3)) + self.failUnlessRWChildURIIs(n, u"unknownchild-rw", + caps['unknown_rwcap'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 2195 - self.failUnlessChildURIIs(n, u"dirchild", dircap)) + self.failUnlessROChildURIIs(n, u"unknownchild-ro", + caps['unknown_rocap'])) + d2.addCallback(lambda ign: + self.failUnlessROChildURIIs(n, u"unknownchild-imm", + caps['unknown_immcap'])) + d2.addCallback(lambda ign: + self.failUnlessRWChildURIIs(n, u"dirchild", + caps['dircap'])) return d2 d.addCallback(_after_mkdir) return d hunk ./src/allmydata/test/test_web.py 2210 def test_POST_mkdir_no_parentdir_unexpected_children(self): # the regular /uri?t=mkdir operation is specified to ignore its body. # Only t=mkdir-with-children pays attention to it. - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.shouldHTTPError("POST t=mkdir unexpected children", 400, "Bad Request", "t=mkdir does not accept children=, " hunk ./src/allmydata/test/test_web.py 2227 return d def test_POST_mkdir_no_parentdir_immutable(self): - (newkids, filecap1, immdircap) = self._create_immutable_children() + (newkids, caps) = self._create_immutable_children() d = self.POST2("/uri?t=mkdir-immutable", simplejson.dumps(newkids)) def _after_mkdir(res): self.failUnless(res.startswith("URI:DIR"), res) hunk ./src/allmydata/test/test_web.py 2234 n = self.s.create_node_from_uri(res) d2 = self.failUnlessNodeKeysAre(n, newkids.keys()) d2.addCallback(lambda ign: - self.failUnlessChildURIIs(n, u"child-imm", filecap1)) + self.failUnlessROChildURIIs(n, u"child-imm", + caps['filecap1'])) + d2.addCallback(lambda ign: + self.failUnlessROChildURIIs(n, u"unknownchild-imm", + caps['unknown_immcap'])) d2.addCallback(lambda ign: hunk ./src/allmydata/test/test_web.py 2240 - self.failUnlessChildURIIs(n, u"dirchild-imm", - immdircap)) + self.failUnlessROChildURIIs(n, u"dirchild-imm", + caps['immdircap'])) return d2 d.addCallback(_after_mkdir) return d hunk ./src/allmydata/test/test_web.py 2247 def test_POST_mkdir_no_parentdir_immutable_bad(self): - (newkids, filecap1, filecap2, filecap3, - dircap) = self._create_initial_children() + (newkids, caps) = self._create_initial_children() d = self.shouldFail2(error.Error, "test_POST_mkdir_no_parentdir_immutable_bad", "400 Bad Request", hunk ./src/allmydata/test/test_web.py 2251 - "a mkdir-immutable operation was given a child that was not itself immutable", + "needed to be immutable but was not", self.POST2, "/uri?t=mkdir-immutable", simplejson.dumps(newkids)) hunk ./src/allmydata/test/test_web.py 2359 d = client.getPage(url, method="POST", postdata=reqbody) def _then(res): - self.failUnlessURIMatchesChild(newuri9, self._foo_node, u"atomic_added_1") - self.failUnlessURIMatchesChild(newuri10, self._foo_node, u"atomic_added_2") - self.failUnlessURIMatchesChild(newuri11, self._foo_node, u"atomic_added_3") + self.failUnlessURIMatchesROChild(newuri9, self._foo_node, u"atomic_added_1") + self.failUnlessURIMatchesROChild(newuri10, self._foo_node, u"atomic_added_2") + self.failUnlessURIMatchesROChild(newuri11, self._foo_node, u"atomic_added_3") d.addCallback(_then) d.addErrback(self.dump_error) hunk ./src/allmydata/test/test_web.py 2373 def test_POST_put_uri(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo", t="uri", name="new.txt", uri=newuri) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"new.txt") + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"new.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(self._foo_node, u"new.txt", contents)) hunk ./src/allmydata/test/test_web.py 2382 def test_POST_put_uri_replace(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo", t="uri", name="bar.txt", uri=newuri) - d.addCallback(self.failUnlessURIMatchesChild, self._foo_node, u"bar.txt") + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"bar.txt") d.addCallback(lambda res: self.failUnlessChildContentsAre(self._foo_node, u"bar.txt", contents)) hunk ./src/allmydata/test/test_web.py 2611 d.addCallback(lambda res: self.failUnlessEqual(res.strip(), new_uri)) d.addCallback(lambda res: - self.failUnlessChildURIIs(self.public_root, - u"foo", - new_uri)) + self.failUnlessRWChildURIIs(self.public_root, + u"foo", + new_uri)) return d d.addCallback(_made_dir) return d hunk ./src/allmydata/test/test_web.py 2630 self.public_url + "/foo?t=uri&replace=false", new_uri) d.addCallback(lambda res: - self.failUnlessChildURIIs(self.public_root, - u"foo", - self._foo_uri)) + self.failUnlessRWChildURIIs(self.public_root, + u"foo", + self._foo_uri)) return d d.addCallback(_made_dir) return d hunk ./src/allmydata/test/test_web.py 2642 "400 Bad Request", "PUT to a directory", self.PUT, self.public_url + "/foo?t=BOGUS", "") d.addCallback(lambda res: - self.failUnlessChildURIIs(self.public_root, - u"foo", - self._foo_uri)) + self.failUnlessRWChildURIIs(self.public_root, + u"foo", + self._foo_uri)) return d def test_PUT_NEWFILEURL_uri(self): hunk ./src/allmydata/test/test_web.py 3171 d.addErrback(self.explain_web_error) return d - def test_unknown(self): + def test_unknown(self, immutable=False): self.basedir = "web/Grid/unknown" hunk ./src/allmydata/test/test_web.py 3173 + if immutable: + self.basedir = "web/Grid/unknown-immutable" + self.set_up_grid() c0 = self.g.clients[0] self.uris = {} hunk ./src/allmydata/test/test_web.py 3181 self.fileurls = {} - future_writecap = "x-tahoe-crazy://I_am_from_the_future." - future_readcap = "x-tahoe-crazy-readonly://I_am_from_the_future." + future_write_uri = "x-tahoe-crazy://I_am_from_the_future." + future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." # the future cap format may contain slashes, which must be tolerated hunk ./src/allmydata/test/test_web.py 3184 - expected_info_url = "uri/%s?t=info" % urllib.quote(future_writecap, + expected_info_url = "uri/%s?t=info" % urllib.quote(future_write_uri, safe="") hunk ./src/allmydata/test/test_web.py 3186 - future_node = UnknownNode(future_writecap, future_readcap) + + if immutable: + name = u"future-imm" + future_node = UnknownNode(None, future_read_uri, deep_immutable=True) + d = c0.create_immutable_dirnode({name: (future_node, {})}) + else: + name = u"future" + future_node = UnknownNode(future_write_uri, future_read_uri) + d = c0.create_dirnode() hunk ./src/allmydata/test/test_web.py 3196 - d = c0.create_dirnode() def _stash_root_and_create_file(n): self.rootnode = n self.rooturl = "uri/" + urllib.quote(n.get_uri()) + "/" hunk ./src/allmydata/test/test_web.py 3200 self.rourl = "uri/" + urllib.quote(n.get_readonly_uri()) + "/" - return self.rootnode.set_node(u"future", future_node) + if not immutable: + return self.rootnode.set_node(name, future_node) d.addCallback(_stash_root_and_create_file) hunk ./src/allmydata/test/test_web.py 3203 + # make sure directory listing tolerates unknown nodes d.addCallback(lambda ign: self.GET(self.rooturl)) hunk ./src/allmydata/test/test_web.py 3206 - def _check_html(res): - self.failUnlessIn("future", res) - # find the More Info link for "future", should be relative + def _check_directory_html(res): + self.failUnlessIn("%s" % (str(name),), res) + # find the More Info link for name, should be relative mo = re.search(r'More Info', res) info_url = mo.group(1) hunk ./src/allmydata/test/test_web.py 3211 - self.failUnlessEqual(info_url, "future?t=info") + self.failUnlessEqual(info_url, "%s?t=info" % (str(name),)) + d.addCallback(_check_directory_html) hunk ./src/allmydata/test/test_web.py 3214 - d.addCallback(_check_html) d.addCallback(lambda ign: self.GET(self.rooturl+"?t=json")) hunk ./src/allmydata/test/test_web.py 3215 - def _check_json(res, expect_writecap): + def _check_directory_json(res, expect_rw_uri): data = simplejson.loads(res) self.failUnlessEqual(data[0], "dirnode") hunk ./src/allmydata/test/test_web.py 3218 - f = data[1]["children"]["future"] + f = data[1]["children"][name] self.failUnlessEqual(f[0], "unknown") hunk ./src/allmydata/test/test_web.py 3220 - if expect_writecap: - self.failUnlessEqual(f[1]["rw_uri"], future_writecap) + if expect_rw_uri: + self.failUnlessEqual(f[1]["rw_uri"], future_write_uri) else: self.failIfIn("rw_uri", f[1]) hunk ./src/allmydata/test/test_web.py 3224 - self.failUnlessEqual(f[1]["ro_uri"], future_readcap) + self.failUnlessEqual(f[1]["ro_uri"], + ("imm." if immutable else "ro.") + future_read_uri) self.failUnless("metadata" in f[1]) hunk ./src/allmydata/test/test_web.py 3227 - d.addCallback(_check_json, expect_writecap=True) - d.addCallback(lambda ign: self.GET(expected_info_url)) - def _check_info(res, expect_readcap): + d.addCallback(_check_directory_json, expect_rw_uri=not immutable) + + def _check_info(res, expect_rw_uri, expect_ro_uri): self.failUnlessIn("Object Type: unknown", res) hunk ./src/allmydata/test/test_web.py 3231 - self.failUnlessIn(future_writecap, res) - if expect_readcap: - self.failUnlessIn(future_readcap, res) + if expect_rw_uri: + self.failUnlessIn(future_write_uri, res) + if expect_ro_uri: + self.failUnlessIn(future_read_uri, res) + else: + self.failIfIn(future_read_uri, res) self.failIfIn("Raw data as", res) self.failIfIn("Directory writecap", res) self.failIfIn("Checker Operations", res) hunk ./src/allmydata/test/test_web.py 3242 self.failIfIn("Mutable File Operations", res) self.failIfIn("Directory Operations", res) - d.addCallback(_check_info, expect_readcap=False) - d.addCallback(lambda ign: self.GET(self.rooturl+"future?t=info")) - d.addCallback(_check_info, expect_readcap=True) + + # FIXME: these should have expect_rw_uri=not immutable; I don't know + # why they fail. Possibly related to ticket #922. + + d.addCallback(lambda ign: self.GET(expected_info_url)) + d.addCallback(_check_info, expect_rw_uri=False, expect_ro_uri=False) + d.addCallback(lambda ign: self.GET("%s%s?t=info" % (self.rooturl, str(name)))) + d.addCallback(_check_info, expect_rw_uri=False, expect_ro_uri=True) + + def _check_json(res, expect_rw_uri): + data = simplejson.loads(res) + self.failUnlessEqual(data[0], "unknown") + if expect_rw_uri: + self.failUnlessEqual(data[1]["rw_uri"], future_write_uri) + else: + self.failIfIn("rw_uri", data[1]) + self.failUnlessEqual(data[1]["ro_uri"], + ("imm." if immutable else "ro.") + future_read_uri) + # TODO: check metadata contents + self.failUnless("metadata" in data[1]) + + d.addCallback(lambda ign: self.GET("%s%s?t=json" % (self.rooturl, str(name)))) + d.addCallback(_check_json, expect_rw_uri=not immutable) # and make sure that a read-only version of the directory can be hunk ./src/allmydata/test/test_web.py 3267 - # rendered too. This version will not have future_writecap + # rendered too. This version will not have future_write_uri, whether + # or not future_node was immutable. d.addCallback(lambda ign: self.GET(self.rourl)) hunk ./src/allmydata/test/test_web.py 3270 - d.addCallback(_check_html) + d.addCallback(_check_directory_html) d.addCallback(lambda ign: self.GET(self.rourl+"?t=json")) hunk ./src/allmydata/test/test_web.py 3272 - d.addCallback(_check_json, expect_writecap=False) + d.addCallback(_check_directory_json, expect_rw_uri=False) + + d.addCallback(lambda ign: self.GET("%s%s?t=json" % (self.rourl, str(name)))) + d.addCallback(_check_json, expect_rw_uri=False) + + # TODO: check that getting t=info from the Info link in the ro directory + # works, and does not include the writecap URI. + return d + + def test_immutable_unknown(self): + return self.test_unknown(immutable=True) + + def test_mutant_dirnodes_are_omitted(self): + self.basedir = "web/Grid/mutant_dirnodes_are_omitted" + + self.set_up_grid() + c = self.g.clients[0] + nm = c.nodemaker + self.uris = {} + self.fileurls = {} + + lonely_uri = "URI:LIT:n5xgk" # LIT for "one" + mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" + mut_read_uri = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + + # This method tests mainly dirnode, but we'd have to duplicate code in order to + # test the dirnode and web layers separately. + + # 'lonely' is a valid LIT child, 'ro' is a mutant child with an SSK-RO readcap, + # and 'write-in-ro' is a mutant child with an SSK writecap in the ro_uri field. + # When the directory is read, the mutants should be silently disposed of, leaving + # their lonely sibling. + # We don't test the case of a retrieving a cap from the encrypted rw_uri field, + # because immutable directories don't have a writecap and therefore that field + # isn't (and can't be) decrypted. + # TODO: The field still exists in the netstring. Technically we should check what + # happens if something is put there (it should be ignored), but that can wait. + + lonely_child = nm.create_from_cap(lonely_uri) + mutant_ro_child = nm.create_from_cap(mut_read_uri) + mutant_write_in_ro_child = nm.create_from_cap(mut_write_uri) + + def _by_hook_or_by_crook(): + return True + for n in [mutant_ro_child, mutant_write_in_ro_child]: + n.is_allowed_in_immutable_directory = _by_hook_or_by_crook + + mutant_write_in_ro_child.get_write_uri = lambda: None + mutant_write_in_ro_child.get_readonly_uri = lambda: mut_write_uri + + kids = {u"lonely": (lonely_child, {}), + u"ro": (mutant_ro_child, {}), + u"write-in-ro": (mutant_write_in_ro_child, {}), + } + d = c.create_immutable_dirnode(kids) + + def _created(dn): + self.failUnless(isinstance(dn, dirnode.DirectoryNode)) + self.failIf(dn.is_mutable()) + self.failUnless(dn.is_readonly()) + # This checks that if we somehow ended up calling dn._decrypt_rwcapdata, it would fail. + self.failIf(hasattr(dn._node, 'get_writekey')) + rep = str(dn) + self.failUnless("RO-IMM" in rep) + cap = dn.get_cap() + self.failUnlessIn("CHK", cap.to_string()) + self.cap = cap + self.rootnode = dn + self.rooturl = "uri/" + urllib.quote(dn.get_uri()) + "/" + return download_to_data(dn._node) + d.addCallback(_created) + + def _check_data(data): + # Decode the netstring representation of the directory to check that all children + # are present. This is a bit of an abstraction violation, but there's not really + # any other way to do it given that the real DirectoryNode._unpack_contents would + # strip the mutant children out (which is what we're trying to test, later). + position = 0 + numkids = 0 + while position < len(data): + entries, position = split_netstring(data, 1, position) + entry = entries[0] + (name, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + name = name.decode("utf-8") + self.failUnless(rwcapdata == "") + ro_uri = ro_uri.strip() + if name in kids: + self.failIfEqual(ro_uri, "") + (expected_child, ign) = kids[name] + self.failUnlessEqual(ro_uri, expected_child.get_readonly_uri()) + numkids += 1 + + self.failUnlessEqual(numkids, 3) + return self.rootnode.list() + d.addCallback(_check_data) + + # Now when we use the real directory listing code, the mutants should be absent. + def _check_kids(children): + self.failUnlessEqual(sorted(children.keys()), [u"lonely"]) + lonely_node, lonely_metadata = children[u"lonely"] + + self.failUnlessEqual(lonely_node.get_write_uri(), None) + self.failUnlessEqual(lonely_node.get_readonly_uri(), lonely_uri) + d.addCallback(_check_kids) + + d.addCallback(lambda ign: nm.create_from_cap(self.cap.to_string())) + d.addCallback(lambda n: n.list()) + d.addCallback(_check_kids) # again with dirnode recreated from cap + + # Make sure the lonely child can be listed in HTML... + d.addCallback(lambda ign: self.GET(self.rooturl)) + def _check_html(res): + self.failIfIn("URI:SSK", res) + get_lonely = "".join([r'FILE', + r'\s+', + r'lonely' % (urllib.quote(lonely_uri),), + r'', + r'\s+%d' % len("one"), + ]) + self.failUnless(re.search(get_lonely, res), res) + + # find the More Info link for name, should be relative + mo = re.search(r'More Info', res) + info_url = mo.group(1) + self.failUnless(info_url.endswith(urllib.quote(lonely_uri) + "?t=info"), info_url) + d.addCallback(_check_html) + + # ... and in JSON. + d.addCallback(lambda ign: self.GET(self.rooturl+"?t=json")) + def _check_json(res): + data = simplejson.loads(res) + self.failUnlessEqual(data[0], "dirnode") + listed_children = data[1]["children"] + self.failUnlessEqual(sorted(listed_children.keys()), [u"lonely"]) + ll_type, ll_data = listed_children[u"lonely"] + self.failUnlessEqual(ll_type, "filenode") + self.failIf("rw_uri" in ll_data) + self.failUnlessEqual(ll_data["ro_uri"], lonely_uri) + d.addCallback(_check_json) return d def test_deep_check(self): hunk ./src/allmydata/test/test_web.py 3443 # this tests that deep-check and stream-manifest will ignore # UnknownNode instances. Hopefully this will also cover deep-stats. - future_writecap = "x-tahoe-crazy://I_am_from_the_future." - future_readcap = "x-tahoe-crazy-readonly://I_am_from_the_future." - future_node = UnknownNode(future_writecap, future_readcap) - d.addCallback(lambda ign: self.rootnode.set_node(u"future",future_node)) + future_write_uri = "x-tahoe-crazy://I_am_from_the_future." + future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." + future_node = UnknownNode(future_write_uri, future_read_uri) + d.addCallback(lambda ign: self.rootnode.set_node(u"future", future_node)) def _clobber_shares(ignored): self.delete_shares_numbered(self.uris["sick"], [0,1]) hunk ./src/allmydata/unknown.py 1 + from zope.interface import implements from twisted.internet import defer hunk ./src/allmydata/unknown.py 4 -from allmydata.interfaces import IFilesystemNode +from allmydata.interfaces import IFilesystemNode, MustNotBeUnknownRWError, \ + MustBeDeepImmutableError +from allmydata import uri +from allmydata.uri import ALLEGED_READONLY_PREFIX, ALLEGED_IMMUTABLE_PREFIX + + +# See ticket #833 for design rationale of UnknownNodes. + +def strip_prefix_for_ro(ro_uri, deep_immutable): + """Strip prefixes when storing an URI in a ro_uri slot.""" + + # It is possible for an alleged-immutable URI to be put into a + # mutable directory. In that case the ALLEGED_IMMUTABLE_PREFIX + # should not be stripped. In other cases, the prefix can safely + # be stripped because it is implied by the context. + + if ro_uri.startswith(ALLEGED_IMMUTABLE_PREFIX): + if not deep_immutable: + return ro_uri + return ro_uri[len(ALLEGED_IMMUTABLE_PREFIX):] + elif ro_uri.startswith(ALLEGED_READONLY_PREFIX): + return ro_uri[len(ALLEGED_READONLY_PREFIX):] + else: + return ro_uri class UnknownNode: implements(IFilesystemNode) hunk ./src/allmydata/unknown.py 31 - def __init__(self, writecap, readcap): - assert writecap is None or isinstance(writecap, str) - self.writecap = writecap - assert readcap is None or isinstance(readcap, str) - self.readcap = readcap + + def __init__(self, given_rw_uri, given_ro_uri, deep_immutable=False, + name=u""): + assert given_rw_uri is None or isinstance(given_rw_uri, str) + assert given_ro_uri is None or isinstance(given_ro_uri, str) + given_rw_uri = given_rw_uri or None + given_ro_uri = given_ro_uri or None + + # We don't raise errors when creating an UnknownNode; we instead create an + # opaque node (with rw_uri and ro_uri both None) that records the error. + # This avoids breaking operations that never store the opaque node. + # Note that this means that if a stored dirnode has only a rw_uri, it + # might be dropped. Any future "write-only" cap formats should have a dummy + # unusable readcap to stop that from happening. + + self.error = None + self.rw_uri = self.ro_uri = None + if given_rw_uri: + if deep_immutable: + if given_rw_uri.startswith(ALLEGED_IMMUTABLE_PREFIX) and not given_ro_uri: + # We needed an immutable cap, and were given one. It was given in the + # rw_uri slot, but that's fine; we'll move it to ro_uri below. + pass + elif not given_ro_uri: + self.error = MustNotBeUnknownRWError("cannot attach unknown rw cap as immutable child", + name, True) + return # node will be opaque + else: + # We could report either error, but this probably makes more sense. + self.error = MustBeDeepImmutableError("cannot attach unknown rw cap as immutable child", + name) + return # node will be opaque + + if not given_ro_uri: + # We were given a single cap argument, or a rw_uri with no ro_uri. + + if not (given_rw_uri.startswith(ALLEGED_READONLY_PREFIX) + or given_rw_uri.startswith(ALLEGED_IMMUTABLE_PREFIX)): + # If the single cap is unprefixed, then we cannot tell whether it is a + # writecap, and we don't know how to diminish it to a readcap if it is one. + # If it didn't *already* have at least an ALLEGED_READONLY_PREFIX, then + # prefixing it would be a bad idea because we have been given no reason + # to believe that it is a readcap, so we might be letting a client + # inadvertently grant excess write authority. + self.error = MustNotBeUnknownRWError("cannot attach unknown rw cap as child", + name, False) + return # node will be opaque + + # OTOH, if the single cap already had a prefix (which is of the required + # strength otherwise an error would have been thrown above), then treat it + # as though it had been given in the ro_uri slot. This has a similar effect + # to the use for known caps of 'bigcap = writecap or readcap' in + # nodemaker.py: create_from_cap. It enables copying of unknown readcaps to + # work in as many cases as we can securely allow. + given_ro_uri = given_rw_uri + given_rw_uri = None + elif given_ro_uri.startswith(ALLEGED_IMMUTABLE_PREFIX): + # Strange corner case: we were given a cap in both slots, with the ro_uri + # alleged to be immutable. A real immutable object wouldn't have a writecap. + self.error = MustBeDeepImmutableError("cannot accept a child entry that specifies " + "both rw_uri, and ro_uri with an imm. prefix", + name) + return # node will be opaque + + # If the ro_uri definitely fails the constraint, it should be treated as opaque and + # the error recorded. + if given_ro_uri: + read_cap = uri.from_string(given_ro_uri, deep_immutable=deep_immutable, name=name) + if isinstance(read_cap, uri.UnknownURI): + self.error = read_cap.get_error() + if self.error: + assert self.rw_uri is None and self.ro_uri is None + return + + if deep_immutable: + assert self.rw_uri is None + # strengthen the constraint on ro_uri to ALLEGED_IMMUTABLE_PREFIX + if given_ro_uri: + if given_ro_uri.startswith(ALLEGED_IMMUTABLE_PREFIX): + self.ro_uri = given_ro_uri + elif given_ro_uri.startswith(ALLEGED_READONLY_PREFIX): + self.ro_uri = ALLEGED_IMMUTABLE_PREFIX + given_ro_uri[len(ALLEGED_READONLY_PREFIX):] + else: + self.ro_uri = ALLEGED_IMMUTABLE_PREFIX + given_ro_uri + else: + # not immutable, so a writecap is allowed + self.rw_uri = given_rw_uri + # strengthen the constraint on ro_uri to ALLEGED_READONLY_PREFIX + if given_ro_uri: + if (given_ro_uri.startswith(ALLEGED_READONLY_PREFIX) or + given_ro_uri.startswith(ALLEGED_IMMUTABLE_PREFIX)): + self.ro_uri = given_ro_uri + else: + self.ro_uri = ALLEGED_READONLY_PREFIX + given_ro_uri + + def get_cap(self): + return uri.UnknownURI(self.rw_uri or self.ro_uri) + + def get_readcap(self): + return uri.UnknownURI(self.ro_uri) + + def is_readonly(self): + raise AssertionError("an UnknownNode might be either read-only or " + "read/write, so we shouldn't be calling is_readonly") + + def is_mutable(self): + raise AssertionError("an UnknownNode might be either mutable or immutable, " + "so we shouldn't be calling is_mutable") + + def is_unknown(self): + return True + + def is_allowed_in_immutable_directory(self): + # An UnknownNode consisting only of a ro_uri is allowed in an + # immutable directory, even though we do not know that it is + # immutable (or even read-only), provided that no error was detected. + return not self.error and not self.rw_uri + + def raise_error(self): + if self.error is not None: + raise self.error + def get_uri(self): hunk ./src/allmydata/unknown.py 154 - return self.writecap + return self.rw_uri or self.ro_uri + + def get_write_uri(self): + return self.rw_uri + def get_readonly_uri(self): hunk ./src/allmydata/unknown.py 160 - return self.readcap + return self.ro_uri + def get_storage_index(self): return None hunk ./src/allmydata/unknown.py 164 + def get_verify_cap(self): return None hunk ./src/allmydata/unknown.py 167 + def get_repair_cap(self): return None hunk ./src/allmydata/unknown.py 170 + def get_size(self): return None hunk ./src/allmydata/unknown.py 173 + def get_current_size(self): return defer.succeed(None) hunk ./src/allmydata/unknown.py 176 + def check(self, monitor, verify, add_lease): return defer.succeed(None) hunk ./src/allmydata/unknown.py 179 + def check_and_repair(self, monitor, verify, add_lease): return defer.succeed(None) hunk ./src/allmydata/uri.py 8 from allmydata.storage.server import si_a2b, si_b2a from allmydata.util import base32, hashutil from allmydata.interfaces import IURI, IDirnodeURI, IFileURI, IImmutableFileURI, \ - IVerifierURI, IMutableFileURI, IDirectoryURI, IReadonlyDirectoryURI + IVerifierURI, IMutableFileURI, IDirectoryURI, IReadonlyDirectoryURI, \ + MustBeDeepImmutableError, MustBeReadonlyError, CapConstraintError hunk ./src/allmydata/uri.py 11 -class BadURIError(Exception): +class BadURIError(CapConstraintError): pass # the URI shall be an ascii representation of the file. It shall contain hunk ./src/allmydata/uri.py 75 def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) if not mo: - raise BadURIError("%s doesn't look like a cap" % (uri,)) + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2)), int(mo.group(3)), int(mo.group(4)), int(mo.group(5))) hunk ./src/allmydata/uri.py 83 def init_from_string(cls, uri): mo = cls.STRING_RE.search(uri) if not mo: - raise BadURIError("%s doesn't look like a cap" % (uri,)) + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2)), int(mo.group(3)), int(mo.group(4)), int(mo.group(5))) hunk ./src/allmydata/uri.py 101 def is_readonly(self): return True + def is_mutable(self): return False hunk ./src/allmydata/uri.py 104 + def get_readonly(self): return self hunk ./src/allmydata/uri.py 140 @classmethod def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) - assert mo, uri + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2)), int(mo.group(3)), int(mo.group(4)), int(mo.group(5))) hunk ./src/allmydata/uri.py 148 @classmethod def init_from_string(cls, uri): mo = cls.STRING_RE.search(uri) - assert mo, (uri, cls, cls.STRING_RE) + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(si_a2b(mo.group(1)), base32.a2b(mo.group(2)), int(mo.group(3)), int(mo.group(4)), int(mo.group(5))) hunk ./src/allmydata/uri.py 165 self.total_shares, self.size)) + def is_readonly(self): + return True + + def is_mutable(self): + return False + + def get_readonly(self): + return self + + def get_verify_cap(self): + return self + class LiteralFileURI(_BaseURI): implements(IURI, IImmutableFileURI) hunk ./src/allmydata/uri.py 193 @classmethod def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) - assert mo, uri + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1))) @classmethod hunk ./src/allmydata/uri.py 200 def init_from_string(cls, uri): mo = cls.STRING_RE.search(uri) - assert mo, uri + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1))) def to_string(self): hunk ./src/allmydata/uri.py 209 def is_readonly(self): return True + def is_mutable(self): return False hunk ./src/allmydata/uri.py 212 + def get_readonly(self): return self hunk ./src/allmydata/uri.py 215 + def get_storage_index(self): return None hunk ./src/allmydata/uri.py 226 def get_size(self): return len(self.data) + class WriteableSSKFileURI(_BaseURI): implements(IURI, IMutableFileURI) hunk ./src/allmydata/uri.py 247 def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) if not mo: - raise BadURIError("'%s' doesn't look like a cap" % (uri,)) + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2))) @classmethod hunk ./src/allmydata/uri.py 268 def abbrev(self): return base32.b2a(self.writekey[:5]) + def abbrev_si(self): return base32.b2a(self.storage_index)[:5] hunk ./src/allmydata/uri.py 274 def is_readonly(self): return False + def is_mutable(self): return True hunk ./src/allmydata/uri.py 277 + def get_readonly(self): return ReadonlySSKFileURI(self.readkey, self.fingerprint) hunk ./src/allmydata/uri.py 280 + def get_verify_cap(self): return SSKVerifierURI(self.storage_index, self.fingerprint) hunk ./src/allmydata/uri.py 284 + class ReadonlySSKFileURI(_BaseURI): implements(IURI, IMutableFileURI) hunk ./src/allmydata/uri.py 302 def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) if not mo: - raise BadURIError("'%s' doesn't look like a cap" % (uri,)) + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2))) @classmethod hunk ./src/allmydata/uri.py 309 def init_from_string(cls, uri): mo = cls.STRING_RE.search(uri) if not mo: - raise BadURIError("'%s' doesn't look like a cap" % (uri,)) + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(base32.a2b(mo.group(1)), base32.a2b(mo.group(2))) def to_string(self): hunk ./src/allmydata/uri.py 323 def abbrev(self): return base32.b2a(self.readkey[:5]) + def abbrev_si(self): return base32.b2a(self.storage_index)[:5] hunk ./src/allmydata/uri.py 329 def is_readonly(self): return True + def is_mutable(self): return True hunk ./src/allmydata/uri.py 332 + def get_readonly(self): return self hunk ./src/allmydata/uri.py 335 + def get_verify_cap(self): return SSKVerifierURI(self.storage_index, self.fingerprint) hunk ./src/allmydata/uri.py 339 + class SSKVerifierURI(_BaseURI): implements(IVerifierURI) hunk ./src/allmydata/uri.py 355 @classmethod def init_from_human_encoding(cls, uri): mo = cls.HUMAN_RE.search(uri) - assert mo, uri + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(si_a2b(mo.group(1)), base32.a2b(mo.group(2))) @classmethod hunk ./src/allmydata/uri.py 362 def init_from_string(cls, uri): mo = cls.STRING_RE.search(uri) - assert mo, (uri, cls) + if not mo: + raise BadURIError("'%s' doesn't look like a %s cap" % (uri, cls)) return cls(si_a2b(mo.group(1)), base32.a2b(mo.group(2))) def to_string(self): hunk ./src/allmydata/uri.py 372 return 'URI:SSK-Verifier:%s:%s' % (si_b2a(self.storage_index), base32.b2a(self.fingerprint)) + def is_readonly(self): + return True + + def is_mutable(self): + return False + + def get_readonly(self): + return self + + def get_verify_cap(self): + return self + class _DirectoryBaseURI(_BaseURI): implements(IURI, IDirnodeURI) def __init__(self, filenode_uri=None): hunk ./src/allmydata/uri.py 423 def abbrev(self): return self._filenode_uri.to_string().split(':')[2][:5] + def abbrev_si(self): return base32.b2a(self._filenode_uri.storage_index)[:5] hunk ./src/allmydata/uri.py 427 - def get_filenode_cap(self): - return self._filenode_uri - def is_mutable(self): return True hunk ./src/allmydata/uri.py 430 + def get_filenode_cap(self): + return self._filenode_uri + def get_verify_cap(self): return DirectoryURIVerifier(self._filenode_uri.get_verify_cap()) hunk ./src/allmydata/uri.py 458 def get_readonly(self): return ReadonlyDirectoryURI(self._filenode_uri.get_readonly()) + class ReadonlyDirectoryURI(_DirectoryBaseURI): implements(IReadonlyDirectoryURI) hunk ./src/allmydata/uri.py 478 def get_readonly(self): return self + class _ImmutableDirectoryBaseURI(_DirectoryBaseURI): def __init__(self, filenode_uri=None): if filenode_uri: hunk ./src/allmydata/uri.py 483 assert isinstance(filenode_uri, self.INNER_URI_CLASS), filenode_uri + assert not filenode_uri.is_mutable() _DirectoryBaseURI.__init__(self, filenode_uri) hunk ./src/allmydata/uri.py 486 - def is_mutable(self): - return False - def is_readonly(self): return True hunk ./src/allmydata/uri.py 489 + def is_mutable(self): + return False + def get_readonly(self): return self hunk ./src/allmydata/uri.py 495 + class ImmutableDirectoryURI(_ImmutableDirectoryBaseURI): BASE_STRING='URI:DIR2-CHK:' BASE_STRING_RE=re.compile('^'+BASE_STRING) hunk ./src/allmydata/uri.py 501 BASE_HUMAN_RE=re.compile('^'+OPTIONALHTTPLEAD+'URI'+SEP+'DIR2-CHK'+SEP) INNER_URI_CLASS=CHKFileURI + def get_verify_cap(self): vcap = self._filenode_uri.get_verify_cap() return ImmutableDirectoryURIVerifier(vcap) hunk ./src/allmydata/uri.py 512 BASE_STRING_RE=re.compile('^'+BASE_STRING) BASE_HUMAN_RE=re.compile('^'+OPTIONALHTTPLEAD+'URI'+SEP+'DIR2-LIT'+SEP) INNER_URI_CLASS=LiteralFileURI + def get_verify_cap(self): # LIT caps have no verifier, since they aren't distributed return None hunk ./src/allmydata/uri.py 517 + def wrap_dirnode_cap(filecap): if isinstance(filecap, WriteableSSKFileURI): return DirectoryURI(filecap) hunk ./src/allmydata/uri.py 527 return ImmutableDirectoryURI(filecap) if isinstance(filecap, LiteralFileURI): return LiteralDirectoryURI(filecap) - assert False, "cannot wrap a dirnode around %s" % filecap.__class__ + assert False, "cannot interpret as a directory cap: %s" % filecap.__class__ + class DirectoryURIVerifier(_DirectoryBaseURI): implements(IVerifierURI) hunk ./src/allmydata/uri.py 546 def get_filenode_cap(self): return self._filenode_uri + def is_mutable(self): + return False + + class ImmutableDirectoryURIVerifier(DirectoryURIVerifier): implements(IVerifierURI) BASE_STRING='URI:DIR2-CHK-Verifier:' hunk ./src/allmydata/uri.py 557 BASE_HUMAN_RE=re.compile('^'+OPTIONALHTTPLEAD+'URI'+SEP+'DIR2-CHK-VERIFIER'+SEP) INNER_URI_CLASS=CHKFileVerifierURI + class UnknownURI: hunk ./src/allmydata/uri.py 559 - def __init__(self, uri): + def __init__(self, uri, error=None): self._uri = uri hunk ./src/allmydata/uri.py 561 + self._error = error + def to_string(self): return self._uri hunk ./src/allmydata/uri.py 566 -def from_string(s): - if not isinstance(s, str): - raise TypeError("unknown URI type: %s.." % str(s)[:100]) - elif s.startswith('URI:CHK:'): - return CHKFileURI.init_from_string(s) - elif s.startswith('URI:CHK-Verifier:'): - return CHKFileVerifierURI.init_from_string(s) - elif s.startswith('URI:LIT:'): - return LiteralFileURI.init_from_string(s) - elif s.startswith('URI:SSK:'): - return WriteableSSKFileURI.init_from_string(s) - elif s.startswith('URI:SSK-RO:'): - return ReadonlySSKFileURI.init_from_string(s) - elif s.startswith('URI:SSK-Verifier:'): - return SSKVerifierURI.init_from_string(s) - elif s.startswith('URI:DIR2:'): - return DirectoryURI.init_from_string(s) - elif s.startswith('URI:DIR2-RO:'): - return ReadonlyDirectoryURI.init_from_string(s) - elif s.startswith('URI:DIR2-Verifier:'): - return DirectoryURIVerifier.init_from_string(s) - elif s.startswith('URI:DIR2-CHK:'): - return ImmutableDirectoryURI.init_from_string(s) - elif s.startswith('URI:DIR2-LIT:'): - return LiteralDirectoryURI.init_from_string(s) - return UnknownURI(s) + def get_readonly(self): + return None + + def get_error(self): + return self._error + + def get_verify_cap(self): + return None + + +ALLEGED_READONLY_PREFIX = 'ro.' +ALLEGED_IMMUTABLE_PREFIX = 'imm.' + +def from_string(u, deep_immutable=False, name=u""): + if not isinstance(u, str): + raise TypeError("unknown URI type: %s.." % str(u)[:100]) + + # We allow and check ALLEGED_READONLY_PREFIX or ALLEGED_IMMUTABLE_PREFIX + # on all URIs, even though we would only strictly need to do so for caps of + # new formats (post Tahoe-LAFS 1.6). URIs that are not consistent with their + # prefix are treated as unknown. This should be revisited when we add the + # new cap formats. See . + s = u + can_be_mutable = can_be_writeable = not deep_immutable + if s.startswith(ALLEGED_IMMUTABLE_PREFIX): + can_be_mutable = can_be_writeable = False + s = s[len(ALLEGED_IMMUTABLE_PREFIX):] + elif s.startswith(ALLEGED_READONLY_PREFIX): + can_be_writeable = False + s = s[len(ALLEGED_READONLY_PREFIX):] + + error = None + kind = "cap" + try: + if s.startswith('URI:CHK:'): + return CHKFileURI.init_from_string(s) + elif s.startswith('URI:CHK-Verifier:'): + return CHKFileVerifierURI.init_from_string(s) + elif s.startswith('URI:LIT:'): + return LiteralFileURI.init_from_string(s) + elif s.startswith('URI:SSK:'): + if can_be_writeable: + return WriteableSSKFileURI.init_from_string(s) + kind = "URI:SSK file writecap" + elif s.startswith('URI:SSK-RO:'): + if can_be_mutable: + return ReadonlySSKFileURI.init_from_string(s) + kind = "URI:SSK-RO readcap to a mutable file" + elif s.startswith('URI:SSK-Verifier:'): + return SSKVerifierURI.init_from_string(s) + elif s.startswith('URI:DIR2:'): + if can_be_writeable: + return DirectoryURI.init_from_string(s) + kind = "URI:DIR2 directory writecap" + elif s.startswith('URI:DIR2-RO:'): + if can_be_mutable: + return ReadonlyDirectoryURI.init_from_string(s) + kind = "URI:DIR2-RO readcap to a mutable directory" + elif s.startswith('URI:DIR2-Verifier:'): + return DirectoryURIVerifier.init_from_string(s) + elif s.startswith('URI:DIR2-CHK:'): + return ImmutableDirectoryURI.init_from_string(s) + elif s.startswith('URI:DIR2-LIT:'): + return LiteralDirectoryURI.init_from_string(s) + elif s.startswith('x-tahoe-future-test-writeable:') and not can_be_writeable: + # For testing how future writeable caps would behave in read-only contexts. + kind = "x-tahoe-future-test-writeable: testing cap" + elif s.startswith('x-tahoe-future-test-mutable:') and not can_be_mutable: + # For testing how future mutable readcaps would behave in immutable contexts. + kind = "x-tahoe-future-test-mutable: testing cap" + else: + return UnknownURI(u) + + # We fell through because a constraint was not met. + # Prefer to report the most specific constraint. + if not can_be_mutable: + error = MustBeDeepImmutableError(kind + " used in an immutable context", name) + else: + error = MustBeReadonlyError(kind + " used in a read-only context", name) + + except BadURIError, e: + error = e + + return UnknownURI(u, error=error) def is_uri(s): try: hunk ./src/allmydata/uri.py 653 - from_string(s) + from_string(s, deep_immutable=False) return True except (TypeError, AssertionError): return False hunk ./src/allmydata/uri.py 658 -def from_string_dirnode(s): - u = from_string(s) +def is_literal_file_uri(s): + if not isinstance(s, str): + return False + return (s.startswith('URI:LIT:') or + s.startswith(ALLEGED_READONLY_PREFIX + 'URI:LIT:') or + s.startswith(ALLEGED_IMMUTABLE_PREFIX + 'URI:LIT:')) + +def has_uri_prefix(s): + if not isinstance(s, str): + return False + return (s.startswith("URI:") or + s.startswith(ALLEGED_READONLY_PREFIX + 'URI:') or + s.startswith(ALLEGED_IMMUTABLE_PREFIX + 'URI:')) + + +# These take the same keyword arguments as from_string above. + +def from_string_dirnode(s, **kwargs): + u = from_string(s, **kwargs) assert IDirnodeURI.providedBy(u) return u hunk ./src/allmydata/uri.py 682 registerAdapter(from_string_dirnode, str, IDirnodeURI) -def from_string_filenode(s): - u = from_string(s) +def from_string_filenode(s, **kwargs): + u = from_string(s, **kwargs) assert IFileURI.providedBy(u) return u hunk ./src/allmydata/uri.py 689 registerAdapter(from_string_filenode, str, IFileURI) -def from_string_mutable_filenode(s): - u = from_string(s) +def from_string_mutable_filenode(s, **kwargs): + u = from_string(s, **kwargs) assert IMutableFileURI.providedBy(u) return u registerAdapter(from_string_mutable_filenode, str, IMutableFileURI) hunk ./src/allmydata/uri.py 695 -def from_string_verifier(s): - u = from_string(s) +def from_string_verifier(s, **kwargs): + u = from_string(s, **kwargs) assert IVerifierURI.providedBy(u) return u registerAdapter(from_string_verifier, str, IVerifierURI) hunk ./src/allmydata/web/common.py 11 from nevow.util import resource_filename from allmydata.interfaces import ExistingChildError, NoSuchChildError, \ FileTooLargeError, NotEnoughSharesError, NoSharesError, \ - NotDeepImmutableError, EmptyPathnameComponentError + EmptyPathnameComponentError, MustBeDeepImmutableError, \ + MustBeReadonlyError, MustNotBeUnknownRWError from allmydata.mutable.common import UnrecoverableFileError from allmydata.util import abbreviate # TODO: consolidate hunk ./src/allmydata/web/common.py 185 "failure, or disk corruption. You should perform a filecheck on " "this object to learn more.") return (t, http.GONE) - if f.check(NotDeepImmutableError): - t = ("NotDeepImmutableError: a mkdir-immutable operation was given " - "a child that was not itself immutable: %s" % (f.value,)) + if f.check(MustNotBeUnknownRWError): + name = f.value.args[1] + immutable = f.value.args[2] + if immutable: + t = ("MustNotBeUnknownRWError: an operation to add a child named " + "'%s' to a directory was given an unknown cap in a write slot.\n" + "If the cap is actually an immutable readcap, then using a " + "webapi server that supports a later version of Tahoe may help.\n\n" + "If you are using the webapi directly, then specifying an immutable " + "readcap in the read slot (ro_uri) of the JSON PROPDICT, and " + "omitting the write slot (rw_uri), would also work in this " + "case.") % name.encode("utf-8") + else: + t = ("MustNotBeUnknownRWError: an operation to add a child named " + "'%s' to a directory was given an unknown cap in a write slot.\n" + "Using a webapi server that supports a later version of Tahoe " + "may help.\n\n" + "If you are using the webapi directly, specifying a readcap in " + "the read slot (ro_uri) of the JSON PROPDICT, as well as a " + "writecap in the write slot if desired, would also work in this " + "case.") % name.encode("utf-8") + return (t, http.BAD_REQUEST) + if f.check(MustBeDeepImmutableError): + name = f.value.args[1] + t = ("MustBeDeepImmutableError: a cap passed to this operation for " + "the child named '%s', needed to be immutable but was not. Either " + "the cap is being added to an immutable directory, or it was " + "originally retrieved from an immutable directory as an unknown " + "cap." % name.encode("utf-8")) + return (t, http.BAD_REQUEST) + if f.check(MustBeReadonlyError): + name = f.value.args[1] + t = ("MustBeReadonlyError: a cap passed to this operation for " + "the child named '%s', needed to be read-only but was not. " + "The cap is being passed in a read slot (ro_uri), or was retrieved " + "from a read slot as an unknown cap." % name.encode("utf-8")) return (t, http.BAD_REQUEST) if f.check(WebError): return (f.value.text, f.value.code) hunk ./src/allmydata/web/directory.py 355 charset = get_arg(req, "_charset", "utf-8") name = name.decode(charset) replace = boolean_of_arg(get_arg(req, "replace", "true")) - d = self.node.set_uri(name, childcap, childcap, overwrite=replace) + + # We mustn't pass childcap for the readcap argument because we don't + # know whether it is a read cap. Passing a read cap as the writecap + # argument will work (it ends up calling NodeMaker.create_from_cap, + # which derives a readcap if necessary and possible). + d = self.node.set_uri(name, childcap, None, overwrite=replace) d.addCallback(lambda res: childcap) return d hunk ./src/allmydata/web/directory.py 371 # won't show up in the resulting encoded form.. the 'name' # field is completely missing. So to allow deletion of an # empty file, we have to pretend that None means ''. The only - # downide of this is a slightly confusing error message if + # downside of this is a slightly confusing error message if # someone does a POST without a name= field. For our own HTML hunk ./src/allmydata/web/directory.py 373 - # thisn't a big deal, because we create the 'delete' POST + # this isn't a big deal, because we create the 'delete' POST # buttons ourselves. name = '' charset = get_arg(req, "_charset", "utf-8") hunk ./src/allmydata/web/directory.py 593 def render_title(self, ctx, data): si_s = abbreviated_dirnode(self.node) header = ["Tahoe-LAFS - Directory SI=%s" % si_s] - if self.node.is_readonly(): + if self.node.is_unknown(): + header.append(" (unknown)") + elif not self.node.is_mutable(): + header.append(" (immutable)") + elif self.node.is_readonly(): header.append(" (read-only)") else: header.append(" (modifiable)") hunk ./src/allmydata/web/directory.py 606 def render_header(self, ctx, data): si_s = abbreviated_dirnode(self.node) header = ["Tahoe-LAFS Directory SI=", T.span(class_="data-chars")[si_s]] - if self.node.is_readonly(): + if self.node.is_unknown(): + header.append(" (unknown)") + elif not self.node.is_mutable(): + header.append(" (immutable)") + elif self.node.is_readonly(): header.append(" (read-only)") return ctx.tag[header] hunk ./src/allmydata/web/directory.py 619 return T.div[T.a(href=link)["Return to Welcome page"]] def render_show_readonly(self, ctx, data): - if self.node.is_readonly(): + if self.node.is_unknown() or self.node.is_readonly(): return "" rocap = self.node.get_readonly_uri() root = get_root(ctx) hunk ./src/allmydata/web/directory.py 646 root = get_root(ctx) here = "%s/uri/%s/" % (root, urllib.quote(self.node.get_uri())) - if self.node.is_readonly(): + if self.node.is_unknown() or self.node.is_readonly(): delete = "-" rename = "-" else: hunk ./src/allmydata/web/directory.py 694 ctx.fillSlots("times", times) assert IFilesystemNode.providedBy(target), target - writecap = target.get_uri() or "" - quoted_uri = urllib.quote(writecap, safe="") # escape slashes too + target_uri = target.get_uri() or "" + quoted_uri = urllib.quote(target_uri, safe="") # escape slashes too if IMutableFileNode.providedBy(target): # to prevent javascript in displayed .html files from stealing a hunk ./src/allmydata/web/directory.py 724 elif IDirectoryNode.providedBy(target): # directory - uri_link = "%s/uri/%s/" % (root, urllib.quote(writecap)) + uri_link = "%s/uri/%s/" % (root, urllib.quote(target_uri)) ctx.fillSlots("filename", T.a(href=uri_link)[html.escape(name)]) if not target.is_mutable(): hunk ./src/allmydata/web/directory.py 811 kids = {} for name, (childnode, metadata) in children.iteritems(): assert IFilesystemNode.providedBy(childnode), childnode - rw_uri = childnode.get_uri() + rw_uri = childnode.get_write_uri() ro_uri = childnode.get_readonly_uri() if IFileNode.providedBy(childnode): hunk ./src/allmydata/web/directory.py 814 - if childnode.is_readonly(): - rw_uri = None kiddata = ("filenode", {'size': childnode.get_size(), 'mutable': childnode.is_mutable(), }) hunk ./src/allmydata/web/directory.py 818 elif IDirectoryNode.providedBy(childnode): - if childnode.is_readonly(): - rw_uri = None kiddata = ("dirnode", {'mutable': childnode.is_mutable()}) else: kiddata = ("unknown", {}) hunk ./src/allmydata/web/directory.py 821 + kiddata[1]["metadata"] = metadata hunk ./src/allmydata/web/directory.py 823 - if ro_uri: - kiddata[1]["ro_uri"] = ro_uri if rw_uri: kiddata[1]["rw_uri"] = rw_uri hunk ./src/allmydata/web/directory.py 825 + if ro_uri: + kiddata[1]["ro_uri"] = ro_uri verifycap = childnode.get_verify_cap() if verifycap: kiddata[1]['verify_uri'] = verifycap.to_string() hunk ./src/allmydata/web/directory.py 830 + kids[name] = kiddata hunk ./src/allmydata/web/directory.py 832 - if dirnode.is_readonly(): - drw_uri = None - dro_uri = dirnode.get_uri() - else: - drw_uri = dirnode.get_uri() - dro_uri = dirnode.get_readonly_uri() + + drw_uri = dirnode.get_write_uri() + dro_uri = dirnode.get_readonly_uri() contents = { 'children': kids } if dro_uri: contents['ro_uri'] = dro_uri hunk ./src/allmydata/web/directory.py 845 contents['verify_uri'] = verifycap.to_string() contents['mutable'] = dirnode.is_mutable() data = ("dirnode", contents) - return simplejson.dumps(data, indent=1) + "\n" + json = simplejson.dumps(data, indent=1) + "\n" + return json d.addCallback(_got) d.addCallback(text_plain, ctx) return d hunk ./src/allmydata/web/directory.py 852 - def DirectoryURI(ctx, dirnode): return text_plain(dirnode.get_uri(), ctx) hunk ./src/allmydata/web/directory.py 1143 self.req.write(j+"\n") return "" -class UnknownNodeHandler(RenderMixin, rend.Page): hunk ./src/allmydata/web/directory.py 1144 +class UnknownNodeHandler(RenderMixin, rend.Page): def __init__(self, client, node, parentnode=None, name=None): rend.Page.__init__(self) assert node hunk ./src/allmydata/web/directory.py 1149 self.node = node + self.parentnode = parentnode + self.name = name def render_GET(self, ctx): req = IRequest(ctx) hunk ./src/allmydata/web/directory.py 1157 t = get_arg(req, "t", "").strip() if t == "info": return MoreInfo(self.node) - raise WebError("GET unknown URI type: can only do t=info, not t=%s" % t) - + if t == "json": + if self.parentnode and self.name: + d = self.parentnode.get_metadata_for(self.name) + else: + d = defer.succeed(None) + d.addCallback(lambda md: UnknownJSONMetadata(ctx, self.node, md)) + return d + raise WebError("GET unknown URI type: can only do t=info and t=json, not t=%s.\n" + "Using a webapi server that supports a later version of Tahoe " + "may help." % t) hunk ./src/allmydata/web/directory.py 1168 +def UnknownJSONMetadata(ctx, filenode, edge_metadata): + rw_uri = filenode.get_write_uri() + ro_uri = filenode.get_readonly_uri() + data = ("unknown", {}) + if ro_uri: + data[1]['ro_uri'] = ro_uri + if rw_uri: + data[1]['rw_uri'] = rw_uri + if edge_metadata is not None: + data[1]['metadata'] = edge_metadata + return text_plain(simplejson.dumps(data, indent=1) + "\n", ctx) hunk ./src/allmydata/web/filenode.py 9 from nevow import url, rend from nevow.inevow import IRequest -from allmydata.interfaces import ExistingChildError, CannotPackUnknownNodeError +from allmydata.interfaces import ExistingChildError from allmydata.monitor import Monitor from allmydata.immutable.upload import FileHandle hunk ./src/allmydata/web/filenode.py 12 -from allmydata.unknown import UnknownNode from allmydata.util import log, base32 from allmydata.web.common import text_plain, WebError, RenderMixin, \ hunk ./src/allmydata/web/filenode.py 22 from allmydata.web.info import MoreInfo class ReplaceMeMixin: - def replace_me_with_a_child(self, req, client, replace): # a new file is being uploaded in our place. mutable = boolean_of_arg(get_arg(req, "mutable", "false")) hunk ./src/allmydata/web/filenode.py 56 def replace_me_with_a_childcap(self, req, client, replace): req.content.seek(0) childcap = req.content.read() - childnode = client.create_node_from_uri(childcap, childcap+"readonly") - if isinstance(childnode, UnknownNode): - # don't be willing to pack unknown nodes: we might accidentally - # put some write-authority into the rocap slot because we don't - # know how to diminish the URI they gave us. We don't even know - # if they gave us a readcap or a writecap. - msg = "cannot attach unknown node as child %s" % str(self.name) - raise CannotPackUnknownNodeError(msg) + childnode = client.create_node_from_uri(childcap, None, name=self.name) d = self.parentnode.set_node(self.name, childnode, overwrite=replace) d.addCallback(lambda res: childnode.get_uri()) return d hunk ./src/allmydata/web/filenode.py 420 def FileJSONMetadata(ctx, filenode, edge_metadata): - if filenode.is_readonly(): - rw_uri = None - ro_uri = filenode.get_uri() - else: - rw_uri = filenode.get_uri() - ro_uri = filenode.get_readonly_uri() + rw_uri = filenode.get_write_uri() + ro_uri = filenode.get_readonly_uri() data = ("filenode", {}) data[1]['size'] = filenode.get_size() if ro_uri: hunk ./src/allmydata/web/info.py 24 def get_type(self): node = self.original if IDirectoryNode.providedBy(node): + if not node.is_mutable(): + return "immutable directory" return "directory" if IFileNode.providedBy(node): si = node.get_storage_index() hunk ./src/allmydata/web/info.py 33 if node.is_mutable(): return "mutable file" return "immutable file" - return "LIT file" + return "immutable LIT file" return "unknown" def render_title(self, ctx, data): hunk ./src/allmydata/web/info.py 73 def render_directory_writecap(self, ctx, data): node = self.original - if node.is_readonly(): - return "" if not IDirectoryNode.providedBy(node): return "" hunk ./src/allmydata/web/info.py 75 + if node.is_readonly(): + return "" return ctx.tag[node.get_uri()] def render_directory_readcap(self, ctx, data): hunk ./src/allmydata/web/info.py 91 return "" return ctx.tag[node.get_verify_cap().to_string()] - def render_file_writecap(self, ctx, data): node = self.original if IDirectoryNode.providedBy(node): hunk ./src/allmydata/web/info.py 95 node = node._node - if ((IDirectoryNode.providedBy(node) or IFileNode.providedBy(node)) - and node.is_readonly()): - return "" - writecap = node.get_uri() - if not writecap: + write_uri = node.get_write_uri() + if not write_uri: return "" hunk ./src/allmydata/web/info.py 98 - return ctx.tag[writecap] + return ctx.tag[write_uri] def render_file_readcap(self, ctx, data): node = self.original hunk ./src/allmydata/web/info.py 104 if IDirectoryNode.providedBy(node): node = node._node - readcap = node.get_readonly_uri() - if not readcap: + read_uri = node.get_readonly_uri() + if not read_uri: return "" hunk ./src/allmydata/web/info.py 107 - return ctx.tag[readcap] + return ctx.tag[read_uri] def render_file_verifycap(self, ctx, data): node = self.original hunk ./src/allmydata/web/root.py 15 from allmydata import get_package_versions_string from allmydata import provisioning from allmydata.util import idlib, log -from allmydata.interfaces import IFileNode, UnhandledCapTypeError +from allmydata.interfaces import IFileNode from allmydata.web import filenode, directory, unlinked, status, operations from allmydata.web import reliability, storage from allmydata.web.common import abbreviate_size, getxmlfile, WebError, \ hunk ./src/allmydata/web/root.py 88 try: node = self.client.create_node_from_uri(name) return directory.make_handler_for(node, self.client) - except (TypeError, UnhandledCapTypeError, AssertionError): + except (TypeError, AssertionError): raise WebError("'%s' is not a valid file- or directory- cap" % name) hunk ./src/allmydata/web/root.py 107 # 'name' must be a file URI try: node = self.client.create_node_from_uri(name) - except (TypeError, UnhandledCapTypeError, AssertionError): + except (TypeError, AssertionError): # I think this can no longer be reached raise WebError("'%s' is not a valid file- or directory- cap" % name) } [Miscellaneous documentation, test, and code formatting tweaks. david-sarah@jacaranda.org**20100127070309 Ignore-this: 84ca7e4bb7c64221ae2c61144ef5edef ] { hunk ./contrib/fuse/impl_c/blackmatch.py 909 class TStat(fuse.Stat): # in fuse 0.2, these are set by fuse.Stat.__init__ - # in fuse 0.2-pre3 (hardy) they are not. badness unsues if they're missing + # in fuse 0.2-pre3 (hardy) they are not. badness ensues if they're missing st_mode = None st_ino = 0 st_dev = 0 hunk ./contrib/fuse/impl_c/blackmatch.py 1022 def get_uri(self): return self.rw_uri or self.ro_uri + # TODO: rename to 'is_writeable', or switch sense to 'is_readonly', for consistency with Tahoe code def writable(self): return self.rw_uri and self.rw_uri != self.ro_uri hunk ./docs/frontends/webapi.txt 73 these tasks. In general, everything that can be done with a PUT or DELETE can also be done with a POST. -Tahoe's web API is designed for two different consumers. The first is a -program that needs to manipulate the virtual file system. Such programs are +Tahoe's web API is designed for two different kinds of consumer. The first is +a program that needs to manipulate the virtual file system. Such programs are expected to use the RESTful interface described above. The second is a human using a standard web browser to work with the filesystem. This user is given a series of HTML pages with links to download files, and forms that use POST hunk ./docs/frontends/webapi.txt 81 actions to upload, rename, and delete files. When an error occurs, the HTTP response code will be set to an appropriate -400-series code (like 404 for an unknown childname, or 400 Gone when a file -is unrecoverable due to insufficient shares), and the HTTP response body will -usually contain a few lines of explanation as to the cause of the error and -possible responses. Unusual exceptions may result in a 500 Internal Server -Error as a catch-all, with a default response body will contain a -Nevow-generated HTML-ized representation of the Python exception stack trace +400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request +when the parameters to a webapi operation are invalid), and the HTTP response +body will usually contain a few lines of explanation as to the cause of the +error and possible responses. Unusual exceptions may result in a +500 Internal Server Error as a catch-all, with a default response body containing +a Nevow-generated HTML-ized representation of the Python exception stack trace that caused the problem. CLI programs which want to copy the response body to stderr should provide an "Accept: text/plain" header to their requests to get a plain text stack trace instead. If the Accept header contains */*, or hunk ./docs/frontends/webapi.txt 111 read- and write- caps, which start with "URI:SSK", and give access to mutable files. -(later versions of Tahoe will make these strings shorter, and will remove the +(Later versions of Tahoe will make these strings shorter, and will remove the unfortunate colons, which must be escaped when these caps are embedded in hunk ./docs/frontends/webapi.txt 113 -URLs). +URLs.) To refer to any Tahoe object through the web API, you simply need to combine a prefix (which indicates the HTTP server to use) with the cap (which hunk ./docs/frontends/webapi.txt 124 http://127.0.0.1:3456/uri/ + $CAP So, to access the directory named above (which happens to be the -publically-writable sample directory on the Tahoe test grid, described at +publically-writeable sample directory on the Tahoe test grid, described at http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be: http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/ hunk ./docs/frontends/webapi.txt 202 representable as such. All Tahoe operations that refer to existing files or directories must include -a suitable read- or write- cap in the URL: the wapi server won't add one +a suitable read- or write- cap in the URL: the webapi server won't add one for you. If you don't know the cap, you can't access the file. This allows hunk ./docs/frontends/webapi.txt 204 -the security properties of Tahoe caps to be extended across the wapi +the security properties of Tahoe caps to be extended across the webapi interface. == Slow Operations, Progress, and Cancelling == hunk ./docs/frontends/webapi.txt 278 since the operation completed) will remain valid for ten minutes. Many "slow" operations can begin to use unacceptable amounts of memory when -operation on large directory structures. The memory usage increases when the +operating on large directory structures. The memory usage increases when the ophandle is polled, as the results must be copied into a JSON string, sent over the wire, then parsed by a client. So, as an alternative, many "slow" operations have streaming equivalents. These equivalents do not use operation hunk ./docs/frontends/webapi.txt 318 retrieve the same contents that were just uploaded. This will create any necessary intermediate subdirectories. - To use the /uri/$FILECAP form, $FILECAP be a write-cap for a mutable file. + To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file. In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a hunk ./docs/frontends/webapi.txt 321 - writable mutable file, that files contents will be overwritten in-place. If + writeable mutable file, that file's contents will be overwritten in-place. If it is a read-cap for a mutable file, an error will occur. If it is an immutable file, the old file will be discarded, and a new one will be put in its place. hunk ./docs/frontends/webapi.txt 340 PUT /uri This uploads a file, and produces a file-cap for the contents, but does not - attach the file into the virtual drive. No directories will be modified by + attach the file into the filesystem. No directories will be modified by this operation. The file-cap is returned as the body of the HTTP response. If "mutable=true" is in the query arguments, the operation will create a hunk ./docs/frontends/webapi.txt 354 Create a new empty directory and return its write-cap as the HTTP response body. This does not make the newly created directory visible from the - virtual drive. The "PUT" operation is provided for backwards compatibility: + filesystem. The "PUT" operation is provided for backwards compatibility: new code should use POST. POST /uri?t=mkdir-with-children hunk ./docs/frontends/webapi.txt 395 "linkcrtime": 1202777696.7564139, "linkmotime": 1202777696.7564139, } } } ] - } + } For forward-compatibility, a mutable directory can also contain caps in a format that is unknown to the webapi server. When such caps are retrieved hunk ./docs/frontends/webapi.txt 525 the immediate parent directory already has a a child named NAME. Note that the name= argument must be passed as a queryarg, because the POST - request body is used for the initial children JSON. + request body is used for the initial children JSON. POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME hunk ./docs/frontends/webapi.txt 628 Then the rw_uri field will be present in the information about a directory if and only if you have read-write access to that directory. The verify_uri - field will be presend if and only if the object has a verify-cap + field will be present if and only if the object has a verify-cap (non-distributed LIT files do not have verify-caps). ==== About the metadata ==== hunk ./docs/frontends/webapi.txt 704 link points. 4. Also, quite apart from Tahoe, you might be confused about the meaning of - the 'ctime' in unix local filesystems, which people sometimes think means - file creation time, but which actually means, in unix local filesystems, the + the 'ctime' in UNIX local filesystems, which people sometimes think means + file creation time, but which actually means, in UNIX local filesystems, the most recent time that the file contents or the file metadata (such as owner, permission bits, extended attributes, etc.) has changed. Note that although hunk ./docs/frontends/webapi.txt 708 - 'ctime' does not mean file creation time in Unix, it does mean link creation + 'ctime' does not mean file creation time in UNIX, it does mean link creation time in Tahoe, unless the "tahoe backup" command has been used on that link, in which case it means something about the local filesystem file which corresponds to the Tahoe file which is pointed at by the link. It means hunk ./docs/frontends/webapi.txt 716 Windows) or file-contents-or-metadata-update-time of the local file (if "tahoe backup" was run on a different operating system). - === Attaching an existing File or Directory by its read- or write- cap === PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri hunk ./docs/frontends/webapi.txt 739 if there is already an object at the given location, rather than overwriting the existing object. To allow the operation to overwrite a file, but return an error when trying to overwrite a directory, use - "replace=only-files" (this behavior is closer to the traditional unix "mv" + "replace=only-files" (this behavior is closer to the traditional UNIX "mv" command). Note that "true", "t", and "1" are all synonyms for "True", and "false", "f", and "0" are synonyms for "False", and the parameter is case-insensitive. hunk ./docs/frontends/webapi.txt 743 + + Note that this operation does not take its child cap in the form of + separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a + child cap in a format unknown to the webapi server, because the server + is not able to attenuate an unknown write cap to a read cap. === Adding multiple files or directories to a parent directory at once === hunk ./docs/frontends/webapi.txt 809 The object will only become completely unreachable once 1: there are no reachable directories that reference it, and 2: nobody is holding a read- or write- cap to the object. (This behavior is very similar to the way - hardlinks and anonymous files work in traditional unix filesystems). + hardlinks and anonymous files work in traditional UNIX filesystems). This operation will not modify more than a single directory. Intermediate directories which were implicitly created by PUT or POST methods will *not* hunk ./docs/frontends/webapi.txt 938 POST /uri?t=upload This uploads a file, and produces a file-cap for the contents, but does not - attach the file into the virtual drive. No directories will be modified by + attach the file into the filesystem. No directories will be modified by this operation. The file must be provided as the "file" field of an HTML encoded form body, hunk ./docs/frontends/webapi.txt 1684 == Static Files in /public_html == -The wapi server will take any request for a URL that starts with /static +The webapi server will take any request for a URL that starts with /static and serve it from a configurable directory which defaults to $BASEDIR/public_html . This is configured by setting the "[node]web.static" value in $BASEDIR/tahoe.cfg . If this is left at the default value of hunk ./docs/frontends/webapi.txt 1692 served with the contents of the file $BASEDIR/public_html/subdir/foo.html . This can be useful to serve a javascript application which provides a -prettier front-end to the rest of the Tahoe wapi. +prettier front-end to the rest of the Tahoe webapi. hunk ./docs/frontends/webapi.txt 1695 -== safety and security issues -- names vs. URIs == +== Safety and security issues -- names vs. URIs == Summary: use explicit file- and dir- caps whenever possible, to reduce the potential for surprises when the filesystem structure is changed. hunk ./docs/frontends/webapi.txt 1781 Tahoe nodes implement internal serialization to make sure that a single Tahoe node cannot conflict with itself. For example, it is safe to issue two -directory modification requests to a single tahoe node's wapi server at the +directory modification requests to a single tahoe node's webapi server at the same time, because the Tahoe node will internally delay one of them until after the other has finished being applied. (This feature was introduced in Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing hunk ./relnotes.txt 1 -ANNOUNCING Tahoe, the Lofty-Atmospheric Filesystem, v1.5 +ANNOUNCING Tahoe, the Lofty-Atmospheric Filesystem, v1.6 The Tahoe-LAFS team is pleased to announce the immediate hunk ./relnotes.txt 4 -availability of version 1.5 of Tahoe, the Lofty Atmospheric +availability of version 1.6 of Tahoe, the Lofty Atmospheric File System. Tahoe-LAFS is the first cloud storage technology which offers hunk ./relnotes.txt 32 COMPATIBILITY -Version 1.5 is fully compatible with the version 1 series of -Tahoe-LAFS. Files written by v1.5 clients can be read by -clients of all versions back to v1.0. v1.5 clients can read -files produced by clients of all versions since v1.0. v1.5 -servers can serve clients of all versions back to v1.0 and v1.5 +Version 1.6 is fully compatible with the version 1 series of +Tahoe-LAFS. Files written by v1.6 clients can be read by +clients of all versions back to v1.0. v1.6 clients can read +files produced by clients of all versions since v1.0. v1.6 +servers can serve clients of all versions back to v1.0 and v1.6 clients can use servers of all versions back to v1.0. hunk ./relnotes.txt 39 -This is the sixth release in the version 1 series. The version -1 series of Tahoe-LAFS will be actively supported and +In addition, version 1.6 improves forward-compatibility with +planned future cap formats, allowing updates to a directory +containing both current and future caps, without loss of +information. + +This is the seventh major release in the version 1 series. The +version 1 series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with Tahoe-LAFS v1. hunk ./src/allmydata/dirnode.py 26 from pycryptopp.cipher.aes import AES from allmydata.util.dictutil import AuxValueDict + +# TODO: {Deleter,MetadataSetter,Adder}.modify all start by unpacking the +# contents and end by repacking them. It might be better to apply them to +# the unpacked contents. + class Deleter: def __init__(self, node, name, must_exist=True): self.node = node hunk ./src/allmydata/interfaces.py 429 """Return True if the data can be modified by *somebody* (perhaps someone who has a more powerful URI than this one).""" + # TODO: rename to get_read_cap() def get_readonly(): """Return another IURI instance, which represents a read-only form of this one. If is_readonly() is True, this returns self.""" hunk ./src/allmydata/test/test_web.py 703 self.PUT, base, "") return d + # TODO: version of this with a Unicode filename def test_GET_FILEURL_save(self): hunk ./src/allmydata/test/test_web.py 705 - d = self.GET(self.public_url + "/foo/bar.txt?filename=bar.txt&save=true") - # TODO: look at the headers, expect a Content-Disposition: attachment - # header. - d.addCallback(self.failUnlessIsBarDotTxt) + d = self.GET(self.public_url + "/foo/bar.txt?filename=bar.txt&save=true", + return_response=True) + def _got((res, statuscode, headers)): + content_disposition = headers["content-disposition"][0] + self.failUnless(content_disposition == 'attachment; filename="bar.txt"', content_disposition) + self.failUnlessIsBarDotTxt(res) + d.addCallback(_got) return d def test_GET_FILEURL_missing(self): hunk ./src/allmydata/test/test_web.py 2265 # Fetch the welcome page. d = self.GET("/") def _after_get_welcome_page(res): - MKDIR_BUTTON_RE=re.compile('', re.I) + MKDIR_BUTTON_RE = re.compile( + '' + '' + '', + re.I) mo = MKDIR_BUTTON_RE.search(res) formaction = mo.group(1) formt = mo.group(2) hunk ./src/allmydata/uri.py 14 class BadURIError(CapConstraintError): pass -# the URI shall be an ascii representation of the file. It shall contain -# enough information to retrieve and validate the contents. It shall be -# expressed in a limited character set (namely [TODO]). +# The URI shall be an ASCII representation of a reference to the file/directory. +# It shall contain enough information to retrieve and validate the contents. +# It shall be expressed in a limited character set (currently base32 plus ':' and +# capital letters, but future URIs might use a larger charset). + +# TODO: +# - rename all of the *URI classes/interfaces to *Cap +# - make variable and method names consistently use _uri for an URI string, +# and _cap for a Cap object (decoded URI) +# - remove the human_encoding methods? BASE32STR_128bits = '(%s{25}%s)' % (base32.BASE32CHAR, base32.BASE32CHAR_3bits) BASE32STR_256bits = '(%s{51}%s)' % (base32.BASE32CHAR, base32.BASE32CHAR_1bits) hunk ./src/allmydata/uri.py 32 NUMBER='([0-9]+)' NUMBER_IGNORE='(?:[0-9]+)' -# URIs (soon to be renamed "caps") are always allowed to come with a leading +# "human-encoded" URIs are allowed to come with a leading # 'http://127.0.0.1:(8123|3456)/uri/' that will be ignored. hunk ./src/allmydata/uri.py 34 +# Note that nothing in the Tahoe code currently uses the human encoding. OPTIONALHTTPLEAD=r'(?:https?://(?:[^:/]+)(?::%s)?/uri/)?' % NUMBER_IGNORE hunk ./src/allmydata/uri.py 41 class _BaseURI: def __hash__(self): return self.to_string().__hash__() + def __eq__(self, them): if isinstance(them, _BaseURI): return self.to_string() == them.to_string() hunk ./src/allmydata/uri.py 47 else: return False + def __ne__(self, them): if isinstance(them, _BaseURI): return self.to_string() != them.to_string() hunk ./src/allmydata/uri.py 53 else: return True + def to_human_encoding(self): return 'http://127.0.0.1:3456/uri/'+self.to_string() hunk ./src/allmydata/uri.py 60 def get_storage_index(self): return self.storage_index + class CHKFileURI(_BaseURI): implements(IURI, IImmutableFileURI) } [Address comments by Kevan on 833 and add test for stripping spaces david-sarah@jacaranda.org**20100127230642 Ignore-this: de36aeaf4afb3ba05dbeb49a5e9a6b26 ] { hunk ./docs/frontends/webapi.txt 746 Note that this operation does not take its child cap in the form of separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a - child cap in a format unknown to the webapi server, because the server - is not able to attenuate an unknown write cap to a read cap. + child cap in a format unknown to the webapi server, unless its URI + starts with "ro." or "imm.". This restriction is necessary because the + server is not able to attenuate an unknown write cap to a read cap. + Unknown URIs starting with "ro." or "imm.", on the other hand, are + assumed to represent read caps. The client should not prefix a write + cap with "ro." or "imm." and pass it to this operation, since that + would result in granting the cap's write authority to holders of the + directory read cap. === Adding multiple files or directories to a parent directory at once === hunk ./docs/frontends/webapi.txt 1037 This attaches a given read- or write- cap "CHILDCAP" to the designated directory, with a specified child name. This behaves much like the PUT t=uri - operation, and is a lot like a UNIX hardlink. + operation, and is a lot like a UNIX hardlink. It is subject to the same + restrictions as that operation on the use of cap formats unknown to the + webapi server. This will create additional intermediate directories as necessary, although since it is expected to be triggered by a form that was retrieved by "GET hunk ./src/allmydata/dirnode.py 268 while position < len(data): entries, position = split_netstring(data, 1, position) entry = entries[0] - (name, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + (name_utf8, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) if not mutable and len(rwcapdata) > 0: raise ValueError("the rwcapdata field of a dirnode in an immutable directory was not empty") hunk ./src/allmydata/dirnode.py 271 - name = name.decode("utf-8") + name = name_utf8.decode("utf-8") rw_uri = "" if writeable: rw_uri = self._decrypt_rwcapdata(rwcapdata) hunk ./src/allmydata/dirnode.py 278 # Since the encryption uses CTR mode, it currently leaks the length of the # plaintext rw_uri -- and therefore whether it is present, i.e. whether the - # dirnode is writeable (ticket #925). By stripping spaces in Tahoe >= 1.6.0, - # we may make it easier for future versions to plug this leak. + # dirnode is writeable (ticket #925). By stripping trailing spaces in + # Tahoe >= 1.6.0, we may make it easier for future versions to plug this leak. # ro_uri is treated in the same way for consistency. # rw_uri and ro_uri will be either None or a non-empty string. hunk ./src/allmydata/dirnode.py 283 - rw_uri = rw_uri.strip(' ') or None - ro_uri = ro_uri.strip(' ') or None + rw_uri = rw_uri.rstrip(' ') or None + ro_uri = ro_uri.rstrip(' ') or None try: child = self._create_and_validate_node(rw_uri, ro_uri, name) hunk ./src/allmydata/dirnode.py 295 children.set_with_aux(name, (child, metadata), auxilliary=entry) else: log.msg(format="mutable cap for child '%(name)s' unpacked from an immutable directory", - name=name.encode("utf-8"), + name=name_utf8, facility="tahoe.webish", level=log.UNUSUAL) except CapConstraintError, e: log.msg(format="unmet constraint on cap for child '%(name)s' unpacked from a directory:\n" hunk ./src/allmydata/dirnode.py 299 - "%(message)s", message=e.args[0], name=name.encode("utf-8"), + "%(message)s", message=e.args[0], name=name_utf8, facility="tahoe.webish", level=log.UNUSUAL) return children hunk ./src/allmydata/test/test_dirnode.py 16 from allmydata.mutable.filenode import MutableFileNode from allmydata.mutable.common import UncoordinatedWriteError from allmydata.util import hashutil, base32 +from allmydata.util.netstring import split_netstring from allmydata.monitor import Monitor from allmydata.test.common import make_chk_file_uri, make_mutable_file_uri, \ ErrorMixin hunk ./src/allmydata/test/test_dirnode.py 52 self.set_up_grid() c = self.g.clients[0] nm = c.nodemaker + setup_py_uri = "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" one_uri = "URI:LIT:n5xgk" # LIT for "one" mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" hunk ./src/allmydata/test/test_dirnode.py 120 bad_future_node = UnknownNode(future_write_uri, None) bad_kids1 = {u"one": (bad_future_node, {})} + # This should fail because we don't know how to diminish the future_write_uri + # cap (given in a write slot and not prefixed with "ro." or "imm.") to a readcap. d.addCallback(lambda ign: self.shouldFail(MustNotBeUnknownRWError, "bad_kids1", "cannot attach unknown", hunk ./src/allmydata/test/test_dirnode.py 140 self.set_up_grid() c = self.g.clients[0] nm = c.nodemaker + setup_py_uri = "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" one_uri = "URI:LIT:n5xgk" # LIT for "one" mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" hunk ./src/allmydata/test/test_dirnode.py 288 d.addCallback(_made_parent) return d + def test_spaces_are_stripped_on_the_way_out(self): + self.basedir = "dirnode/Dirnode/test_spaces_are_stripped_on_the_way_out" + self.set_up_grid() + c = self.g.clients[0] + nm = c.nodemaker + + # This test checks that any trailing spaces in URIs are retained in the + # encoded directory, but stripped when we get them out of the directory. + # See ticket #925 for why we want that. + + stripped_write_uri = "lafs://from_the_future\t" + stripped_read_uri = "lafs://readonly_from_the_future\t" + spacedout_write_uri = stripped_write_uri + " " + spacedout_read_uri = stripped_read_uri + " " + + child = nm.create_from_cap(spacedout_write_uri, spacedout_read_uri) + self.failUnlessEqual(child.get_write_uri(), spacedout_write_uri) + self.failUnlessEqual(child.get_readonly_uri(), "ro." + spacedout_read_uri) + + kids = {u"child": (child, {})} + d = c.create_dirnode(kids) + + def _created(dn): + self.failUnless(isinstance(dn, dirnode.DirectoryNode)) + self.failUnless(dn.is_mutable()) + self.failIf(dn.is_readonly()) + dn.raise_error() + self.cap = dn.get_cap() + self.rootnode = dn + return dn._node.download_best_version() + d.addCallback(_created) + + def _check_data(data): + # Decode the netstring representation of the directory to check that the + # spaces are retained when the URIs are stored. + position = 0 + numkids = 0 + while position < len(data): + entries, position = split_netstring(data, 1, position) + entry = entries[0] + (name_utf8, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + name = name_utf8.decode("utf-8") + rw_uri = self.rootnode._decrypt_rwcapdata(rwcapdata) + self.failUnless(name in kids) + (expected_child, ign) = kids[name] + self.failUnlessEqual(rw_uri, expected_child.get_write_uri()) + self.failUnlessEqual("ro." + ro_uri, expected_child.get_readonly_uri()) + numkids += 1 + + self.failUnlessEqual(numkids, 1) + return self.rootnode.list() + d.addCallback(_check_data) + + # Now when we use the real directory listing code, the trailing spaces + # should have been stripped (and "ro." should have been prepended to the + # ro_uri, since it's unknown). + def _check_kids(children): + self.failUnlessEqual(sorted(children.keys()), [u"child"]) + child_node, child_metadata = children[u"child"] + + self.failUnlessEqual(child_node.get_write_uri(), stripped_write_uri) + self.failUnlessEqual(child_node.get_readonly_uri(), "ro." + stripped_read_uri) + d.addCallback(_check_kids) + + d.addCallback(lambda ign: nm.create_from_cap(self.cap.to_string())) + d.addCallback(lambda n: n.list()) + d.addCallback(_check_kids) # again with dirnode recreated from cap + return d + def test_check(self): self.basedir = "dirnode/Dirnode/test_check" self.set_up_grid() hunk ./src/allmydata/test/test_dirnode.py 1198 def is_allowed_in_immutable_directory(self): return False + def raise_error(self): + pass + def modify(self, modifier): self.data = modifier(self.data, None, True) return defer.succeed(None) hunk ./src/allmydata/test/test_filenode.py 42 self.failUnlessEqual(fn1.get_uri(), u.to_string()) self.failUnlessEqual(fn1.get_cap(), u) self.failUnlessEqual(fn1.get_readcap(), u) - self.failUnlessEqual(fn1.is_readonly(), True) - self.failUnlessEqual(fn1.is_mutable(), False) - self.failUnlessEqual(fn1.is_unknown(), False) - self.failUnlessEqual(fn1.is_allowed_in_immutable_directory(), True) + self.failUnless(fn1.is_readonly()) + self.failIf(fn1.is_mutable()) + self.failIf(fn1.is_unknown()) + self.failUnless(fn1.is_allowed_in_immutable_directory()) self.failUnlessEqual(fn1.get_write_uri(), None) self.failUnlessEqual(fn1.get_readonly_uri(), u.to_string()) self.failUnlessEqual(fn1.get_size(), 1000) hunk ./src/allmydata/test/test_filenode.py 57 v = fn1.get_verify_cap() self.failUnless(isinstance(v, uri.CHKFileVerifierURI)) self.failUnlessEqual(fn1.get_repair_cap(), v) - self.failUnlessEqual(v.is_readonly(), True) - self.failUnlessEqual(v.is_mutable(), False) + self.failUnless(v.is_readonly()) + self.failIf(v.is_mutable()) def test_literal_filenode(self): hunk ./src/allmydata/test/test_filenode.py 72 self.failUnlessEqual(fn1.get_uri(), u.to_string()) self.failUnlessEqual(fn1.get_cap(), u) self.failUnlessEqual(fn1.get_readcap(), u) - self.failUnlessEqual(fn1.is_readonly(), True) - self.failUnlessEqual(fn1.is_mutable(), False) - self.failUnlessEqual(fn1.is_unknown(), False) - self.failUnlessEqual(fn1.is_allowed_in_immutable_directory(), True) + self.failUnless(fn1.is_readonly()) + self.failIf(fn1.is_mutable()) + self.failIf(fn1.is_unknown()) + self.failUnless(fn1.is_allowed_in_immutable_directory()) self.failUnlessEqual(fn1.get_write_uri(), None) self.failUnlessEqual(fn1.get_readonly_uri(), u.to_string()) self.failUnlessEqual(fn1.get_size(), len(DATA)) hunk ./src/allmydata/test/test_filenode.py 125 self.failUnlessEqual(n.get_readonly_uri(), u.get_readonly().to_string()) self.failUnlessEqual(n.get_cap(), u) self.failUnlessEqual(n.get_readcap(), u.get_readonly()) - self.failUnlessEqual(n.is_mutable(), True) - self.failUnlessEqual(n.is_readonly(), False) - self.failUnlessEqual(n.is_unknown(), False) - self.failUnlessEqual(n.is_allowed_in_immutable_directory(), False) + self.failUnless(n.is_mutable()) + self.failIf(n.is_readonly()) + self.failIf(n.is_unknown()) + self.failIf(n.is_allowed_in_immutable_directory()) n.raise_error() n2 = MutableFileNode(None, None, client.get_encoding_parameters(), hunk ./src/allmydata/test/test_filenode.py 147 self.failUnlessEqual(nro.get_readonly(), nro) self.failUnlessEqual(nro.get_cap(), u.get_readonly()) self.failUnlessEqual(nro.get_readcap(), u.get_readonly()) - self.failUnlessEqual(nro.is_mutable(), True) - self.failUnlessEqual(nro.is_readonly(), True) - self.failUnlessEqual(nro.is_unknown(), False) - self.failUnlessEqual(nro.is_allowed_in_immutable_directory(), False) + self.failUnless(nro.is_mutable()) + self.failUnless(nro.is_readonly()) + self.failIf(nro.is_unknown()) + self.failIf(nro.is_allowed_in_immutable_directory()) nro_u = nro.get_uri() self.failUnlessEqual(nro_u, nro.get_readonly_uri()) self.failUnlessEqual(nro_u, u.get_readonly().to_string()) hunk ./src/allmydata/test/test_web.py 2379 def test_POST_set_children_with_hyphen(self): return self.test_POST_set_children(command_name="set-children") - def test_POST_put_uri(self): + def test_POST_link_uri(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo", t="uri", name="new.txt", uri=newuri) d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"new.txt") hunk ./src/allmydata/test/test_web.py 2388 contents)) return d - def test_POST_put_uri_replace(self): + def test_POST_link_uri_replace(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo", t="uri", name="bar.txt", uri=newuri) d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"bar.txt") hunk ./src/allmydata/test/test_web.py 2397 contents)) return d - def test_POST_put_uri_no_replace_queryarg(self): + def test_POST_link_uri_unknown_bad(self): + newuri = "lafs://from_the_future" + d = self.POST(self.public_url + "/foo", t="uri", name="future.txt", uri=newuri) + d.addBoth(self.shouldFail, error.Error, + "POST_link_uri_unknown_bad", + "400 Bad Request", + "unknown cap in a write slot") + return d + + def test_POST_link_uri_unknown_ro_good(self): + newuri = "ro.lafs://readonly_from_the_future" + d = self.POST(self.public_url + "/foo", t="uri", name="future-ro.txt", uri=newuri) + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"future-ro.txt") + return d + + def test_POST_link_uri_unknown_imm_good(self): + newuri = "imm.lafs://immutable_from_the_future" + d = self.POST(self.public_url + "/foo", t="uri", name="future-imm.txt", uri=newuri) + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, u"future-imm.txt") + return d + + def test_POST_link_uri_no_replace_queryarg(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo?replace=false", t="uri", name="bar.txt", uri=newuri) hunk ./src/allmydata/test/test_web.py 2423 d.addBoth(self.shouldFail, error.Error, - "POST_put_uri_no_replace_queryarg", + "POST_link_uri_no_replace_queryarg", "409 Conflict", "There was already a child by that name, and you asked me " "to not replace it") hunk ./src/allmydata/test/test_web.py 2431 d.addCallback(self.failUnlessIsBarDotTxt) return d - def test_POST_put_uri_no_replace_field(self): + def test_POST_link_uri_no_replace_field(self): contents, n, newuri = self.makefile(8) d = self.POST(self.public_url + "/foo", t="uri", replace="false", name="bar.txt", uri=newuri) hunk ./src/allmydata/test/test_web.py 2436 d.addBoth(self.shouldFail, error.Error, - "POST_put_uri_no_replace_field", + "POST_link_uri_no_replace_field", "409 Conflict", "There was already a child by that name, and you asked me " "to not replace it") hunk ./src/allmydata/test/test_web.py 2704 "to not replace it") return d + def test_PUT_NEWFILEURL_uri_unknown_bad(self): + new_uri = "lafs://from_the_future" + d = self.PUT(self.public_url + "/foo/put-future.txt?t=uri", new_uri) + d.addBoth(self.shouldFail, error.Error, + "POST_put_uri_unknown_bad", + "400 Bad Request", + "unknown cap in a write slot") + return d + + def test_PUT_NEWFILEURL_uri_unknown_ro_good(self): + new_uri = "ro.lafs://readonly_from_the_future" + d = self.PUT(self.public_url + "/foo/put-future-ro.txt?t=uri", new_uri) + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, + u"put-future-ro.txt") + return d + + def test_PUT_NEWFILEURL_uri_unknown_imm_good(self): + new_uri = "imm.lafs://immutable_from_the_future" + d = self.PUT(self.public_url + "/foo/put-future-imm.txt?t=uri", new_uri) + d.addCallback(self.failUnlessURIMatchesROChild, self._foo_node, + u"put-future-imm.txt") + return d + def test_PUT_NEWFILE_URI(self): file_contents = "New file contents here\n" d = self.PUT("/uri", file_contents) hunk ./src/allmydata/test/test_web.py 3407 while position < len(data): entries, position = split_netstring(data, 1, position) entry = entries[0] - (name, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) - name = name.decode("utf-8") + (name_utf8, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4) + name = name_utf8.decode("utf-8") self.failUnless(rwcapdata == "") hunk ./src/allmydata/test/test_web.py 3410 - ro_uri = ro_uri.strip() - if name in kids: - self.failIfEqual(ro_uri, "") - (expected_child, ign) = kids[name] - self.failUnlessEqual(ro_uri, expected_child.get_readonly_uri()) - numkids += 1 + self.failUnless(name in kids) + (expected_child, ign) = kids[name] + self.failUnlessEqual(ro_uri, expected_child.get_readonly_uri()) + numkids += 1 self.failUnlessEqual(numkids, 3) return self.rootnode.list() } Context: [test_runner: cleanup, refactor common code into a non-executable method Brian Warner **20100127224040 Ignore-this: 4cb4aada87777771f688edfd8129ffca Having both test_node() and test_client() (one of which calls the other) felt confusing to me, so I changed it to have test_node(), test_client(), and a common do_create() helper method. ] [scripts/runner.py: simplify David-Sarah's clever grouped-commands usage trick Brian Warner **20100127223758 Ignore-this: 70877ebf06ae59f32960b0aa4ce1d1ae ] [tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641. Brian Warner **20100127223517 Ignore-this: ab5cf05158d32a575ca8efc0f650033f ] [NEWS: update with all recent user-visible changes Brian Warner **20100127222209 Ignore-this: 277d24568018bf4f3fb7736fda64eceb ] ["tahoe backup": fix --exclude-vcs docs to include Git Brian Warner **20100127201044 Ignore-this: 756a58dde21bdc65aa62b81803605b5 ] [docs: fix references to --no-storage, explanation of [storage] section Brian Warner **20100127200956 Ignore-this: f4be1763a585e1ac6299a4f1b94a59e0 ] [cli: merge the better version of David-Sarah's split-usage-and-help patch with the earlier version that I mistakenly committed zooko@zooko.com**20100126044559 Ignore-this: 284d188e13b7901013cbb650168e6447 ] [Split tahoe --help options into groups. david-sarah@jacaranda.org**20100112043935 Ignore-this: 610f9c41b00e6863e3cd047379733e3a ] [docs: further CREDITS level-ups for Nils, Kevan, David-Sarah zooko@zooko.com**20100126170021 Ignore-this: 1e513e85cf7b7abf57f056e6d7544b38 ] [ftpd: clearer error message if Twisted needs a patch (by Nils Durner) zooko@zooko.com**20100126143411 Ignore-this: 440e6831ae6da5135c1edd081c93871f ] [Add 'docs/performance.txt', which (for the moment) describes mutable file performance issues Kevan Carstensen **20100115204500 Ignore-this: ade4e500217db2509aee35aacc8c5dbf ] [docs: more CREDITS for François, Kevan, and David-Sarah zooko@zooko.com**20100126132133 Ignore-this: f37d4977c13066fcac088ba98a31b02e ] [tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729. francois@ctrlaltdel.ch**20100120094249 Ignore-this: 7006ea4b0910b6d29af6ab4a3997a8f9 This patch displays a warning to the user in two cases: 1. When special files like symlinks, fifos, devices, etc. are found in the local source. 2. If files or directories are not readables by the user running the 'tahoe backup' command. In verbose mode, the number of skipped files and directories is printed at the end of the backup. Exit status returned by 'tahoe backup': - 0 everything went fine - 1 the backup failed - 2 files were skipped during the backup ] [Message saying that we couldn't find bin/tahoe should say where we looked david-sarah@jacaranda.org**20100116204556 Ignore-this: 1068576fd59ea470f1e19196315d1bb ] [Change running.html to describe 'tahoe run' david-sarah@jacaranda.org**20100112044409 Ignore-this: 23ad0114643ce31b56e19bb14e011e4f ] [cli: split usage strings into groups (patch by David-Sarah Hopwood) zooko@zooko.com**20100126043921 Ignore-this: 51928d266a7292b873f87f7d53c9a01e ] [Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760) david-sarah@jacaranda.org**20100116052055 Ignore-this: 47d08b18c69738685e13ff365738d5a ] [contrib/fuse/runtests.py: Fix #888, configure settings in tahoe.cfg and don't treat warnings as failure francois@ctrlaltdel.ch**20100109123010 Ignore-this: 2590d44044acd7dfa3690c416cae945c Fix a few bitrotten pieces in the FUSE test script. It now configures tahoe node settings by editing tahoe.cfg which is the new supported method. It alos tolerate warnings issued by the mount command, the cause of these warnings is the same as in #876 (contrib/fuse/runtests.py doesn't tolerate deprecations warnings). ] [Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919. Brian Warner **20100121065052 Ignore-this: 1f20ea0a0f1f6d6c1e8e14f193a92c87 ] [Fix boodlegrid use of set_children david-sarah@jacaranda.org**20100126063414 Ignore-this: 3aa2d4836f76303b2bacecd09611f999 ] [Remove replace= parameter to mkdir-immutable and mkdir-with-children david-sarah@jacaranda.org**20100124224325 Ignore-this: 25207bcc946c0c43d9528718e76ba7b ] [Warn about test failures due to setting FLOG* env vars david-sarah@jacaranda.org**20100124220629 Ignore-this: 1c25247ca0f0840390a1b7259a9f4a3c ] [Patch to accept t=set-children as well as t=set_children david-sarah@jacaranda.org**20100124030020 Ignore-this: 2c061f12af817cdf77feeeb64098ec3a ] [tahoe_add_alias.py: minor refactoring Brian Warner **20100115064220 Ignore-this: 29910e81ad11209c9e493d65fd2dab9b ] [test_dirnode.py: reduce scope of a Client instance, suggested by Kevan. Brian Warner **20100115062713 Ignore-this: b35efd9e6027e43de6c6f509bfb4ccaa ] [test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood. Brian Warner **20100115014632 Ignore-this: 9989de7f1e00907706d2b63153138219 ] [web/directory.py mkdir-immutable: hush pyflakes, add TODO for #903 behavior Brian Warner **20100114222804 Ignore-this: 717cd3b9a1c8aeee76938c9641db7356 ] [hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900. Brian Warner **20100114221719 Ignore-this: f774f4637e256ad55502659413a811a8 This includes one fix (in test_web) which was testing the wrong thing. ] [hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900. Brian Warner **20100114221529 Ignore-this: e96106c8f1a99fbf93306fbfe9a294cf ] [tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias Brian Warner **20100114210246 Ignore-this: 9c994792e53a85159d708760a9b1b000 file. Closes #741. ] [change docs and --help to use "grid" instead of "virtual drive": closes #892. Brian Warner **20100114201119 Ignore-this: a20d4a4dcc4de4e3b404ff72d40fc29b Thanks to David-Sarah Hopwood for the patch. ] [backupdb.txt: fix ST_CTIME reference Brian Warner **20100114194052 Ignore-this: 5a189c7a1181b07dd87f0a08ea31b6d3 ] [client.py: fix/update comments on KeyGenerator Brian Warner **20100113004226 Ignore-this: 2208adbb3fd6a911c9f44e814583cabd ] [Clean up log.err calls, for one of the issues in #889. Brian Warner **20100112013343 Ignore-this: f58455ce15f1fda647c5fb25d234d2db allmydata.util.log.err() either takes a Failure as the first positional argument, or takes no positional arguments and must be invoked in an exception handler. Fixed its signature to match both foolscap.logging.log.err and twisted.python.log.err . Included a brief unit test. ] [tidy up DeadReferenceError handling, ignore them in add_lease calls Brian Warner **20100112000723 Ignore-this: 72f1444e826fd0b9db6d318f89603c38 Stop checking separately for ConnectionDone/ConnectionLost, since those have been folded into DeadReferenceError since foolscap-0.3.1 . Write rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve code coverage. ] [NEWS: improve "tahoe backup" notes, mention first-backup-after-upgrade duration Brian Warner **20100111190132 Ignore-this: 10347c590b3375964579ba6c2b0edb4f Thanks to Francois Deppierraz for the suggestion. ] [test_repairer: add (commented-out) test_each_byte, to see exactly what the Brian Warner **20100110203552 Ignore-this: 8e84277d5304752edeff052b97821815 Verifier misses The results (described in #819) match our expectations: it misses corruption in unused share fields and in most container fields (which are only visible to the storage server, not the client). 1265 bytes of a 2753 byte share (hosting a 56-byte file with an artifically small segment size) are unused, mostly in the unused tail of the overallocated UEB space (765 bytes), and the allocated-but-unwritten plaintext_hash_tree (480 bytes). ] [repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian zooko@zooko.com**20100110203721 Ignore-this: 20604a609db8706555578612c1c12feb fixes #819 ] [test_repairer: fix colliding basedir names, which caused test inconsistencies Brian Warner **20100110084619 Ignore-this: b1d56dd27e6ab99a7730f74ba10abd23 ] [repairer: add deterministic test for #819, mark as TODO zooko@zooko.com**20100110013619 Ignore-this: 4cb8bb30b25246de58ed2b96fa447d68 ] [contrib/fuse/runtests.py: Tolerate the tahoe CLI returning deprecation warnings francois@ctrlaltdel.ch**20100109175946 Ignore-this: 419c354d9f2f6eaec03deb9b83752aee Depending on the versions of external libraries such as Twisted of Foolscap, the tahoe CLI can display deprecation warnings on stdout. The tests should not interpret those warnings as a failure if the node is in fact correctly started. See http://allmydata.org/trac/tahoe/ticket/859 for an example of deprecation warnings. fixes #876 ] [contrib: fix fuse_impl_c to use new Python API zooko@zooko.com**20100109174956 Ignore-this: 51ca1ec7c2a92a0862e9b99e52542179 original patch by Thomas Delaet, fixed by François, reviewed by Brian, committed by me ] [docs: CREDITS: add David-Sarah to the CREDITS file zooko@zooko.com**20100109060435 Ignore-this: 896062396ad85f9d2d4806762632f25a ] [mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877 Brian Warner **20100102220841 Ignore-this: b200e707b3f13aa8251981362b8a3e61 The bug was that a disconnected server could cause us to re-enter the initial loop() call, sending multiple queries to a single server, provoking an incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually() ] [immutable/checker.py: oops, forgot some imports. Also hush pyflakes. Brian Warner **20091229233909 Ignore-this: 4d61bd3f8113015a4773fd4768176e51 ] [mutable repair: return successful=False when numshares**20091229233746 Ignore-this: d881c3275ff8c8bee42f6a80ca48441e instead of weird errors. Closes #874 and #786. Previously, if the file had 0 shares, this would raise TypeError as it tried to call download_version(None). If the file had some shares but fewer than 'k', it would incorrectly raise MustForceRepairError. Added get_successful() to the IRepairResults API, to give repair() a place to report non-code-bug problems like this. ] [node.py/interfaces.py: minor docs fixes Brian Warner **20091229230409 Ignore-this: c86ad6342ef0f95d50639b4f99cd4ddf ] [NEWS: fix 1.4.1 announcement w.r.t. add-lease behavior in older releases Brian Warner **20091229230310 Ignore-this: bbbbb9c961f3bbcc6e5dbe0b1594822 ] [checker: don't let failures in add-lease affect checker results. Closes #875. Brian Warner **20091229230108 Ignore-this: ef1a367b93e4d01298c2b1e6ca59c492 Mutable servermap updates and the immutable checker, when run with add_lease=True, send both the do-you-have-block and add-lease commands in parallel, to avoid an extra round trip time. Many older servers have problems with add-lease and raise various exceptions, which don't generally matter. The client-side code was catching+ignoring some of them, but unrecognized exceptions were passed through to the DYHB code, concealing the DYHB results from the checker, making it think the server had no shares. The fix is to separate the code paths. Both commands are sent at the same time, but the errback path from add-lease is handled separately. Known exceptions are ignored, the others (both unknown-remote and all-local) are logged (log.WEIRD, which will trigger an Incident), but neither will affect the DYHB results. The add-lease message is sent first, and we know that the server handles them synchronously. So when the checker is done, we can be sure that all the add-lease messages have been retired. This makes life easier for unit tests. ] [test_cli: verify fix for "tahoe get" not creating empty file on error (#121) Brian Warner **20091227235444 Ignore-this: 6444d52413b68eb7c11bc3dfdc69c55f ] [addendum to "Fix 'tahoe ls' on files (#771)" Brian Warner **20091227232149 Ignore-this: 6dd5e25f8072a3153ba200b7fdd49491 tahoe_ls.py: tolerate missing metadata web/filenode.py: minor cleanups test_cli.py: test 'tahoe ls FILECAP' ] [Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen. Brian Warner **20091227225443 Ignore-this: 8bf8c7b1cd14ea4b0ebd453434f4fe07 web/filenode.py: also serve edge metadata when using t=json on a DIRCAP/childname object. tahoe_ls.py: list file objects as if we were listing one-entry directories. Show edge metadata if we have it, which will be true when doing 'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls FILECAP' ] [tahoe_get: don't create the output file on error. Closes #121. Brian Warner **20091227220404 Ignore-this: 58d5e793a77ec6e87d9394ade074b926 ] [webapi: don't accept zero-length childnames during traversal. Closes #358, #676. Brian Warner **20091227201043 Ignore-this: a9119dec89e1c7741f2289b0cad6497b This forbids operations that would implicitly create a directory with a zero-length (empty string) name, like what you'd get if you did "tahoe put local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error message is fairly friendly too. Also added code to "tahoe put" to catch this error beforehand and suggest the correct syntax (i.e. without the leading slash). ] [CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646. Brian Warner **20091227195828 Ignore-this: 44c258d4d4c7dac0ed58adb22f73331 The webapi has been looking for an Accept header since 1.4.0, but it treats a missing header as equal to */* (to honor RFC2616). This change finally modifies our CLI tools to ask for "text/plain, application/octet-stream", which seems roughly correct (we either want a plain-text traceback or error message, or an uninterpreted chunk of binary data to save to disk). Some day we'll figure out how JSON fits into this scheme. ] [Makefile: upload-tarballs: switch from xfer-client to flappclient, closes #350 Brian Warner **20091227163703 Ignore-this: 3beeecdf2ad9c2438ab57f0e33dcb357 I've also set up a new flappserver on source@allmydata.org to receive the tarballs. We still need to replace the gutsy buildslave (which is where the tarballs used to be generated+uploaded) and give it the new FURL. ] [misc/ringsim.py: make it deterministic, more detail about grid-is-full behavior Brian Warner **20091227024832 Ignore-this: a691cc763fb2e98a4ce1767c36e8e73f ] [misc/ringsim.py: tool to discuss #302 Brian Warner **20091226060339 Ignore-this: fc171369b8f0d97afeeb8213e29d10ed ] [docs: fix helper.txt to describe new config style zooko@zooko.com**20091224223522 Ignore-this: 102e7692dc414a4b466307f7d78601fe ] [docs/stats.txt: add TOC, notes about controlling gatherer's listening port Brian Warner **20091224202133 Ignore-this: 8eef63b0e18db5aa8249c2eafde02c05 Thanks to Jody Harris for the suggestions. ] [Add docs/stats.py, explaining Tahoe stats, the gatherer, and the munin plugins. Brian Warner **20091223052400 Ignore-this: 7c9eeb6e5644eceda98b59a67730ccd5 ] [more #859: avoid deprecation warning for unit tests too, hush pyflakes Brian Warner **20091215000147 Ignore-this: 193622e24d31077da825a11ed2325fd3 * factor maybe-import-sha logic into util.hashutil ] [use hashlib module if available, thus avoiding a DeprecationWarning for importing the old sha module; fixes #859 zooko@zooko.com**20091214212703 Ignore-this: 8d0f230a4bf8581dbc1b07389d76029c ] [docs: reflow architecture.txt to 78-char lines zooko@zooko.com**20091208232943 Ignore-this: 88f55166415f15192e39407815141f77 ] [docs: update the about.html a little zooko@zooko.com**20091208212737 Ignore-this: 3fe2d9653c6de0727d3e82bd70f2a8ed ] [docs: remove obsolete doc file "codemap.txt" zooko@zooko.com**20091113163033 Ignore-this: 16bc21a1835546e71d1b344c06c61ebb I started to update this to reflect the current codebase, but then I thought (a) nobody seemed to notice that it hasn't been updated since December 2007, and (b) it will just bit-rot again, so I'm removing it. ] [mutable/retrieve.py: stop reaching into private MutableFileNode attributes Brian Warner **20091208172921 Ignore-this: 61e548798c1105aed66a792bf26ceef7 ] [mutable/servermap.py: stop reaching into private MutableFileNode attributes Brian Warner **20091208172608 Ignore-this: b40a6b62f623f9285ad96fda139c2ef2 ] [mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e Brian Warner **20091208171156 Ignore-this: 3497f4ab70dae906759007c3cfa43bc under normal conditions, this wouldn't cause any problems, but if the shares are really sparse (perhaps because new servers were added), then file-modifies might stop looking too early and leave old shares in place ] [control.py: fix speedtest: use download_best_version (not read) on mutable nodes Brian Warner **20091207060512 Ignore-this: 7125eabfe74837e05f9291dd6414f917 ] [FTP-and-SFTP.txt: fix ssh-keygen pointer Brian Warner **20091207052803 Ignore-this: bc2a70ee8c58ec314e79c1262ccb22f7 ] [setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir zooko@zooko.com**20091206184835 Ignore-this: 6066bd160f0db36d7bf60aba405558d2 ] [remove MutableFileNode.download(), prefer download_best_version() instead Brian Warner **20091201225438 Ignore-this: 5733eb373a902063e09fd52cc858dec0 ] [Simplify immutable download API: use just filenode.read(consumer, offset, size) Brian Warner **20091201225330 Ignore-this: bdedfb488ac23738bf52ae6d4ab3a3fb * remove Downloader.download_to_data/download_to_filename/download_to_filehandle * remove download.Data/FileName/FileHandle targets * remove filenode.download/download_to_data/download_to_filename methods * leave Downloader.download (the whole Downloader will go away eventually) * add util.consumer.MemoryConsumer/download_to_data, for convenience (this is mostly used by unit tests, but it gets used by enough non-test code to warrant putting it in allmydata.util) * update tests * removes about 180 lines of code. Yay negative code days! Overall plan is to rewrite immutable/download.py and leave filenode.read() as the sole read-side API. ] [server.py: undo my bogus 'correction' of David-Sarah's comment fix Brian Warner **20091201024607 Ignore-this: ff4bb58f6a9e045b900ac3a89d6f506a and move it to a better line ] [Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen. "Brian Warner "**20091130211009] [storage.py: update comment "Brian Warner "**20091130195913] [storage server: detect disk space usage on Windows too (fixes #637) david-sarah@jacaranda.org**20091121055644 Ignore-this: 20fb30498174ce997befac7701fab056 ] [make status of finished operations consistently "Finished" david-sarah@jacaranda.org**20091121061543 Ignore-this: 97d483e8536ccfc2934549ceff7055a3 ] [NEWS: update with all user-visible changes since the last release Brian Warner **20091127224217 Ignore-this: 741da6cd928e939fb6d21a61ea3daf0b ] [update "tahoe backup" docs, and webapi.txt's mkdir-with-children Brian Warner **20091127055900 Ignore-this: defac1fb9a2335b0af3ef9dbbcc67b7e ] [Add dirnodes to backupdb and "tahoe backup", closes #606. Brian Warner **20091126234257 Ignore-this: fa88796fcad1763c6a2bf81f56103223 * backups now share dirnodes with any previous backup, in any location, so renames and moves are handled very efficiently * "tahoe backup" no longer bothers reading the previous snapshot * if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite, to force new uploads of all files and directories ] [webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories) Brian Warner **20091126232731 Ignore-this: 8513c890525c69c1eca0e80d53a231f8 ] [PipelineError: fix str() on python2.4 . Closes #842. Brian Warner **20091124212512 Ignore-this: e62c92ea9ede2ab7d11fe63f43b9c942 ] [test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore Brian Warner **20091120075553 Ignore-this: 61c8ef5e45a9d966873a610d8349b830 ] [interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode Brian Warner **20091120075255 Ignore-this: e3d193c229e2463e1d0b0c92306de27f The proper hierarchy is: IFilesystemNode +IFileNode ++IMutableFileNode ++IImmutableFileNode +IDirectoryNode Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types. ] [class name cleanups: s/FileNode/ImmutableFileNode/ Brian Warner **20091120072239 Ignore-this: 4b3218f2d0e585c62827e14ad8ed8ac1 also fix test/bench_dirnode.py for recent dirnode changes ] [Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828 Brian Warner **20091118192813 Ignore-this: a4720529c9bc6bc8b22a3d3265925491 ] [web/directory.py: use "DIR-IMM" to describe immutable directories, not DIR-RO Brian Warner **20091118191832 Ignore-this: aceafd6ab4bf1cc0c2a719ef7319ac03 ] [web/info.py: hush pyflakes Brian Warner **20091118191736 Ignore-this: edc5f128a2b8095fb20686a75747c8 ] [make get_size/get_current_size consistent for all IFilesystemNode classes Brian Warner **20091118191624 Ignore-this: bd3449cf96e4827abaaf962672c1665a * stop caching most_recent_size in dirnode, rely upon backing filenode for it * start caching most_recent_size in MutableFileNode * return None when you don't know, not "?" * only render None as "?" in the web "more info" page * add get_size/get_current_size to UnknownNode ] [ImmutableDirectoryURIVerifier: fix verifycap handling Brian Warner **20091118164238 Ignore-this: 6bba5c717b54352262eabca6e805d590 ] [Add t=mkdir-immutable to the webapi. Closes #607. Brian Warner **20091118070900 Ignore-this: 311e5fab9a5f28b9e8a28d3d08f3c0d * change t=mkdir-with-children to not use multipart/form encoding. Instead, the request body is all JSON. t=mkdir-immutable uses this format too. * make nodemaker.create_immutable_dirnode() get convergence from SecretHolder, but let callers override it * raise NotDeepImmutableError instead of using assert() * add mutable= argument to DirectoryNode.create_subdirectory(), default True ] [move convergence secret into SecretHolder, next to lease secret Brian Warner **20091118015444 Ignore-this: 312f85978a339f2d04deb5bcb8f511bc ] [nodemaker: implement immutable directories (internal interface), for #607 Brian Warner **20091112002233 Ignore-this: d09fccf41813fdf7e0db177ed9e5e130 * nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT * client.create_immutable_dirnode() is used to create them * no webapi yet ] [stop using IURI()/etc as an adapter Brian Warner **20091111224542 Ignore-this: 9611da7ea6a4696de2a3b8c08776e6e0 ] [clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings Brian Warner **20091111222619 Ignore-this: 93626385f6e7f039ada71f54feefe267 * "cap" means a python instance which encapsulates a filecap/dircap (uri.py) * "uri" means a string with a "URI:" prefix * FileNode instances are created with (and retain) a cap instance, and generate uri strings on demand * .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances * .get_uri/get_readonly_uri return uri strings * add filenode.download_to_filename() for control.py, should find a better way * use MutableFileNode.init_from_cap, not .init_from_uri * directory URI instances: use get_filenode_cap, not get_filenode_uri * update/cleanup bench_dirnode.py to match, add Makefile target to run it ] [add parser for immutable directory caps: DIR2-CHK, DIR2-LIT, DIR2-CHK-Verifier Brian Warner **20091104181351 Ignore-this: 854398cc7a75bada57fa97c367b67518 ] [wui: s/TahoeLAFS/Tahoe-LAFS/ zooko@zooko.com**20091029035050 Ignore-this: 901e64cd862e492ed3132bd298583c26 ] [tests: bump up the timeout on test_repairer to see if 120 seconds was too short for François's ARM box to do the test even when it was doing it right. zooko@zooko.com**20091027224800 Ignore-this: 95e93dc2e018b9948253c2045d506f56 ] [dirnode.pack_children(): add deep_immutable= argument Brian Warner **20091026162809 Ignore-this: d5a2371e47662c4bc6eff273e8181b00 This will be used by DIR2:CHK to enforce the deep-immutability requirement. ] [webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir . Brian Warner **20091026011321 Ignore-this: 769cab30b6ab50db95000b6c5a524916 This is safer: in the earlier API, an old webapi server would silently ignore the initial children, and clients trying to set them would have to fetch the newly-created directory to discover the incompatibility. In the new API, clients using t=mkdir-with-children against an old webapi server will get a clear error. ] [nodemaker.create_new_mutable_directory: pack_children() in initial_contents= Brian Warner **20091020005118 Ignore-this: bd43c4eefe06fd32b7492bcb0a55d07e instead of creating an empty file and then adding the children later. This should speed up mkdir(initial_children) considerably, removing two roundtrips and an entire read-modify-write cycle, probably bringing it down to a single roundtrip. A quick test (against the volunteergrid) suggests a 30% speedup. test_dirnode: add new tests to enforce the restrictions that interfaces.py claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts ] [test_dirnode.py: add tests of initial_children= args to client.create_dirnode Brian Warner **20091017194159 Ignore-this: 2e2da28323a4d5d815466387914abc1b and nodemaker.create_new_mutable_directory ] [update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps Brian Warner **20091017192829 Ignore-this: b35472285143862a856bf4b361d692f0 interfaces.py: define INodeMaker, document argument values, change create_new_mutable_directory() to take dict-of-nodes. Change dirnode.set_nodes() and dirnode.create_subdirectory() too. nodemaker.py: use INodeMaker, update create_new_mutable_directory() client.py: have create_dirnode() delegate initial_children= to nodemaker dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which updates set_nodes() and create_subdirectory() web/common.py (convert_initial_children_json): create dict-of-nodes web/directory.py: same web/unlinked.py: same test_dirnode.py: update tests to match ] [dirnode.py: move pack_children() out to a function, for eventual use by others Brian Warner **20091017180707 Ignore-this: 6a823fb61f2c180fd38d6742d3196a7a ] [move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names, Brian Warner **20091017180005 Ignore-this: b086933cf429df0fcea16a308d2640dd improve tests. Let dirnode _pack_children accept either dict or AuxValueDict. ] [test/common.py: update FakeMutableFileNode to new contents= callable scheme Brian Warner **20091013052154 Ignore-this: 62f00a76454a2190d1c8641c5993632f ] [The initial_children= argument to nodemaker.create_new_mutable_directory is Brian Warner **20091013031922 Ignore-this: 72e45317c21f9eb9ec3bd79bd4311f48 now enabled. ] [client.create_mutable_file(contents=) now accepts a callable, which is Brian Warner **20091013031232 Ignore-this: 3c89d2f50c1e652b83f20bd3f4f27c4b invoked with the new MutableFileNode and is supposed to return the initial contents. This can be used by e.g. a new dirnode which needs the filenode's writekey to encrypt its initial children. create_mutable_file() still accepts a bytestring too, or None for an empty file. ] [webapi: t=mkdir now accepts initial children, using the same JSON that t=json Brian Warner **20091013023444 Ignore-this: 574a46ed46af4251abf8c9580fd31ef7 emits. client.create_dirnode(initial_children=) now works. ] [replace dirnode.create_empty_directory() with create_subdirectory(), which Brian Warner **20091013021520 Ignore-this: 6b57cb51bcfcc6058d0df569fdc8a9cf takes an initial_children= argument ] [dirnode.set_children: change return value: fire with self instead of None Brian Warner **20091013015026 Ignore-this: f1d14e67e084e4b2a4e25fa849b0e753 ] [dirnode.set_nodes: change return value: fire with self instead of None Brian Warner **20091013014546 Ignore-this: b75b3829fb53f7399693f1c1a39aacae ] [dirnode.set_children: take a dict, not a list Brian Warner **20091013002440 Ignore-this: 540ce72ce2727ee053afaae1ff124e21 ] [dirnode.set_uri/set_children: change signature to take writecap+readcap Brian Warner **20091012235126 Ignore-this: 5df617b2d379a51c79148a857e6026b1 instead of a single cap. The webapi t=set_children call benefits too. ] [replace Client.create_empty_dirnode() with create_dirnode(), in anticipation Brian Warner **20091012224506 Ignore-this: cbdaa4266ecb3c6496ffceab4f95709d of adding initial_children= argument. Includes stubbed-out initial_children= support. ] [test_web.py: use a less-fake client, making test harness smaller Brian Warner **20091012222808 Ignore-this: 29e95147f8c94282885c65b411d100bb ] [webapi.txt: document t=set_children, other small edits Brian Warner **20091009200446 Ignore-this: 4d7e76b04a7b8eaa0a981879f778ea5d ] [Verifier: check the full cryptext-hash tree on each share. Removed .todos Brian Warner **20091005221849 Ignore-this: 6fb039c5584812017d91725e687323a5 from the last few test_repairer tests that were waiting on this. ] [Verifier: check the full block-hash-tree on each share Brian Warner **20091005214844 Ignore-this: 3f7ccf6d253f32340f1bf1da27803eee Removed the .todo from two test_repairer tests that check this. The only remaining .todos are on the three crypttext-hash-tree tests. ] [Verifier: check the full share-hash chain on each share Brian Warner **20091005213443 Ignore-this: 3d30111904158bec06a4eac22fd39d17 Removed the .todo from two test_repairer tests that check this. ] [test_repairer: rename Verifier test cases to be more precise and less verbose Brian Warner **20091005201115 Ignore-this: 64be7094e33338c7c2aea9387e138771 ] [immutable/checker.py: rearrange code a little bit, make it easier to follow Brian Warner **20091005200252 Ignore-this: 91cc303fab66faf717433a709f785fb5 ] [test/common.py: wrap docstrings to 80cols so I can read them more easily Brian Warner **20091005200143 Ignore-this: b180a3a0235cbe309c87bd5e873cbbb3 ] [immutable/download.py: wrap to 80cols, no functional changes Brian Warner **20091005192542 Ignore-this: 6b05fe3dc6d78832323e708b9e6a1fe ] [CHK-hashes.svg: cross out plaintext hashes, since we don't include Brian Warner **20091005010803 Ignore-this: bea2e953b65ec7359363aa20de8cb603 them (until we finish #453) ] [docs: a few licensing clarifications requested by Ubuntu zooko@zooko.com**20090927033226 Ignore-this: 749fc8c9aeb6dc643669854a3e81baa7 ] [setup: remove binary WinFUSE modules zooko@zooko.com**20090924211436 Ignore-this: 8aefc571d2ae22b9405fc650f2c2062 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate o r acquire the binaries as needed. Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic. (Technicall y, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, bu t it is easier for now to remove the binaries from the source tree.) In this case, the binaries are from the tahoe-w32-client project: http://allmydata.org/trac/tahoe-w32-client , from which you can also get the source. ] [setup: remove binary _fusemodule.so 's zooko@zooko.com**20090924211130 Ignore-this: 74487bbe27d280762ac5dd5f51e24186 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate or acquire the binaries as needed. Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic. (Technically, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, but it is easier for now to remove the binaries from the source tree.) In this case, these modules come from the MacFUSE project: http://code.google.com/p/macfuse/ ] [doc: add a copy of LGPL2 for documentation purposes for ubuntu zooko@zooko.com**20090924054218 Ignore-this: 6a073b48678a7c84dc4fbcef9292ab5b ] [setup: remove a convenience copy of figleaf, to ease inclusion into Ubuntu Karmic Koala zooko@zooko.com**20090924053215 Ignore-this: a0b0c990d6e2ee65c53a24391365ac8d We need to carefully document the licence of figleaf in order to get Tahoe-LAFS into Ubuntu Karmic Koala. However, figleaf isn't really a part of Tahoe-LAFS per se -- this is just a "convenience copy" of a development tool. The quickest way to make Tahoe-LAFS acceptable for Karmic then, is to remove figleaf from the Tahoe-LAFS tarball itself. People who want to run figleaf on Tahoe-LAFS (as everyone should want) can install figleaf themselves. I haven't tested this -- there may be incompatibilities between upstream figleaf and the copy that we had here... ] [setup: shebang for misc/build-deb.py to fail quickly zooko@zooko.com**20090819135626 Ignore-this: 5a1b893234d2d0bb7b7346e84b0a6b4d Without this patch, when I ran "chmod +x ./misc/build-deb.py && ./misc/build-deb.py" then it hung indefinitely. (I wonder what it was doing.) ] [docs: Shawn Willden grants permission for his contributions under GPL2+|TGPPL1+ zooko@zooko.com**20090921164651 Ignore-this: ef1912010d07ff2ffd9678e7abfd0d57 ] [docs: Csaba Henk granted permission to license fuse.py under the same terms as Tahoe-LAFS itself zooko@zooko.com**20090921154659 Ignore-this: c61ba48dcb7206a89a57ca18a0450c53 ] [setup: mark setup.py as having utf-8 encoding in it zooko@zooko.com**20090920180343 Ignore-this: 9d3850733700a44ba7291e9c5e36bb91 ] [doc: licensing cleanups zooko@zooko.com**20090920171631 Ignore-this: 7654f2854bf3c13e6f4d4597633a6630 Use nice utf-8 © instead of "(c)". Remove licensing statements on utility modules that have been assigned to allmydata.com by their original authors. (Nattraverso was not assigned to allmydata.com -- it was LGPL'ed -- but I checked and src/allmydata/util/iputil.py was completely rewritten and doesn't contain any line of code from nattraverso.) Add notes to misc/debian/copyright about licensing on files that aren't just allmydata.com-licensed. ] [build-deb.py: run darcsver early, otherwise we get the wrong version later on Brian Warner **20090918033620 Ignore-this: 6635c5b85e84f8aed0d8390490c5392a ] [new approach for debian packaging, sharing pieces across distributions. Still experimental, still only works for sid. warner@lothar.com**20090818190527 Ignore-this: a75eb63db9106b3269badbfcdd7f5ce1 ] [new experimental deb-packaging rules. Only works for sid so far. Brian Warner **20090818014052 Ignore-this: 3a26ad188668098f8f3cc10a7c0c2f27 ] [setup.py: read _version.py and pass to setup(version=), so more commands work Brian Warner **20090818010057 Ignore-this: b290eb50216938e19f72db211f82147e like "setup.py --version" and "setup.py --fullname" ] [test/check_speed.py: fix shbang line Brian Warner **20090818005948 Ignore-this: 7f3a37caf349c4c4de704d0feb561f8d ] [setup: remove bundled version of darcsver-1.2.1 zooko@zooko.com**20090816233432 Ignore-this: 5357f26d2803db2d39159125dddb963a That version of darcsver emits a scary error message when the darcs executable or the _darcs subdirectory is not found. This error is hidden (unless the --loud option is passed) in darcsver >= 1.3.1. Fixes #788. ] [de-Service-ify Helper, pass in storage_broker and secret_holder directly. Brian Warner **20090815201737 Ignore-this: 86b8ac0f90f77a1036cd604dd1304d8b This makes it more obvious that the Helper currently generates leases with the Helper's own secrets, rather than getting values from the client, which is arguably a bug that will likely be resolved with the Accounting project. ] [immutable.Downloader: pass StorageBroker to constructor, stop being a Service Brian Warner **20090815192543 Ignore-this: af5ab12dbf75377640a670c689838479 child of the client, access with client.downloader instead of client.getServiceNamed("downloader"). The single "Downloader" instance is scheduled for demolition anyways, to be replaced by individual filenode.download calls. ] [tests: double the timeout on test_runner.RunNode.test_introducer since feisty hit a timeout zooko@zooko.com**20090815160512 Ignore-this: ca7358bce4bdabe8eea75dedc39c0e67 I'm not sure if this is an actual timing issue (feisty is running on an overloaded VM if I recall correctly), or it there is a deeper bug. ] [stop making History be a Service, it wasn't necessary Brian Warner **20090815114415 Ignore-this: b60449231557f1934a751c7effa93cfe ] [Overhaul IFilesystemNode handling, to simplify tests and use POLA internally. Brian Warner **20090815112846 Ignore-this: 1db1b9c149a60a310228aba04c5c8e5f * stop using IURI as an adapter * pass cap strings around instead of URI instances * move filenode/dirnode creation duties from Client to new NodeMaker class * move other Client duties to KeyGenerator, SecretHolder, History classes * stop passing Client reference to dirnode/filenode constructors - pass less-powerful references instead, like StorageBroker or Uploader * always create DirectoryNodes by wrapping a filenode (mutable for now) * remove some specialized mock classes from unit tests Detailed list of changes (done one at a time, then merged together) always pass a string to create_node_from_uri(), not an IURI instance always pass a string to IFilesystemNode constructors, not an IURI instance stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri() client.py: move SecretHolder code out to a separate class test_web.py: hush pyflakes client.py: move NodeMaker functionality out into a separate object LiteralFileNode: stop storing a Client reference immutable Checker: remove Client reference, it only needs a SecretHolder immutable Upload: remove Client reference, leave SecretHolder and StorageBroker immutable Repairer: replace Client reference with StorageBroker and SecretHolder immutable FileNode: remove Client reference mutable.Publish: stop passing Client mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference MutableChecker: reference StorageBroker and History directly, not through Client mutable.FileNode: removed unused indirection to checker classes mutable.FileNode: remove Client reference client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker move create_mutable_file() into NodeMaker test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests. test_mutable.py: clean up basedir names client.py: move create_empty_dirnode() into NodeMaker dirnode.py: get rid of DirectoryNode.create remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker remove Client from NodeMaker move helper status into History, pass History to web.Status instead of Client test_mutable.py: fix minor typo ] [docs: edits for docs/running.html from Sam Mason zooko@zooko.com**20090809201416 Ignore-this: 2207e80449943ebd4ed50cea57c43143 ] [docs: install.html: instruct Debian users to use this document and not to go find the DownloadDebianPackages page, ignore the warning at the top of it, and try it zooko@zooko.com**20090804123840 Ignore-this: 49da654f19d377ffc5a1eff0c820e026 http://allmydata.org/pipermail/tahoe-dev/2009-August/002507.html ] [docs: relnotes.txt: reflow to 63 chars wide because google groups and some web forms seem to wrap to that zooko@zooko.com**20090802135016 Ignore-this: 53b1493a0491bc30fb2935fad283caeb ] [docs: about.html: fix English usage noticed by Amber zooko@zooko.com**20090802050533 Ignore-this: 89965c4650f9bd100a615c401181a956 ] [docs: fix mis-spelled word in about.html zooko@zooko.com**20090802050320 Ignore-this: fdfd0397bc7cef9edfde425dddeb67e5 ] [TAG allmydata-tahoe-1.5.0 zooko@zooko.com**20090802031303 Ignore-this: 94e5558e7225c39a86aae666ea00f166 ] Patch bundle hash: 20c5d329ef51ad6be17e8796387ecd5663a6424d