integrity failure in mutable retrieve (SDMF+MDMF) in 1.9.0

[warner]

This is the ticket where we will track the details of a failure of assurance of integrity. For now, here is the announcement:

Dear Tahoe-LAFS Users:

Kevan Carstensen of the Tahoe-LAFS core team has discovered a security
vulnerability in Tahoe-LAFS v1.9.0 which would allow a sufficiently clever
attacker to corrupt the retrieval of mutable files or directories which are
retrieved with v1.9.0 or, in some cases, to corrupt the stored copy of
mutable files or directories which are updated with v1.9.0.

The recommended defensive action for all users is to downgrade to v1.8.3, or
to refrain from using mutable files (either SDMF or MDMF) with 1.9.0.

A FAQ about downgrading from 1.9.0 to 1.8.3, which was written before we
discovered this critical security vulnerability, is here:

https://tahoe-lafs.org/pipermail/tahoe-dev/2011-December/006905.html

The FAQ is no longer accurate about 1.9.0 being free of dangerous flaws, but
it is still accurate about 1.8.3 being free of compatibility problems.

We'll be providing a patch soon. We are still writing tests for it and
searching for other similar bugs and so on. Of course, as soon as we release
the patch, this will inform any attackers of exactly what they could do to
users of 1.9.0. Therefore, if there are any users who are especially
security-sensitive, then they should downgrade to 1.8.3 before we release the
patch, or else they should suspend their use of mutable files and directories
until we released the patch and they've applied it.

Once we are ready to publish the details of the issue we will post them to
this issue tracker ticket:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1654

Please feel free to contact me with any questions or concerns, using GPG
encryption. Please Cc: Brian Warner, David-Sarah Hopwood, and Kevan
Carstensen on all such email.

Regards,

Zooko Wilcox-O'Hearn, on behalf of the Tahoe-LAFS core team


GPG fingerprints:

Brian Warner <warner-tahoe@lothar.com> 967E FE06 6998 7241 1A77  DF36 D43B 4C9C 7322 5AAF
David-Sarah Hopwood <david-sarah@jacaranda.org> 3D6A 08E9 1262 3E9A 00B2  1BDC 067F 4920 98CF 2762
Kevan Carstensen <kevan@isnotajoke.com> 7E1E 99DB 97B1 DD5F 8154  5973 8E6B 2106 2425 D7AE
Zooko Wilcox-O'Hearn <zooko@zooko.com> A60B 7EE1 7164 D0C5 F137  3868 5F22 F428 242B E85F

[warner]

fixed formatting of announcement, removed pubkeys, removed signature wrapper (not verifiable here)

[Brian Warner <warner@…>]

In d10f260f36a9da6f:

retrieve.py: unconditionally check share-hash-tree. Fixes #1654.

Add Kevan's unit test, update known_issues.rst

[Brian Warner <warner@…>]

In d10f260f36a9da6f:

retrieve.py: unconditionally check share-hash-tree. Fixes #1654.

Add Kevan's unit test, update known_issues.rst

[warner]

Dear Tahoe-LAFS Users:

On 08-Jan-2012, Tahoe-LAFS core member Kevan Carstensen (author of the MDMF code) discovered a serious bug in v1.9.0 (the current stable release) that allows attackers to corrupt downloads of mutable files in certain cases. We've released Tahoe-LAFS v1.9.1 which removes this vulnerability. All users are encouraged to upgrade immediately to v1.9.1, or to downgrade to v1.8.3.

v1.9.0 was released about two months ago. As far as we know, ArchLinux? is the only distribution to have packaged v1.9.0 (the others are still on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a non-ArchLinux? package, you're probably fine. If you build it yourself, you should upgrade.

In Tahoe, files are encrypted, and then encoded into multiple redundant shares. Integrity-checking information (Merkle hash trees) are included in the shares to detect corruption. When downloading, these hashes are checked before combining the shares in the decoder, which generates ciphertext that can be decrypted into the original file. Mutable files have two sets of hash trees, the "share hash tree" (which covers all shares), and the "block hash trees" (which sit under the share-hash-tree and cover the individual blocks that make up each share, one block per 128KiB segment of the original file).

The new mutable downloader released in v1.9.0, which supports both the old-style SDMF format and the new MDMF format, has a bug in which the share-hash-tree check is accidentaly bypassed when the Merkle hash tree is already fully populated. This doesn't normally occur, but shares can contain additional hash-tree nodes beyond the ones they strictly need. An attacker could modify one share to include the entire tree, then change the block data in the remaining shares. They would need to update the block-hash-trees in those doctored shares, but because of the bug, these tree roots will not be compared against the share hash tree.

The attacker is thus able to control the input to the ZFEC decoder for all but the first share received (which must have valid block data). This gives them the ability to flip bits of the plaintext without triggering the CorruptShareError? exceptions that share corruption would normally produce, causing corrupted plaintext to be delivered to an unwitting client.

To exploit this bug, the attacker must be able to deliver multiple modified shares to your client, in a particular order: this means they must control one or more of your storage servers.

Note that this does not directly reveal the plaintext to the attacker (this is an integrity failure, not a confidentiality failure). However, "encryption without authentication" is never a safe state of affairs, and can frequently be exploited to reveal information about the plaintext (perhaps by inducing observable failures by flipping bits in messages of a known format). In addition, clients which read corrupted data as part of a read-modify-write operation (such as directory modifications) may then write the corrupted data back out to the file, making the corruption persist even after the client has been fixed.

v1.9.1 fixes this by removing the accidental "if" clause, making the share-hash-tree check unconditional.

The specific bug is in source:src/allmydata/mutable/retrieve.py​, Retrieve._validate_block, around the call to share_hash_tree.set_hashes(), and was introduced in git revisionid [ac3b2647dd2c45cd1ddbf5b130ee5a780c66c73b] with the MDMF-capable downloader rewrite around 01-Aug-2011. The bug was first present in shipping code in Tahoe-LAFS-1.9.0, on 30-Oct-2011. It was fixed in commit [9b4b03a474a2c9050c8347459ab6698839be7288], shipped in Tahoe-LAFS-1.9.1 on 12-Jan-2012. We are continuing to audit the 1.9.x mutable downloader code to search for other potential bugs.

[gdt]

As far as we know, ArchLinux?? is the only distribution to have packaged v1.9.0 (the others are still on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a non-ArchLinux?? package, you're probably fine.

pkgsrc also packaged 1.9.0. (However, pkgsrc isn't a "distribution", in that it is not a Linux kernel and a group of user-space programs. Instead, it is a packaging system that supports multiple operating systems.) The tahoe-lafs entry in pkgsrc was updated about 2011-01-13T0115Z to 1.9.1.

[zooko]

I got mail from Kurt Seifried <kseifried@…> saying to use CVE-2012-0051 to denote this vulnerability.