<!--{{{-->
<link rel='alternate' type='application/rss+xml' title='RSS' href='index.xml' />
<!--}}}-->

Background: #fff
Foreground: #000
PrimaryPale: #8cf
PrimaryLight: #18f
PrimaryMid: #04b
PrimaryDark: #014
SecondaryPale: #ffc
SecondaryLight: #fe8
SecondaryMid: #db4
SecondaryDark: #841
TertiaryPale: #eee
TertiaryLight: #ccc
TertiaryMid: #999
TertiaryDark: #666
Error: #f88

/*{{{*/
body {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}

a {color:[[ColorPalette::PrimaryMid]];}
a:hover {background-color:[[ColorPalette::PrimaryMid]]; color:[[ColorPalette::Background]];}
a img {border:0;}

h1,h2,h3,h4,h5,h6 {color:[[ColorPalette::SecondaryDark]]; background:transparent;}
h1 {border-bottom:2px solid [[ColorPalette::TertiaryLight]];}
h2,h3 {border-bottom:1px solid [[ColorPalette::TertiaryLight]];}

.button {color:[[ColorPalette::PrimaryDark]]; border:1px solid [[ColorPalette::Background]];}
.button:hover {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::SecondaryLight]]; border-color:[[ColorPalette::SecondaryMid]];}
.button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::SecondaryDark]];}

.header {background:[[ColorPalette::PrimaryMid]];}
.headerShadow {color:[[ColorPalette::Foreground]];}
.headerShadow a {font-weight:normal; color:[[ColorPalette::Foreground]];}
.headerForeground {color:[[ColorPalette::Background]];}
.headerForeground a {font-weight:normal; color:[[ColorPalette::PrimaryPale]];}

.tabSelected{color:[[ColorPalette::PrimaryDark]];
	background:[[ColorPalette::TertiaryPale]];
	border-left:1px solid [[ColorPalette::TertiaryLight]];
	border-top:1px solid [[ColorPalette::TertiaryLight]];
	border-right:1px solid [[ColorPalette::TertiaryLight]];
}
.tabUnselected {color:[[ColorPalette::Background]]; background:[[ColorPalette::TertiaryMid]];}
.tabContents {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::TertiaryPale]]; border:1px solid [[ColorPalette::TertiaryLight]];}
.tabContents .button {border:0;}

#sidebar {}
#sidebarOptions input {border:1px solid [[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel {background:[[ColorPalette::PrimaryPale]];}
#sidebarOptions .sliderPanel a {border:none;color:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:hover {color:[[ColorPalette::Background]]; background:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:active {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::Background]];}

.wizard {background:[[ColorPalette::PrimaryPale]]; border:1px solid [[ColorPalette::PrimaryMid]];}
.wizard h1 {color:[[ColorPalette::PrimaryDark]]; border:none;}
.wizard h2 {color:[[ColorPalette::Foreground]]; border:none;}
.wizardStep {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];
	border:1px solid [[ColorPalette::PrimaryMid]];}
.wizardStep.wizardStepDone {background:[[ColorPalette::TertiaryLight]];}
.wizardFooter {background:[[ColorPalette::PrimaryPale]];}
.wizardFooter .status {background:[[ColorPalette::PrimaryDark]]; color:[[ColorPalette::Background]];}
.wizard .button {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryLight]]; border: 1px solid;
	border-color:[[ColorPalette::SecondaryPale]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryPale]];}
.wizard .button:hover {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Background]];}
.wizard .button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::Foreground]]; border: 1px solid;
	border-color:[[ColorPalette::PrimaryDark]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryDark]];}

.wizard .notChanged {background:transparent;}
.wizard .changedLocally {background:#80ff80;}
.wizard .changedServer {background:#8080ff;}
.wizard .changedBoth {background:#ff8080;}
.wizard .notFound {background:#ffff80;}
.wizard .putToServer {background:#ff80ff;}
.wizard .gotFromServer {background:#80ffff;}

#messageArea {border:1px solid [[ColorPalette::SecondaryMid]]; background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]];}
#messageArea .button {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::SecondaryPale]]; border:none;}

.popupTiddler {background:[[ColorPalette::TertiaryPale]]; border:2px solid [[ColorPalette::TertiaryMid]];}

.popup {background:[[ColorPalette::TertiaryPale]]; color:[[ColorPalette::TertiaryDark]]; border-left:1px solid [[ColorPalette::TertiaryMid]]; border-top:1px solid [[ColorPalette::TertiaryMid]]; border-right:2px solid [[ColorPalette::TertiaryDark]]; border-bottom:2px solid [[ColorPalette::TertiaryDark]];}
.popup hr {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::PrimaryDark]]; border-bottom:1px;}
.popup li.disabled {color:[[ColorPalette::TertiaryMid]];}
.popup li a, .popup li a:visited {color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:active {background:[[ColorPalette::SecondaryPale]]; color:[[ColorPalette::Foreground]]; border: none;}
.popupHighlight {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
.listBreak div {border-bottom:1px solid [[ColorPalette::TertiaryDark]];}

.tiddler .defaultCommand {font-weight:bold;}

.shadow .title {color:[[ColorPalette::TertiaryDark]];}

.title {color:[[ColorPalette::SecondaryDark]];}
.subtitle {color:[[ColorPalette::TertiaryDark]];}

.toolbar {color:[[ColorPalette::PrimaryMid]];}
.toolbar a {color:[[ColorPalette::TertiaryLight]];}
.selected .toolbar a {color:[[ColorPalette::TertiaryMid]];}
.selected .toolbar a:hover {color:[[ColorPalette::Foreground]];}

.tagging, .tagged {border:1px solid [[ColorPalette::TertiaryPale]]; background-color:[[ColorPalette::TertiaryPale]];}
.selected .tagging, .selected .tagged {background-color:[[ColorPalette::TertiaryLight]]; border:1px solid [[ColorPalette::TertiaryMid]];}
.tagging .listTitle, .tagged .listTitle {color:[[ColorPalette::PrimaryDark]];}
.tagging .button, .tagged .button {border:none;}

.footer {color:[[ColorPalette::TertiaryLight]];}
.selected .footer {color:[[ColorPalette::TertiaryMid]];}

.sparkline {background:[[ColorPalette::PrimaryPale]]; border:0;}
.sparktick {background:[[ColorPalette::PrimaryDark]];}

.error, .errorButton {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Error]];}
.warning {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryPale]];}
.lowlight {background:[[ColorPalette::TertiaryLight]];}

.zoomer {background:none; color:[[ColorPalette::TertiaryMid]]; border:3px solid [[ColorPalette::TertiaryMid]];}

.imageLink, #displayArea .imageLink {background:transparent;}

.annotation {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border:2px solid [[ColorPalette::SecondaryMid]];}

.viewer .listTitle {list-style-type:none; margin-left:-2em;}
.viewer .button {border:1px solid [[ColorPalette::SecondaryMid]];}
.viewer blockquote {border-left:3px solid [[ColorPalette::TertiaryDark]];}

.viewer table, table.twtable {border:2px solid [[ColorPalette::TertiaryDark]];}
.viewer th, .viewer thead td, .twtable th, .twtable thead td {background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::Background]];}
.viewer td, .viewer tr, .twtable td, .twtable tr {border:1px solid [[ColorPalette::TertiaryDark]];}

.viewer pre {border:1px solid [[ColorPalette::SecondaryLight]]; background:[[ColorPalette::SecondaryPale]];}
.viewer code {color:[[ColorPalette::SecondaryDark]];}
.viewer hr {border:0; border-top:dashed 1px [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::TertiaryDark]];}

.highlight, .marked {background:[[ColorPalette::SecondaryLight]];}

.editor input {border:1px solid [[ColorPalette::PrimaryMid]];}
.editor textarea {border:1px solid [[ColorPalette::PrimaryMid]]; width:100%;}
.editorFooter {color:[[ColorPalette::TertiaryMid]];}

#backstageArea {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::TertiaryMid]];}
#backstageArea a {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstageArea a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; }
#backstageArea a.backstageSelTab {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
#backstageButton a {background:none; color:[[ColorPalette::Background]]; border:none;}
#backstageButton a:hover {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstagePanel {background:[[ColorPalette::Background]]; border-color: [[ColorPalette::Background]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]];}
.backstagePanelFooter .button {border:none; color:[[ColorPalette::Background]];}
.backstagePanelFooter .button:hover {color:[[ColorPalette::Foreground]];}
#backstageCloak {background:[[ColorPalette::Foreground]]; opacity:0.6; filter:'alpha(opacity:60)';}
/*}}}*/

/*{{{*/
* html .tiddler {height:1%;}

body {font-size:.75em; font-family:arial,helvetica; margin:0; padding:0;}

h1,h2,h3,h4,h5,h6 {font-weight:bold; text-decoration:none;}
h1,h2,h3 {padding-bottom:1px; margin-top:1.2em;margin-bottom:0.3em;}
h4,h5,h6 {margin-top:1em;}
h1 {font-size:1.35em;}
h2 {font-size:1.25em;}
h3 {font-size:1.1em;}
h4 {font-size:1em;}
h5 {font-size:.9em;}

hr {height:1px;}

a {text-decoration:none;}

dt {font-weight:bold;}

ol {list-style-type:decimal;}
ol ol {list-style-type:lower-alpha;}
ol ol ol {list-style-type:lower-roman;}
ol ol ol ol {list-style-type:decimal;}
ol ol ol ol ol {list-style-type:lower-alpha;}
ol ol ol ol ol ol {list-style-type:lower-roman;}
ol ol ol ol ol ol ol {list-style-type:decimal;}

.txtOptionInput {width:11em;}

#contentWrapper .chkOptionInput {border:0;}

.externalLink {text-decoration:underline;}

.indent {margin-left:3em;}
.outdent {margin-left:3em; text-indent:-3em;}
code.escaped {white-space:nowrap;}

.tiddlyLinkExisting {font-weight:bold;}
.tiddlyLinkNonExisting {font-style:italic;}

/* the 'a' is required for IE, otherwise it renders the whole tiddler in bold */
a.tiddlyLinkNonExisting.shadow {font-weight:bold;}

#mainMenu .tiddlyLinkExisting,
	#mainMenu .tiddlyLinkNonExisting,
	#sidebarTabs .tiddlyLinkNonExisting {font-weight:normal; font-style:normal;}
#sidebarTabs .tiddlyLinkExisting {font-weight:bold; font-style:normal;}

.header {position:relative;}
.header a:hover {background:transparent;}
.headerShadow {position:relative; padding:4.5em 0em 1em 1em; left:-1px; top:-1px;}
.headerForeground {position:absolute; padding:4.5em 0em 1em 1em; left:0px; top:0px;}

.siteTitle {font-size:3em;}
.siteSubtitle {font-size:1.2em;}

#mainMenu {position:absolute; left:0; width:10em; text-align:right; line-height:1.6em; padding:1.5em 0.5em 0.5em 0.5em; font-size:1.1em;}

#sidebar {position:absolute; right:3px; width:16em; font-size:.9em;}
#sidebarOptions {padding-top:0.3em;}
#sidebarOptions a {margin:0em 0.2em; padding:0.2em 0.3em; display:block;}
#sidebarOptions input {margin:0.4em 0.5em;}
#sidebarOptions .sliderPanel {margin-left:1em; padding:0.5em; font-size:.85em;}
#sidebarOptions .sliderPanel a {font-weight:bold; display:inline; padding:0;}
#sidebarOptions .sliderPanel input {margin:0 0 .3em 0;}
#sidebarTabs .tabContents {width:15em; overflow:hidden;}

.wizard {padding:0.1em 1em 0em 2em;}
.wizard h1 {font-size:2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizard h2 {font-size:1.2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizardStep {padding:1em 1em 1em 1em;}
.wizard .button {margin:0.5em 0em 0em 0em; font-size:1.2em;}
.wizardFooter {padding:0.8em 0.4em 0.8em 0em;}
.wizardFooter .status {padding:0em 0.4em 0em 0.4em; margin-left:1em;}
.wizard .button {padding:0.1em 0.2em 0.1em 0.2em;}

#messageArea {position:fixed; top:2em; right:0em; margin:0.5em; padding:0.5em; z-index:2000; _position:absolute;}
.messageToolbar {display:block; text-align:right; padding:0.2em 0.2em 0.2em 0.2em;}
#messageArea a {text-decoration:underline;}

.tiddlerPopupButton {padding:0.2em 0.2em 0.2em 0.2em;}
.popupTiddler {position: absolute; z-index:300; padding:1em 1em 1em 1em; margin:0;}

.popup {position:absolute; z-index:300; font-size:.9em; padding:0; list-style:none; margin:0;}
.popup .popupMessage {padding:0.4em;}
.popup hr {display:block; height:1px; width:auto; padding:0; margin:0.2em 0em;}
.popup li.disabled {padding:0.4em;}
.popup li a {display:block; padding:0.4em; font-weight:normal; cursor:pointer;}
.listBreak {font-size:1px; line-height:1px;}
.listBreak div {margin:2px 0;}

.tabset {padding:1em 0em 0em 0.5em;}
.tab {margin:0em 0em 0em 0.25em; padding:2px;}
.tabContents {padding:0.5em;}
.tabContents ul, .tabContents ol {margin:0; padding:0;}
.txtMainTab .tabContents li {list-style:none;}
.tabContents li.listLink { margin-left:.75em;}

#contentWrapper {display:block;}
#splashScreen {display:none;}

#displayArea {margin:1em 17em 0em 14em;}

.toolbar {text-align:right; font-size:.9em;}

.tiddler {padding:1em 1em 0em 1em;}

.missing .viewer,.missing .title {font-style:italic;}

.title {font-size:1.6em; font-weight:bold;}

.missing .subtitle {display:none;}
.subtitle {font-size:1.1em;}

.tiddler .button {padding:0.2em 0.4em;}

.tagging {margin:0.5em 0.5em 0.5em 0; float:left; display:none;}
.isTag .tagging {display:block;}
.tagged {margin:0.5em; float:right;}
.tagging, .tagged {font-size:0.9em; padding:0.25em;}
.tagging ul, .tagged ul {list-style:none; margin:0.25em; padding:0;}
.tagClear {clear:both;}

.footer {font-size:.9em;}
.footer li {display:inline;}

.annotation {padding:0.5em; margin:0.5em;}

* html .viewer pre {width:99%; padding:0 0 1em 0;}
.viewer {line-height:1.4em; padding-top:0.5em;}
.viewer .button {margin:0em 0.25em; padding:0em 0.25em;}
.viewer blockquote {line-height:1.5em; padding-left:0.8em;margin-left:2.5em;}
.viewer ul, .viewer ol {margin-left:0.5em; padding-left:1.5em;}

.viewer table, table.twtable {border-collapse:collapse; margin:0.8em 1.0em;}
.viewer th, .viewer td, .viewer tr,.viewer caption,.twtable th, .twtable td, .twtable tr,.twtable caption {padding:3px;}
table.listView {font-size:0.85em; margin:0.8em 1.0em;}
table.listView th, table.listView td, table.listView tr {padding:0px 3px 0px 3px;}

.viewer pre {padding:0.5em; margin-left:0.5em; font-size:1.2em; line-height:1.4em; overflow:auto;}
.viewer code {font-size:1.2em; line-height:1.4em;}

.editor {font-size:1.1em;}
.editor input, .editor textarea {display:block; width:100%; font:inherit;}
.editorFooter {padding:0.25em 0em; font-size:.9em;}
.editorFooter .button {padding-top:0px; padding-bottom:0px;}

.fieldsetFix {border:0; padding:0; margin:1px 0px 1px 0px;}

.sparkline {line-height:1em;}
.sparktick {outline:0;}

.zoomer {font-size:1.1em; position:absolute; overflow:hidden;}
.zoomer div {padding:1em;}

* html #backstage {width:99%;}
* html #backstageArea {width:99%;}
#backstageArea {display:none; position:relative; overflow: hidden; z-index:150; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageToolbar {position:relative;}
#backstageArea a {font-weight:bold; margin-left:0.5em; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageButton {display:none; position:absolute; z-index:175; top:0em; right:0em;}
#backstageButton a {padding:0.1em 0.4em 0.1em 0.4em; margin:0.1em 0.1em 0.1em 0.1em;}
#backstage {position:relative; width:100%; z-index:50;}
#backstagePanel {display:none; z-index:100; position:absolute; width:90%; margin:0em 3em 0em 3em; padding:1em 1em 1em 1em;}
.backstagePanelFooter {padding-top:0.2em; float:right;}
.backstagePanelFooter a {padding:0.2em 0.4em 0.2em 0.4em;}
#backstageCloak {display:none; z-index:20; position:absolute; width:100%; height:100px;}

.whenBackstage {display:none;}
.backstageVisible .whenBackstage {display:block;}
/*}}}*/

/***
StyleSheet for use when a translation requires any css style changes.
This StyleSheet can be used directly by languages such as Chinese, Japanese and Korean which need larger font sizes.
***/
/*{{{*/
body {font-size:0.8em;}
#sidebarOptions {font-size:1.05em;}
#sidebarOptions a {font-style:normal;}
#sidebarOptions .sliderPanel {font-size:0.95em;}
.subtitle {font-size:0.8em;}
.viewer table.listView {font-size:0.95em;}
/*}}}*/

/*{{{*/
@media print {
#mainMenu, #sidebar, #messageArea, .toolbar, #backstageButton, #backstageArea {display: none ! important;}
#displayArea {margin: 1em 1em 0em 1em;}
/* Fixes a feature in Firefox 1.5.0.2 where print preview displays the noscript content */
noscript {display:none;}
}
/*}}}*/

<!--{{{-->
<div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'>
<div class='headerShadow'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div class='headerForeground'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
</div>
<div id='mainMenu' refresh='content' tiddler='MainMenu'></div>
<div id='sidebar'>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->

<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::ViewToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class='subtitle'><span macro='view modifier link'></span>, <span macro='view modified date'></span> (<span macro='message views.wikified.createdPrompt'></span> <span macro='view created date'></span>)</div>
<div class='tagging' macro='tagging'></div>
<div class='tagged' macro='tags'></div>
<div class='viewer' macro='view text wikified'></div>
<div class='tagClear'></div>
<!--}}}-->

<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::EditToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class='editor' macro='edit title'></div>
<div macro='annotations'></div>
<div class='editor' macro='edit text'></div>
<div class='editor' macro='edit tags'></div><div class='editorFooter'><span macro='message views.editor.tagPrompt'></span><span macro='tagChooser excludeLists'></span></div>
<!--}}}-->

To get started with this blank TiddlyWiki, you'll need to modify the following tiddlers:
* SiteTitle & SiteSubtitle: The title and subtitle of the site, as shown above (after saving, they will also appear in the browser title bar)
* MainMenu: The menu (usually on the left)
* DefaultTiddlers: Contains the names of the tiddlers that you want to appear when the TiddlyWiki is opened
You'll also need to enter your username for signing your edits: <<option txtUserName>>

These InterfaceOptions for customising TiddlyWiki are saved in your browser

Your username for signing your edits. Write it as a WikiWord (eg JoeBloggs)

<<option txtUserName>>
<<option chkSaveBackups>> SaveBackups
<<option chkAutoSave>> AutoSave
<<option chkRegExpSearch>> RegExpSearch
<<option chkCaseSensitiveSearch>> CaseSensitiveSearch
<<option chkAnimate>> EnableAnimations

----
Also see [[AdvancedOptions]]

<<importTiddlers>>

Christian Grothoff is the third (or zeroth, depending on how you count) winner of the "Hack Tahoe!" contest!  Christian found a cryptographic flaw in ~Tahoe-LAFS a few months back, and we confirmed that it was real and fixed it.  We designed a custom t-shirt for him which has an explanation of his exploit on it, and awarded it to him in person. He has been added to [[the "Hack Tahoe!" Hall of Fame|http://hacktahoe.org]].

Yesterday I posted [[a note|http://allmydata.org/pipermail/tahoe-dev/2008-July/000729.html]] to tahoe-dev about garbage collection and did some work on [[my paper for StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].  It turns out that they've changed the due date from August 3 to August 18.  Great!  Now I'm starting to work on it and it isn't the last minute!

Today I'm going to pair up with Brian, either on the paper or on immutable file checking and repair (and possibly improved download), or perhaps on both.

I'm investigating [[this Debian/Ubuntu bug|https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035]], the better to make Brian accept Tahoe's reliance on setuptools.  We could work around this Debian/Ubuntu bug by using only our own bundled copy of setuptools that comes with Tahoe, but I'm sure Brian would be more accepting of it if the version of setuptools that came with Debian/Ubuntu worked...

I'm feeling overwhelmed with things I Ought To Do.  The top priority is new checker/verifier/repairer for immutable files for Tahoe -- it is, unless I'm forgetting something -- the last feature that needs to be implemented for the Tahoe 1.3 release.

But I also have a hard deadline of August 18 for the final version of [[my paper for StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].

Per some of [[the reviewers' suggestions|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]] I'm (re-)reading some of the storage papers by the estimable [[David Mazières|http://www.scs.stanford.edu/~dm]], starting with [[the SUNDR paper|http://www.scs.stanford.edu/~dm/home/papers/li:sundr.pdf]].

Also some hackers have started [[revitalizing the darcs project|http://lists.osuosl.org/pipermail/darcs-users/2008-August/012939.html]], and I have some responsibilities, to do with building Windows binaries of darcs, but more-over to do with automation: automated testing, automated building of binaries on all supported platforms, and automated performance measurement.  Unfortunately I'm not going to be able to contribute anything to this project until at least after August 18.

Of course there are also one zillion other things that I need to, want to, or ought to do ASAP...

I [[offered|http://mail.python.org/pipermail/python-dev/2008-August/081758.html]] some ascii-encoding code of mine to the Python project.

I updated a [[darcs code browser web site|http://allmydata.org/trac/darcs-2/browser]] so that the darcs folks can link to it from [[the darcs home page|http://darcs.net]].

I contributed [[a patch|http://mail.python.org/pipermail/distutils-sig/2008-August/009815.html]] for the [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]] documents.

I updated a couple of tickets on [[http://allmydata.org|http://allmydata.org]]: [[#456 (it would be nice if the dependency on OpenSSL could be automatically resolved)|http://allmydata.org/trac/tahoe/ticket/456#comment:6]] and [[#402 (bug in Twisted, triggered by pyOpenSSL-0.7)|http://allmydata.org/trac/tahoe/ticket/402#comment:18]].

I worked on reproducing [[this problem|http://allmydata.org/pipermail/tahoe-dev/2008-August/000736.html]] with installing Tahoe on Mac OS X.  I'm pretty sure that the problem in Tahoe-1.2.0 was already fixed by [[this patch|http://allmydata.org/trac/tahoe/changeset/2803]], but in attempting to reproduce David Evans's experience to be sure before I wrote back to him I ran into a related bug in the Mac OS X packaging of pyOpenSSL.

So I helped the [[pyOpenSSL|https://launchpad.net/pyopenssl]] developers run unit tests of pyOpenSSL on my Mac: [[https://bugs.launchpad.net/pyopenssl/+bug/236170|https://bugs.launchpad.net/pyopenssl/+bug/236170 #  test failures on Mac OS X with openssl-0.9.8[gh] ]].  Here is the [[pyOpenSSL buildbot|http://buildbot.twistedmatrix.com/waterfall-pyopenssl]].

I chatted quite a bit with ~RobK, the allmydata, UK arm.  I encouraged him to spend some time creating beautiful, eye-popping, richly informative visualizations of Tahoe.

I queried Brian Warner about exactly what he wants from darcs and took notes.  Tomorrow I'll post those notes to the darcs folks.  Maybe Brian will get what he wants out of darcs.  Or maybe someone else will eventually benefit from those kinds of improvements even if it is too late for Tahoe to do.  Or if that doesn't work then at the very least there will be a good document showing what it was about darcs that was unsatisfactory.

Okay, it is now time for me to head home and prepare for aikido class, and I've done exactly nothing on the two major urgent important tasks that I mentioned at the beginning of the day: ~StorageSS08 paper and new file checking/verifier/repair.  I think I should plan some complete blanket block on distractions tomorrow -- no e-mail, no IRC, no IM.  If Brian (or anyone) wants to talk to me about ~StorageSS08 paper or about new file checking/verifier/repair then they'll have to call me...  Oh wait, I still [[don't have phone service|http://www.dailycamera.com/news/2008/aug/08/south-boulder-phone-repair-70-percent-finished]].  I guess they'll have to call my wife's cell phone and ask her to come find me and tell me to call them back.

Good morning, world!  It is 7:10 AM in Boulder, Colorado and the sun is shining!

My wife has some meetings and classes today, so I am going to be responsible for children in the morning and in the evening.  I'll probably just tell the boys to run and play and then get work done.  They are a lot better at entertaining themselves for hours at a time than they were last summer.

School starts next week -- [[Boulder Valley School District calendar|http://www.bvsd.org/calendar/Pages/default.aspx]].

I updated http://allmydata.org/buildbot-darcs to always redirect to http://buildbot.darcs.net, which is a nicer URL and which also correctly serves up [[its master.cfg file|http://<b style="color: black; background-color: rgb(255, 255, 102);">buildbot</b>.darcs.net/master.cfg]].  This was because [[Petr|http://is.muni.cz/lide/?fakulta=1433;obdobi=2825;studium=150956;jazyk=en;uco=139761]] [[Rockai|http://behindkde.org/people/mornfall/]] tried to access the master.cfg from the old URL and it gave him an error.  Hopefully the reason he is looking at that master.cfg is in order to automate more [[darcs|http://darcs.net]] processes such as building binaries and running benchmarks.

I see that my Mac OS 10.4 builder is now green on [[the pyOpenSSL buildbot|http://buildbot.twistedmatrix.com/waterfall-pyopenssl]].

I updated [[Twisted ticket #2234 (Select default reactor based on platform and available libraries)|http://twistedmatrix.com/trac/ticket/2234#comment:6]], about what reactor Twisted should use if the user didn't specify any preference, and incidentally about integrating trial with setuptools.

I read some of the recent [[darcs-users mailing list traffic|http://lists.osuosl.org/pipermail/darcs-users/2008-August/date.html]].  There are a lot of people eager to improve darcs in various ways.  I was reminded of an earlier [[post from Brian Warner|http://allmydata.org/pipermail/tahoe-dev/2007-November/000228.html]] explaining what he wants from his revision control tool.  (I was reminded of it because it was hyperlinked from [[the darcs wiki|http://wiki.darcs.net]].)  In order to communicate his needs to the darcs developers, I probably just need to link to that letter and then reiterate the essential points, which he expressed succinctly in IRC conversation yesterday.

Finally closed [[ticket #11 (I don't like pyOpenSSL)|http://allmydata.org/trac/tahoe/ticket/11]].

I noticed that the darcs network tests are hanging on two solaris boxes and an ~OpenBSD box -- see [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] for tall yellow stripes.  One of those solaris boxes is mine, so I'm going to try upgrading the [[libcurl|http://curl.haxx.se/]] on that box and see if that changes the behavior.

I opened [[a new ticket|http://bugs.darcs.net/issue991]] about a bug in darcs -- stack space overflow when you try to "darcs put" 10,000 patches over ssh.

In the attempt to get an instant messaging client with [[OTR|http://www.cypherpunks.ca/otr]] I installed [[Adium|http://www.adiumx.com]] on my Macbook Pro and [[kopete|http://kopete.kde.org]] on my Linux laptop.  Kopete didn't work for me, so I opened [[a bug report|https://bugs.launchpad.net/ubuntu/+source/kopete-plugin-otr-kde4/+bug/257390]] for it on launchpad and installed [[pidgin|http://www.pidgin.im]] which did work.

I installed [[iTerm|http://iterm.sourceforge.net]] on my Macbook Pro.  Hopefully this will end my persistent problems with Terminal.app, such as cut and paste eating the last character of every line and/or eating the line feed chars.

Okay, now I need to figure out how to integrate this blog with the internal allmydata.com employees' blog so that my co-workers can easily see what I am doing.

Oh, yesterday I finished re-reading [[SUNDR|http://www.scs.cs.nyu.edu/~dm/papers/li:sundr.pdf]].  I didn't actually understand the key mechanism to enable concurrency while retaining fork-consistency, but I don't think I really need to in order to correctly cite it in [[my StorageSS08 paper|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]].

I opened a ticket on the darcs issue tracker to record Brian's desire for short, secure, efficient version identifiers.  I think it is a valuable feature, and that it is more feasible for darcs than people think.  [[http://bugs.darcs.net/issue992 (short, secure, fast version identifiers)|http://bugs.darcs.net/issue992]]

I made a small note about [[issue #348 (BuildBot step to run tests from package)|http://allmydata.org/trac/tahoe/ticket/348#comment:5]].

The openssl developers [[were arguing|http://www.mail-archive.com/openssl-dev@openssl.org/msg24270.html]] about how to be compatible with valgrind debugging while minimizing the chances of another [["Debian OpenSSL debacle"|http://google.com/search?q=debian+openssl+debacle]] or other problems.  I subscribed to the openssl-dev list just to post [[a suggestion|http://marc.info/?l=openssl-dev&m=121856771027223&w=2]] for what seems like the best way to do it.

Ooh, Danny O'Brien wrote back and asked if I would mind if he wrote some documentation for Tahoe that was less technical than what we've got now.  Would I MIND?  Whoo!  That would be great!

[[Professor James Plank|http://www.cs.utk.edu/~plank]] and his student Katie Schuman have done some benchmarks of open source erasure coding software.  Here are [[their results so far|http://www.cs.utk.edu/~plank/plank/papers/CS-08-625.html]].  Zfec comes out looking pretty good.  I need to write a few suggestions for them on their paper.

Yesterday I wrote [[a letter to Jeremy Fitzhardinge on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-August/000743.html]] about convergent encryption in Tahoe, the "Chosen Protocol Attack" (Kelsey, Schneier, Wagner), and the Tiger hash.  And I wrote to [img[Danny O'Brien|/file/URI%3ACHK%3Av3rks3qwih4ietrtzwrfuur4ru%3Akz3gcobjest4bdr2yugwphk26bzx3beb3t6baqvusn2eo3smw5ja%3A3%3A10%3A108574/@@named=/danny_obrien_on_cover_of_wired.png]] <-- Danny O'Brien urging him to write docs for Tahoe and suggesting that he contact my business leader at [[allmydata.com|http://allmydata.com]], [img[Peter Secor|http://www.npost.com/images/interviews/peter_secor.jpg]] <-- Peter Secor.

One advantage of having Danny O'Brien write the user docs for Tahoe is that they'll probably turn out [[a lot funnier|http://www.ntk.net]] than the ones [img[Zooko being silly|/file/URI:CHK:ywcrkt5vm3jiwdtyb43n4kgfpu:qydseoirjn5kbpoydgptlaim2spas7a6fsbhdcg6umj2f3sqqdpq:3:10:49576/@@named=/Zooko_being_silly.jpg]] <-- I write.

Today I learned how to inline images into tiddlywiki.

I spent some time working darcs today -- chatting with darcs hackers who are very active in the #darcs channel on irc.freenode.net nowadays, debugged an issue with the latest libcurl and the latest darcs on solaris, and urged someone (Eric Kow) to plan a darcs-2.0.3 release in order to get the latest improvements out there and to exercise the machinery of non-~David-Roundy-led releases.  I really ought to test out Gwern's cabalization patches on Windows soon.

Okay now I would really like to explore using FND's ~HTTPSavingPlugin so that I can use modern non-hacked ~TiddlyWiki code on my tahoe-hosted blog, but instead I'm going to work on [[my StorageSS08 paper|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].

[[agl|http://www.imperialviolet.org]] wrote to tell me about his Curve22519 implementation.  On request, he explained his reasons to choose Curve25519 instead of FIPS curves.  He says that other curves than Curve25519 ''may'' require validation of the public key before computation of ~Diffie-Hellman and ''may'' be insecure if you use the same public key for many ~Diffie-Hellman agreements.  I think and hope that he is wrong about both, but I need to investigate more...

Hooray!  [[Allmydata.com|http://www.allmydata.com/index.php?tracking=zookos_hlog]] just released the new native Windows client for Tahoe.  It is called Allmydata 3.1.  If you or someone you know uses Windows, check it out!

Good morning, World!

It is 8:54 here, and it looks like another beautiful day.  So far this morning, I've done a bunch of home administration work (with ~OpenOffice, Firefox, ~XEmacs, and Darcs -- maybe adding Gnucash in the future), some chatting on the #darcs IRC channel and opened [[a ticket on the darcs bug tracker|http://bugs.darcs.net/issue997]] showing them example code from Tahoe of how to program your buildbot to compile and upload packages.

Yay!  The new [[LWN.net|http://lwn.net]] is here!

Followed [[Danny O'Brien's blog|http://www.oblomovka.com]] to [[Danny O'Brien's delicious links|http://delicious.com/malaclyps]] to [[Wes Felter's blog|http://wmf.editthispage.com]].  Oh dear -- I need to stop reading the Internet and starting [[writing|http://zooko.com/lafs.pdf]].

I got up early this morning.

I'm making some progress on [[the paper|http://zooko.com/lafs.pdf]].  There are a lot more useful [[reviewer comments|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]] that I need to use.

I printed out [[Efficient byzantine-tolerant erasure-coded storage (2004)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.2568]], which I probably need to cite in lafs.pdf, and I wrote [[a note to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-August/000756.html]] about [[A Performance Comparison of Open-Source Erasure Coding Libraries for Storage Applications|http://www.cs.utk.edu/~plank/plank/papers/CS-08-625.html]].

I chatted quite a bit with ~R-K, the mysterious off-shore hacker.  I urged him to dive right in and write a Mac client for Tahoe, doing it just like he wants in terms of technology (Cocoa) and features (whatever features he likes), instead of spending time formulating a business requirements document.  And I urged him to Release Early, Release Often and make it open source.

Because, you know, Time is of the Essence, and cranking out some cool hack in the way that you like is a good strategy.

Hooray!  The paper for [[StorageSS'08|http://storagess.org]] is finished!

[[http://zooko.com/lafs.pdf|http://zooko.com/lafs.pdf]]

I submitted [[a patch to setuptools|http://bugs.python.org/setuptools/issue14]].

I updated [[dupfilefind|http://allmydata.org/trac/dupfilefind]] and [[posted about it|http://allmydata.org/pipermail/tahoe-dev/2008-August/000760.html]].

* Worked on getting the pyOpenSSL project to [[build and host binaries|https://bugs.launchpad.net/pyopenssl/+bug/238658]] so that people can install Tahoe without a C compiler and ~OpenSSL development headers.
* Helped the Debian developer fellow, Micah, post to the tahoe-dev list.
* Helped the darcs folks set up [[a buildslave on NetBSD|http://buildbot.darcs.net/waterfall]] and tried to figure out why the darcs buildslave on our Windows virtual machine gets hung and can't kill the darcs process.
* Read the latest news from [[the SHA-3 competition|http://ehash.iaik.tugraz.at/wiki/The_SHA-3_Zoo]].  Boole is broken!  ~CubeHash is analyzed by two external teams!  ~EnRUPT is updated!
* Tweaked [[the spreadsheet|http://docs.google.com]] -- I had incorrectly told Peter that the HP DL 185 G5 cost a mere $1650, which would reduce our "Storage cost per GB per year for user space" from $0.94 to $0.34.  The actual [[cost of the HP when properly configured|http://allmydata.org/~zooko/hp-DL185-G5-configured.html]] is $2028, which apparently changes our storage cost per GB per year for user space from $0.34 to $0.35.  On the other hand, we can get Seagate Barracuda 7200.11 1.5 TB for [[around $150 each|http://storagereview.pricegrabber.com/search_getprod.php/masterid=88928483/st=product_tab]], which puts it back down to $0.33.  Also the Redundancy percentage was set to 101% instead of its current value of 333%, but that makes no difference as the other numbers were calculated to allow for 300%.  I wrote to my favorite system builder [[Kevin Chalker|http://kc-computers.com]] to ask if he could build rackmount systems with a ~SATA-drives-per-dollar as high as the HP.
* Brian pointed out that [[zbase32|http://pypi.python.org/pypi/zbase32]] didn't work on Python 2.6 because of the new reserved status of the token "as".  I uploaded a fix.
* Hooray!!  [[My hlog|http://tahoebs1.allmydata.com:8123/uri/URI:DIR2-RO:hgvn7nhforxhfxbx3nbej53qoi:yhbnnuxl4o2hr4sxuocoi735t6lcosdin72axkrcboulfslwbfwq/wiki.html]] is back!!
* See also [[my old-fashioned, centralized blog|https://zooko.com/log.html]].

This morning I read Scott Aaronson's research statement about [[computational complexity theory and quantum computing|http://www.scottaaronson.com/research.pdf]].  I understood little of it, except that our secure hashes need to have 3n bits for n-bit resistance to collisions, in case quantum computers ever work out.  I also read the ~SHA-3 submission for the [[SWIFFTX|http://www.eecs.harvard.edu/~alon/PAPERS/lattices/swifftx.pdf]] -- it is interesting because being able to break a random variant of SWIFFTX would entail that you can solve a hard problem in algebraic number theory.  But it is too big and slow for my purposes -- it looks like you would have to use thousands of bits of state to compute it at all, and even if you can use a desktop CPU with SIMD, it still takes 57 cycles per byte.

My plan to organize a Boulder Hack Fest while Jake Appelbaum is in town hasn't worked out because it is Thanksgiving.  I'm looking into having a Hack Fest the following week.

This morning I made sure that the Windows buildslave for darcs was running and posted [[about the plan|http://lists.osuosl.org/pipermail/darcs-users/2008-November/016025.html]] for releasing a new version of darcs for Windows.

I checked on the status  of my tickets on the divmod trac: [[nevow #2798 (setup.py install --home is broken :-()|http://divmod.org/trac/ticket/2798]], [[nevow #2699 (build nevow without importing nevow)|http://divmod.org/trac/ticket/2699]], [[nevow #2698 (please mail me when my tickets change)|http://divmod.org/trac/ticket/2698]], [[nevow #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way)|http://divmod.org/trac/ticket/2629]], [[nevow #2527 (easy_install compatibility)|http://divmod.org/trac/ticket/2527]]. They are all blocked waiting for someone else to do something.  Same with the pyOpenSSL binaries ticket [[pyopenssl #238658 (please provide binaries)|https://bugs.launchpad.net/pyopenssl/+bug/238658]].

I spent a minute forlornly wishing that someone other than me would spend the time to fix [[buildbot #212 (buildbot doesn't respond to darcs tags)|http://buildbot.net/trac/ticket/212]].

I made a new release of [[pycryptopp|http://allmydata.org/trac/pycryptopp]] -- v0.5.11.  The improvement over v0.5.10 is that the buggy ecdsa wrapper has been commented out.

I helped Dan ~McNair set up [[an Arch Linux buildbot for Tahoe|http://allmydata.org/buildbot/builders/Dan%20ArchLinux]] and helped Micah Anderson subscribe to tahoe-dev.

Oh, good, it turns out that the quantum algorithm to find hash collisions, which takes only about 2^(n/3) time, also takes about 2^(n/3) size of quantum computer.  Therefore the overall cost to find collisions in an ideal hash function, whether using classical or quantum computation, is probably about 2^(n/2).  On the other hand there ''is'' something that quantum computers -- if they ever come into existence -- will be able to do cheaper than classical computers, and that is find pre-images of an ideal hash function in a mere 2^(n/2) cost instead of the 2^n cost that a classical computer requires (Grover's Algorithm).

So as long as you use a hash function which prevents people from cheaply finding collisions (in today's, classical, terms), then you'll probably be okay.  This means it is still probably a reasonable strategy for Tahoe to use 192-bit-output hash functions in the next revision of its crypto capability scheme.

Thanks to ~DJBernstein for posting about that topic to the NIST mailing list.

I might go to [[the Boulder Hacking In Public Society|http://hackingsociety.org/chapters/blug]] meeting tonight.

I worked on packaging issues -- older installs of Tahoe could conflict with newer ones.  The fix to that (using the """"--multi-version"""" flag) led to other problems.  At the moment, most of the buildbots are red because of this.

Worked with Chris Galvan on setuptools_trial and packaging.  Worked with Greg Hazel on making darcs work on Windows for him and on making Python 2.6 work for me on Windows.  Worked with Brian Warner on versioning for Tahoe.  Worked on upgrading my hlog (Amber points out that it should be called a "clog") to the new version of Tiddly Wiki and the separate ~HTTPSavingPlugin.  It doesn't work yet.

Things I did yesterday -- 2008-11-24:
* upgrade this klog to use the current version of ~TiddlyWiki and the ~HTTPSavingPlugin thanks to help from FND
* open some tickets about improvements I'd like to see:
** [[tiddly_on_tahoe #1 (it says "saving please wait...done" *after* it is finished saving)|http://allmydata.org/trac/tiddly_on_tahoe/ticket/1]]
** [[tiddly_on_tahoe #2 (don't offer the option to save changes when you are viewing read-only)|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]
* spent a couple of minutes helping the darcs hackers wrangle [[the darcs buildbot|http://buildbot.darcs.net]] -- they now have automated uploads of binaries for Solaris and ~OpenBSD
* give Francois Deppierraz an account on the allmydata.com dapper machine so he can work on [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]]
* wrote a [[letter|http://www.nabble.com/poly1305-p20684551.html]] to Wei Dai about MAC algorithms in Crypto++ (finished that letter and sent it today - -2008-11-25)
* investigated a problem on the Tahoe trac as reported by Nathan Wilcox
* wrote a letter to Ian Goldberg about his advice on how to prove security (and improve security) for [[semi-private keys|http://allmydata.org/~zooko/lafs.pdf]]
* looked at the list of open tickets from my [[21 November 2008]] journal entry -- alas, none of them have been improved by anyone else since the last time I looked
* planned ~ETAs for immutable file repairer:
** ~ValidatedUEB ETA 2008-11-25
** Verifier ETA 2008-11-26
** Repairer ETA 2008-12-02

Things I did so far today -- 2008-11-25:
* chat with Debian Python packagers about using [[my stdeb hacks|https://code.launchpad.net/~astraw/stdeb/autofind-depends]] in [[their find_python_dependencies.py|http://svn.debian.org/viewsvn/*checkout*/python-modules/tools/find_python_dependencies.py]]
* learn how to use proper (~ISO-8601'ish) daystamps on ~TiddlyWiki and to automatically open today's journal entry, thanks to the ever-helpful FND a.k.a. ~Ace_NoOne
* fix the tahoe/pyutil "platform description string" feature to detect Arch Linux correctly, with the help of Daenyth from IRC
* add Dan ~McNair's builder to the list of builders to be run on patch commits and restart [[the tahoe buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]]
* checked whether the version of simplejson could explain [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]] -- unlikely
* posted [[about TiddlyWiki on Tahoe|http://allmydata.org/pipermail/tahoe-dev/2008-November/000893.html]] to the tahoe-dev list
* chatted with Brian for a few minute about [[tahoe #540 (inappropriate "uncoordinated write error" after handling a server failure)|http://allmydata.org/trac/tahoe/ticket/540]]
* helped Francois fix [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]]
* try to persuade maintainers of setuptools and setuptools forks to accept my patch to respect PYTHONPATH: [[message to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2008-November/010539.html]] (actually the "Enstaller" setuptools fork already accepted my patches, but that one is not publicly supported at this time -- it used only internally by the Enthought corporation).  Hm, this seems to have triggered at least one setuptools fork to become more active.  See the distutils-sig mailing list archives for details.
* respond to Denis Bonnenfant's [[message to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-November/000892.html]]
* change the default port number from 8123 to 3456 [[tahoe ticket #536 (port number conflict: 8123 is (or was) used by polipo and is blocked by TorButton)|http://allmydata.org/trac/tahoe/ticket/536]]

Today is our eighth wedding anniversary.

Things I did so far today:

* chatted with Roger Dingledine of Tor about the port conflict
* read about [[the LANE hash function|http://www.cosic.esat.kuleuven.be/lane/moreinfo.shtml]] which was submitted for the ~SHA-3 contest (see [[SHA-3 benchmarks and the importance of 32-bit CPUs]])
* learned about ~ECIES-ISO -- [[Shoup's ISO doc|http://www.shoup.net/iso/std6.pdf]], [[David Hopwood's page|http://www.users.zetnet.co.uk/hopwood/crypto/scan/ca.html#ECIES-ISO]], [[wikipedia page|http://en.wikipedia.org/wiki/Integrated_Encryption_Scheme]]; ~ECIES-ISO is a nice simple scheme compared to [[the ElGamal encryption scheme|http://en.wikipedia.org/wiki/ElGamal_encryption]].
* learned about ~PSEC-KEM and ~ACE-KEM and decided that ~ECIES-ISO would serve Tahoe better (with ~CofactorMode=0, ~OldCofactorMode=0, ~CheckMode=1 and ~SingleHashMode=0)
* With the help of cdent and Jeremy Ruston, I fixed a bug in [[Tiddly-On-Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]] in which it was extra-encoding the body when saving.  Each time I clicked "Save Changes", it escaped all of the previous escape characters, thus doubling the size of the strings encoding encodings of encodings of "David Mazières"; I noticed once the file size was up to 8 MB and it was taking an excessively long time to load and my web browsers were offering to kill the ~JavaScript because it was taking too long.
* noticed that there exists this project named [[CorePy|http://corepy.org]], which is an assembly language programming tool integrated with Python.  They announced their 1.0 release on [[lwn.net|http://lwn.net]] this week.  Mmmm...
* I tried in vain to configure my server -- nooxie.zooko.com -- to send outgoing mail from [[nmh|http://www.nongnu.org/nmh]]; So my quest to free myself from the Apple Mail.app MUA didn't make much progress.
* I did a little bit of work on the ~ValidatedUEB part of repairer -- the step of repairer that was supposed to be done yesterday.  :-(  But at least I got started.  :-)

For my morning's economic education, I listened to [[this week's econtalk|http://www.econtalk.org/archives/2008/11/hazlett_on_tele.html]], Thomas Hazlett on net neutrality, the Microsoft anti-trust case, the original browser wars, etc..  This isn't a good econtalk.  Econtalk doesn't work very well when the guest and the host (Russ Roberts) share biases and don't succeed at critically challenging their own biases.  (This one also seems to be marred by factual errors on the part of Thomas Hazlett -- see the comments on the econtalk blog for extensive discussion.)

If you are new to econtalk try one of the good ones: [[Shirky on Coase, Collaboration and Here Comes Everybody|http://www.econtalk.org/archives/2008/10/shirky_on_coase.html]], [[Patri Friedman on Seasteading|http://www.econtalk.org/archives/2008/10/patri_friedman.html]], [[Kling on Freddie and Fannie and the Recent History of the U.S. Housing Market|http://www.econtalk.org/archives/2008/09/kling_on_freddi.html]], [[Brook on Vermeer's Hat and the Dawn of Global Trade|http://www.econtalk.org/archives/2008/02/brook_on_vermee.html]], [[Hanson on Health|http://www.econtalk.org/archives/2007/05/hanson_on_healt.html]], [[Taleb on Black Swans|http://www.econtalk.org/archives/2007/04/taleb_on_black.html]].

11:00 ~UTC-7

I created a new tiddler: [[things to read]].

I'm learning how to customize ~TiddlyWiki.  Today I learned how to put things on the sidebar over on the right -- the first thing I put there is a blogroll, and the next thing is a link to [[things to read]].

Everything that I've learned about how to use ~TiddlyWiki is due to the friendly ~TiddlyWiki hacker [[FND|http://fnd.lewcid.org/blog]], who answers my questions on the #tiddlywiki IRC channel.

reading web pages:
* [[Matthew Garrett on power management|http://www.codon.org.uk/~mjg59/power/good_practices.html]] (found via [[LWN.net|http://lwn.net]])

There is a big-picture issue that people are missing; should your operating system and CPU be designed to execute tasks as quickly as possible in order to get them over with, or in a slow, foot-dragging way in order to save energy?  Obviously in an ideal world the applications are doing things that users need done, and so obviously the first strategy is the only answer.  However, in the real world many or even most of the tasks are useless, such as busy loops checking "Has anything changed yet?  Has anything changed yet?  Has anything changed yet?" every 100 milliseconds.  Those busy loops are not going to run to completion and then stop, they are going to keep running endlessly, and they are not going to accomplish anything for the user (until the time comes that the answer changes from "No" to "Yes").  If that is the workload, then the second strategy is better, because it can endure the wastage so that there is still some power left when the user wants to actually do something.

The wintel world ([[led by Transmeta|http://en.wikipedia.org/wiki/LongRun]]) seems to have pursued the latter strategy. Linux hackers like Matthew Garrett naturally tend toward the former strategy, along with the injunction the app writers ought to "Fix your apps!".  That's fine too -- //viva la difference!// -- but I wonder what happens when [[the app writers don't|http://lwn.net/Articles/308446]] (Beagle).

This reminds me of the trouble that we had with memory protection and scheduling in the previous generation of operating systems -- Windows < NT and Mac OS < X.  Back then (in the 90's), users would say "Hey my whole system just froze up!", and experts would explain to them "Ah, you need to figure out which of the apps you were running was naughty and tell the author to fix his app!".  These experts would argue that it wasn't the operating system's fault if apps misbehaved like that.

Eventually the makers of the operating systems took responsibility for preserving the user experience //even in the presence of naughty apps//, by enforcing memory protection and pre-emption at the OS level.  Today, the Linux community thinks of battery life as a communal resource that any app can drink from as much as it likes, and therefore they complain about apps which drink too much (see link "the app writers don't", above, about Beagle).  It would be interesting if the users could tell the operating system to stop giving power to Beagle until further notice.  The hard part is how the user can know what they want and communicate it to the OS.  It's a user interface issue.  Imagine a little animation showing different apps sucking juice out of your battery.  The user might notice that the Beagle app is //always// there, draining the battery, and they might choose to gesture to the OS to cut Beagle off and see what happens.

found [[a bug in TiddlyWiki|http://trac.tiddlywiki.org/ticket/373]]

* wrote to Denis Bonnenfant on [[tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-December/date.html]]
* tried and failed to get ~OpenSSL to build on Windows using gcc's -mno-cygwin option to produce native win32 executables; ~OpenSSL's build system is an awful mess of shell and perl scripts. It appears that the developers assumed that if you want to build it for Windows then (a) you have a Microsoft compiler and (b) you don't mind doing a whole bunch of custom steps manually; On the other hand, I have a Free Software compiler, and I would like for it to JUST WORK.  Wouldn't it be nice if the same command, such as "./config && make", would work on both unix //and// Windows?  Argh.  pyOpenSSL's build system is an awful mess of Python scripts, and the rest of my criticism of ~OpenSSL's build system applies equally well to pyOpenSSL's build system.  I was amused to notice that the pyOpenSSL build script has a hardcoded assumption that if you are building pyOpenSSL on Macintosh that you've already installed the //fink// version of ~OpenSSL!  Great.
* complained bitterly about ~OpenSSL and pyOpenSSL and beg someone to build binary eggs for Windows [[launchpad #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]
* contribute [[small patch to pyOpenSSL|http://bazaar.launchpad.net/~zooko/pyopenssl/buildbinaries/revision/81]]
* update [[setuptools_darcs|http://allmydata.org/trac/setuptools_darcs/timeline]]
* Ugh.  I'm sick.  'Tis the season; everyone else I know has been sick recently.  Oh well.
* wrote [[a letter to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-December/000924.html]] about caps-in-~URLs

* spent most of yesterday trying to build ~OpenSSL and pyOpenSSL for Windows using mingw
* read about [[the TIB3 hash function|http://ehash.iaik.tugraz.at/wiki/TIB3]]; I like it because it is a traditional, conservative design, because they explain their motivations in detail in the document, and because it seems to have good performance -- around 8 to 16 cycles per byte on either 64-bit //or// 32-bit ~CPUs (see my earlier note [[SHA-3 benchmarks and the importance of 32-bit CPUs]]).
* listened to [[this week's econtalk|http://www.econtalk.org/archives/2008/12/rauchway_on_the.html]] -- "Rauchway on the Great Depresson and the New Deal"
* posted to openssl-dev about my attempt to compile ~OpenSSL for Windows using mingw
* did a lot more work on ~ValidatedUEB

* 5:00 AM: updated the tickets about packaging pywin32 [[sourceforge #1799934|https://sourceforge.net/tracker2/?func=detail&aid=1799934&group_id=78018&atid=551954]], [[setuptools #18|http://bugs.python.org/setuptools/issue18]], [[twisted #3238|http://twistedmatrix.com/trac/ticket/3238]]
* my [[letter to openssl-dev|http://www.mail-archive.com/openssl-dev@openssl.org/msg24959.html]] finally went through
* posted to tahoe-dev about [[managing port numbers|http://allmydata.org/pipermail/tahoe-dev/2008-December/000928.html]]
* got almost all the tests passing with the new ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy ; worried about how much longer before Repairer is actually done
* saw this [[cool 5-minute video|http://project.haskell.org/camp/unique]] showing camp (a darcs variant) and git side-by-side with graphical patch trees; Nice!
* If you haven't seen this note of mine before, and you are interested in revision control tools, read this: [[bad merge|https://zooko.com/badmerge/concrete-good-semantics.html]]

* tests finally pass with the new ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy!
* here's something I want to be able to find/remember later, so I'm mentioning it on my klog: [[JACK Timemachine|http://plugin.org.uk/timemachine]]
* tried to get pyOpenSSL packaged for setuptools for Windows ([[tahoe #456|http://allmydata.org/trac/tahoe/ticket/456]] == [[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]])

Beautiful, soft, dry white snow has been drifting down all morning.  The boys and Amber happily shoveled the driveway and sidewalk.  Irby asked if he could please take the snow shovel with him and shovel the sidewalk all the way to school, but I explained that we would be late in that case, and maybe we could try it another day when we left early.

I updated the list of [[issue tickets]] issues in open source projects that I'm contributing to in one way or another.  [[issue tickets]] is always visible in my side bar over there """---->""".

(at the top of the right-hand-side-bar)

Here is my paper on Tahoe: [[lafs.pdf|http://testgrid.allmydata.org:3567/file/URI%3ACHK%3Alqigipzuthqasj6x2f2tsyrwsu%3Akesgqz3z3prnyuzli7hl5vpigbpipjmzzyhtyngvak2wet55rspq%3A3%3A10%3A275101/@@named=/lafs.pdf]].

There is a thick blanket of soft snow lying around the house.  Amber just brought me fresh-baked gingerbread cookies, milk, and coffee.  Elliot has a friend over for a play-date, and the play-date involved baking cookies.

* I wrote [[a letter to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2008-December/010593.html]].  I'm concerned that the fact that Tahoe behaves differently when you pass {{{?t=info}}} (and [[a few other query args|http://allmydata.org/trac/tahoe/browser/docs/frontends/webapi.txt]], when apache just ignored the query arg might inhibit Tahoe being used as a generic web host.
* ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy pass all tests!  And code coverage is excellent and Brian reviewed the code and I changed most of the things he mentioned.
* I finally got ~OpenSSL and pyOpenSSL working on Windows, and tested out the new setup/build/install process on Windows.  It is almost there -- I just have to decide how to make the tahoe executable plus PYTHONPATH available.

* [[happiness is contagious|http://www.latimes.com/features/health/la-sci-happy5-2008dec05,0,5449915.story]]; This news article contains many interesting assertions.
* We've finally got pyOpenSSL binary eggs for Windows.  Thanks, Chris Galvan!
* worked on the new packaging system for Tahoe, with Chris Galvan's help
* realized that my darcs executable was very slow due to a bug in ~GHC-6.8.2, and tried (with mixed success) to compile ~GHC-6.10.1 on various machines
* learned more about how {{{easy_install}}} parses web pages looking for package files
* fixed [[tahoe #553|http://allmydata.org/trac/tahoe/ticket/553]]: "More Info" link should point to a file/dir, not a dir+childname
* learned about [[transclusion in TiddlyWiki|http://www.tiddlywiki.org/wiki/Including_External_Content]] -- hopefully ~TiddlyWiki on Tahoe with transclusion will make the ultimate decentralized, shared wiki
* finally committed [[refactor handling of URI Extension Block and crypttext hash tree|http://allmydata.org/trac/tahoe/changeset/20081205141754-92b7f-f3ee4370feeab166962456792d95fdca9bb2cfab]]

The next step for repairer: integrate refactored URI Extension Block with my new checker code.

* I updated my patch to Dungeon Crawl to make a better god Xom.  To play it, telnet to {{{crawl.develz.org 345}}} and play the current trunk version.

I updated a few [[issue tickets]]:

[[buildbot #236|http://buildbot.net/trac/ticket/236]]: show elapsed time for steps -- @@fixed@@
[[buildbot #395|http://buildbot.net/trac/ticket/395]]: when i change the vcs executable, buildslave stops being able to invoke it until I restart buildslave
[[buildbot #396|http://buildbot.net/trac/ticket/396]]: Older builds
[[buildbot #252|http://buildbot.net/trac/ticket/252]]: side-effecty operations (Force Builder) should be ~POSTs

Today is Game Day at Chez O'Whielacronx.

I [[complained about Python 3.0|http://lwn.net/Articles/310071]] on lwn.net.

[[tahoe #555|http://allmydata.org/trac/tahoe/ticket/555]]: tahoe .deb cannot be installed on hardy: simplejson dependency is too new
[[tahoe #530|http://allmydata.org/trac/tahoe/ticket/530]]: use setuptools's """--multi-version""" mode
[[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue

FND helped me with CSS for this klog.

* using my newly gained knowledge of Tiddly CSS thanks to FND, I made the "side stuff" tiddler at the top of the sidebar prettier and more noticeable
* [[tahoe hacking|http://allmydata.org/trac/tahoe/timeline?from=2008-12-08&daysback=1&ticket=on&ticket_details=on&changeset=on&milestone=on&wiki=on&update=Update]]
** loosened requirements on {{{simplejson}}} and {{{setuptools}}} to ease packaging for Ubuntu gutsy and hardy (this results in a test failure on dapper, but nobody understands why)
** more tidying up of download code in preparation for merging new repairer
** merged new repairer and tested it, but it doesn't pass all tests so it isn't committed to trunk yet
** worked with Brian on refactoring and optimizing download code -- fetch only the needed parts of the Merkle Trees

Allmydata.com has an iPhone product now!  [[the allmydata.com blog|http://www.allmydata.com/blog]]

* I submitted a patch to Twisted: [[twisted #3568|http://twistedmatrix.com/trac/ticket/3568]]: ERROR from conch test when pycrypto is not installed
* I've found a major performance issue in darcs -- sometimes it tries repeatedly to establish a connection to a server, and it waits to see if the server answers, so if that server isn't answering then it takes hours to do anything.  [[darcs #1153|http://bugs.darcs.net/issue1153]]: darcs waits to hear back from servers unnecessarily
* I'm working on a patch for Twisted to avoid {{{repr}}}'ing strings in {{{Failure}}}s: [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory
* --I'm working on a bug report for ghc-6.10.1, which doesn't build out of the box on my GNU/~OpenSolaris server.-- (Forget it: couldn't reproduce it.)
* Uploaded videos encoded in [[the dirac codec|http://diracvideo.org]] to [[testgrid-shared-directory/video/dirac codec -- use VLC to play|http://testgrid.allmydata.org:3567/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/video/dirac codec -- use VLC to play]]
* wrote to [[Kevin Chalker|http://kc-computers.com]] asking how many SATA connectors per dollar he could build
* [[setuptools #17|http://bugs.python.org/setuptools/issue17]]: easy_install will install a package that is already there; This issue should probably be renamed in light of the fact that it seems to cause a worse failure nowadays with the proposed Debian packages for {{{foolscap}}} and {{{tahoe-lafs}}}.

* posted to the NIST ~SHA-3 forum quoting [[SHA-3 benchmarks and the importance of 32-bit CPUs]]
* started tracking this issue: [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]]: ~SiteUrl as current document location
* chatted with FND on the ~TiddlyWiki IRC channel:
** about [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]] ({{{SiteUrl}}} as current document location), which breaks the tiddly_on_tahoe RSS feed in its default setting; fixed the RSS feed of this klog per Peter Secor's bug report.
** about the cryptographic-capabilities-in-~URLs security model (I referenced [[Nathan Wilcox's page on the "Hack Tahoe!" hall of fame|http://hacktahoe.org/nathan_wilcox.html]] and [[this post on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2008-December/011844.html]])
** about ~JavaScript -- he taught me how to use Firebug and I learned some ~JavaScript and fixed [[tiddly_on_tahoe #2|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]: don't offer the option to save changes when you are viewing read-only -- @@fixed!@@
* I chatted quite a bit with Kragen Sitaker.  One topic was how [[git|http://git.or.cz]] has wide acceptance and is widely used as a substrate to build new things (such as [[github|http://github.com]], which is a social networking site built on a decentralized revision control tool), and how that came about.  I am hoping to apply some of those lessons to Tahoe.

Amber updated [[her home page|http://www.cs.toronto.edu/~amber]] for the first time since, I think, 2003.

Today I added a bunch of items to my tiddler named [[things to read]]:
* IEEE Spectrum: [[A Fairer, Faster Internet Protocol|http://spectrum.ieee.org/dec08/7027]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/04]])
* Ben Laurie, Abe Singer: [[Choose the Red Pill and the Blue Pill|http://www.links.org/files/nspw36.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/08]])
* Ben Laurie, Eric Sachs: [[Usability of Stronger Authentication Options|http://sites.google.com/site/oauthgoog/UXFedLogin/strongauth]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/03]])
* Jeff Bonwick (inventor of ZFS) on timeouts (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/11/29]])
* [[this week's LWN|http://lwn.net/Articles/309663]]

* I'm going to [[Boulder Linux Users Group|http://lug.boulder.co.us]] tonight.
* added ticket: [[tiddly_on_tahoe #3|http://allmydata.org/trac/tiddly_on_tahoe/ticket/3]] (offer a read-only cap to the user)
* finished reading and removed from my [[things to read]]: [[this week's LWN|http://lwn.net/Articles/309663]] (featuring an interview with Vernor Vinge); The "interview" with Vernor Vinge was only mildly interesting -- it sounded like it was excerpts from a much more interesting conversation or speech.
* added [[pycryptopp #12|http://allmydata.org/trac/pycryptopp/ticket/12]] (automatic wrappers for all of Crypto++) to [[issue tickets]]
* read these slides by Josh Berkus on [[Ten Ways To Destroy Your Open-Source Community|http://www.powerpostgresql.com/download/TFCKUpload/25.pdf]] and a few follow-up comments about it: [[Zack Urlocker|http://weblog.infoworld.com/openresource/archives/2008/05/josh_berkus_on.html]], [[Savio Rodrigues|http://weblog.infoworld.com/openresource/archives/2008/05/questioning_jos.html]], [[Josh Berkus again|http://it.toolbox.com/blogs/database-soup/community-destroyers-24309]]
* Here are some more [[things to read]] from Wes Felter's blog.  Does he really read all of these?  He must be a fast reader.
** Greenan, Long, Miller, Schwarz, Wylie: [[A Spin-Up Saved is Energy Earned: Achieving Power-Efficient, Erasure-Coded Storage|http://www.usenix.org/events/hotdep08/tech/full_papers/greenan/greenan_html]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
** Andreas Merkel, Frank Bellosa: [[Memory-aware Scheduling for Energy Efficiency on Multicore Processors|http://www.usenix.org/events/hotpower08/tech/full_papers/merkel/merkel_html]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
----
Yesterday I suggested to a group of eminent computer security researchers from major research institutions that instead of asking the U.S. Government for money to build a maze inside our farmhouse to protect us from the zombie hordes, they should ask for money to build a forcefield.
----
Here's a funny picture I found via Sameer's blog:
[img[Big Three Advertisement|/file/URI%3ACHK%3Av4kdjf3r7ekuwlcixqcrxxxnde%3Awz6chffifbpggdgsxgytgusgvfipedazpe7v6uopbwk4b5zs4kua%3A3%3A10%3A77968/@@named=/bigthree.jpg]] 

* read about [[CubeHash|http://cubehash.cr.yp.to]]
I like it!  It's very simple to understand, has obvious tuning knobs, and offers very good performance (depending on the next few years of cryptanalysis).  The only thing that I don't like about it is that there is no provision for having a smaller state.  Contrast with [[EnRUPT|http://enrupt.com]], which is specified to use a smaller state for smaller outputs, for example the 192-bit output size that I would like to use for tahoe-lafs would use a mere 384-bit state, where for 512-bit output, it would specify to use 1024 bits of state (just like ~CubeHash does).  Requiring fewer bytes of RAM for state can make the algorithm fit into more uses.  My friend Sebastian was skeptical of my claim that 128 bytes of RAM might turn out to be too much to ask for some applications, until I said "The fewer bytes of RAM we need, then //the smaller the motes of dust// that we can run ~SHA-3 on.".  That convinced him.
* finished "Scott Contini, Ron Steinfeld, Josef Pieprzyk, Krystian Matusiewicz: [[A critical look at cryptographic hash function literature (2007)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]]" with Amber and moved it from [[things to read]] to [[things read]]
It was a good survey and correctly pointed out some theoretical inconsistencies. The most novel suggestion for improvement was to evaluate the security of a hash function in terms of some application of that hash function.  What would such-and-such an application require from its hash function for such-and-such an application to be secure?  It's an interesting idea.  In the context of public-key cryptography secure hash functions are often called upon to do jobs that are easier than the job of being a collision-resistant, pre-image resistant, and second-pre-image resistant.  For example, ~ECIES-ISO needs a ~Key-Derivation Function, and what it needs from that function is merely that if compute the function without letting your enemy know what inputs you are feeding to it, then it doesn't produce outputs that your enemy can guess.  (Oh, and also that if you call it twice with two diversifiers to get two different keys, that it doesn't give you a pair of keys such that if you use one of them for a Message Authentication Code and the other one for a Cipher, that this doesn't somehow ruin the security of your Message Authentication Code and/or your Cipher.  If it were even //possible// for the ~Key-Derivation Function to do this then this would mean that your MAC and/or Cipher were badly flawed, of course.)

What does tahoe-lafs need from its secure hash functions?  Well, we use them in lots of ways, including ~Key-Derivation, just like ~ECIES-ISO does.  One way that tahoe-lafs uses a secure hash function is as an //identifier// of a file.  In that use the traditional requirements of collision-resistance, pre-image resistance and second-pre-image resistance seem just right.  Except for the caveat, as I [[mentioned earlier|today's crypto education]] that what we //really// care about is //confidentiality// which seems to diverge slightly from what cryptographers currently understand as pre-image resistance.  My notion of //confidentiality// of a secure hash function is that I don't want you to be able to find //my pre-image// -- the one I started with -- after I tell you the image.  I don't care if you can pick a random image and then find its pre-image, and I don't care if you can find a //different// pre-image for my image!  (If you do so, then you and I have collectively violated collision resistance, but you haven't violated my confidentiality.)

* the December edition of Bruce Schneier's newsletter [[CryptoGram|http://www.schneier.com/crypto-gram-0812.html]] is out; It seems to be chock full of provocative and interesting observations.  {{{CryptoGram}}} doesn't contain any text that hasn't previously been published on Bruce Schneier's daily blog or in magazine articles, and yet I feel like I get much better value for my time by reading the monthly edition than by reading the daily edition.  I suspect that same improvement could be applied to many sources of news and views.  Nicholas Nassim Taleb mentioned something similar in [[The Black Swan|http://www.amazon.com/exec/obidos/ASIN/1400063515]].
* the ~TahoePlugin for ~TiddlyWiki is almost finished
* I submitted [[The Transitive Grace Period Public Licence, v1.0|https://zooko.com/repos/pyutil/pyutil/COPYING.TGPPL.html]] to the [[Open Source Initiative for approval|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:462:200812:ofpndmgcgmbhbmimpkpe]].
* Okay! {{{tiddly_on_tahoe}}} is ready for wider use!  The way to get started is to follow [[the instructions to create a tiddly_on_tahoe instance|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:7h7syiurogz5erc2au74tjwguu:h7bdxvjtvidlkcdbld3j2d5sbgyzsbqs7wdnu6yznqrejzssc5za/wiki.html]].

* buying/making Christmas presents
* chatting with Chris Galvan about tahoe-lafs packaging

TODO:
* enter Eric Shulman's bug reports about {{{tiddly_on_tahoe}}} into [[http://allmydata.org/trac/tiddly_on_tahoe|http://allmydata.org/trac/tiddly_on_tahoe]] or get him to do so

NEXT:
* announce {{{tiddly_on_tahoe}}} @@finished@@
* work on Repairer
** next step: commit the logging refactoring patch that I was using to debug the new Checker

* created [[tiddly_on_tahoe #7|http://allmydata.org/trac/tiddly_on_tahoe/ticket/7]] (wrong error message when server is unreachable) and added it to [[issue tickets]]
* This morning I thought about making a challenge problem for the ~SHA-3 contest to illustrate [[my ideas about target-pre-image-resistance|today's crypto education]].  I didn't quite figure out how to illustrate it.  Maybe at some point (Christmas vacation?) I'll study [[the LANE hash function|http://www.cosic.esat.kuleuven.be/lane]] more.
* 10:30 Finally finished with morning take-the-children-to-school chores and starting the day's Repairer work.
* Yay!  There's somebody else besides me who thinks that {{{tiddly_on_tahoe}}} is [["completely frickin' brilliant"|http://www.eros-os.org/pipermail/cap-talk/2008-December/011878.html]].
* Today I've been spending a lot of time on the phone with credit card companies persuading them to lower my interest rates.
* added [[tahoe #558|http://allmydata.org/trac/tahoe/ticket/558]] (kpreid says that the -SUMO tarballs don't exist) to [[issue tickets]]

* "How the Barack Obama tapped into a powerful and only [[recently studied human emotion called 'elevation'|http://www.slate.com/id/2205150/pagenum/all/#p2]]." -- from [[Paul Hsieh|http://geekpress.com]]
* been hacking on Verifier/Repairer so far today

* Checker Verifier Repairer Checker Verifier Repairer Checker Verifier Repairer

* added [[twisted 3586|http://twistedmatrix.com/trac/ticket/3586]] (I want to install twisted without a c compiler) to [[issue tickets]]
* working on Download/Checker/Verifier/Repairer
* my brother Josh is here!  Hooray!  :-)

* Christmas party was a big success
* hacking on Downloader/Checker/Verifier/Repairer ; I love how thorough our tests are.

We drove to my mom's house to get there ahead of [[the storm|http://www.wunderground.com/wundermap/?lat=35.19670&lon=-101.84660&zoom=4&type=hyb&units=english&rad=1&rad.num=1&rad.spd=25&rad.opa=70&rad.stm=0&wxsn=1&wxsn.mode=tw&svr=1&svr.opa=70&cams=0&sat=0&riv=0&mm=0&hur=0&fire=0&tor=0&ndfd=0&pix=0]].

I hacked on Downloader/Checker/Verifier/Repairer in the car until I accidentally fell asleep at about 3 AM.  My brother Josh helped me debug.  I spent many hours trying to figure out where in the code was something handling all kinds of Failure and silently dropping it.  (This is the Twisted Python equivalent of catching all kinds of exception and then silently dropping it.)  It turned out to be in the stub Verifier class which is just a place-holder.  This is a class which I have already rewritten but I wasn't using my new version -- I was testing on trunk because I'm trying to test and commit each of my changes in a modular way.  Anyway, it was a frustrating expenditure of hours trying to figure out why exceptions/Failures were vanishing, but at least I got more familiar with the code and the tools.  Also I got to train Josh up on the details of Twisted failure handling.

Note to Python and Twisted programmers: please be very careful about catching all kinds of exception with an except: block and likewise about handling all kinds of Failure with an errback.

* This afternoon I wrote [[a short letter|will SHA-3 replace the current standard secure hash algorithm -- MD5?]] to the cryptographers designing ~SHA-3.
* Added [[buildbot #407|http://buildbot.net/trac/ticket/407]] ({{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better @@patch submitted@@) to [[issue tickets]]
* fixed the tahoe-lafs trac so that drewp and nejucomo can open tickets

Now Terrell can't log into the trac.  Grr.

I'm stumped about unicode handling in the tahoe cli.  I posted [[a plea for help|http://allmydata.org/pipermail/tahoe-dev/2008-December/000948.html]].

It is the day after Christmas.

 * [[The science of shopping|http://www.economist.com/science/displayStory.cfm?story_id=12792420&source=hptextfeature]] by The Economist; Okay people, now is the time to talk about the rules for machines that manipulate the brains of consumers.  Now, while it is still mostly theoretical.
 * http://events.ccc.de/congress/2008/wiki/Distributed_Storage_Grid
 * added [[pyopenssl #311600|https://bugs.launchpad.net/pyopenssl/+bug/311600]] (please update http://pypi.python.org/simple/pyOpenSSL) to [[issue tickets]]
 * added James Hamilton -- [[The Cost of Bulk Cold Storage|http://perspectives.mvdirona.com/2008/12/22/TheCostOfBulkColdStorage.aspx]] as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/27]] to [[things to read]]
 * Downloader/Checker/Verifier/Repairer; Wow, we really have some good test coverage here.

* http://events.ccc.de/congress/2008/Fahrplan/events/2875.en.html / [[Tor Bjørstad's slides|http://rosetta.nwo.no/~tor/slides.pdf]]
* http://events.ccc.de/congress/2008/Fahrplan/events/3023.en.html
* added [[foolscap #105|http://foolscap.lothar.com/trac/ticket/105]] //(make it easy to distinguish server-side failures/exceptions from client-side)// to [[issue tickets]]

Today we drive home to Boulder from my mom's farm house in New Mexico.  I stayed up late hacking on Downloader/Checker/Verifier/Repairer and updating release docs.  I'm going to be sleepy!  Maybe I can borrow a pillow from my mom...

* Yep, that CCC presentation was a doozy all right.
* Downloader/Storage/Checker/Verifier/Repairer
* Happy New Year, folks!

Happy New Year!

[[Help Zooko Choose Books]]

* added [[darcs #1298|http://bugs.darcs.net/issue1298]] //darcs failed: Malformed patch bundle: '{' is not 'Context:'// to [[issue tickets]]

It's the 3rd day of the New Year here at Zooko World Headquarters, and it is one day since we posted our literary invitation, and so far nobody has written in to [[Help Zooko Choose Books]].  Act now, during your moment of opportunity!

It occurs to us that many readers might find themselves devoid of opinions about that particular slate of books, and so we hereby open up the voting to any book that you recommend that we can get from amazon.com using our Christmas gift certificate.

* I created a new tiddler: [[collection of failures]].

* {{{<kpreid> guards with authority is a hazardous thing...}}}

* added [[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]] //python-foolscap: should advertise [secure_connections] feature to setuptools// to [[issue tickets]]

* added [[foolscap #108|http://foolscap.lothar.com/trac/ticket/108]] //set base to "." if not running from source (so {{{flogtool}}} works on Windows)// to [[issue tickets]]
* posted [[a note about the "Hack Tahoe!" contest|http://www.eros-os.org/pipermail/cap-talk/2009-January/011931.html]] in the attempt to contribute to [[the W3C Technical Architecture Group discussion|http://www.w3.org/2001/tag/2008/12/10-minutes]]
* added [[darcs #1303|http://bugs.darcs.net/issue1303]] //proposal: make "darcs changes" interactive by default// to [[issue tickets]]
* added [[ubuntu #314468|https://bugs.launchpad.net/hardy-backports/+bug/314468]] //Please backport setuptools-0.6c9 from Intrepid// to [[issue tickets]]
* responded to Brian's patch review [[on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/thread.html#962]]

* added [[darcs #26|http://bugs.darcs.net/issue26]] //Darcs needs real MIME parsing, fails with Mail.app, Courier// to [[issue tickets]]
* finally got tests passing for download-with-no-decryption; Now hooking together download-with-no-decryption and upload-with-no-encryption should result in Repairer.

Oh boy!  [[CodeCon|http://codecon.org]] is back!  I hope I can arrange to be in San Francisco that week.

Here's an experiment in using {{{tiddly_on_tahoe}}} to collaborate with my brother, Josh: [[biowiki|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:5ktwwvxbracbwbdrctmalqhu6e:i6d7l5ylzz4dx3zzxmrtqq7lxzhauz63ufhxnn7ckm2e7squhk4a/biowiki.html]].  Other are invited.

The topic that I want to share notes with him on is the prospect that low-carb dieting makes you age slower, and of course any related topics that attract our attention along the way.

* added //[[Martin C. Atkins|http://www.mca-ltd.com/martin/]] [[An Introduction to Ten15 - A personal retrospective.|http://www.mca-ltd.com/martin/Ten15/introduction.html]]// to [[things to read]]
* Repairer works!  Hooray!  There are a few more tests to fix or to mark TODO, and I'll write a commit message and push it into trunk tomorrow.

* I read [[caja-spec-2008-06-07|http://code.google.com/p/google-caja/downloads/detail?name=caja-spec-2008-06-07.pdf]].  Oh boy -- I'm excited about Caja!  I especially like the idea of Cajita: a nice clean, securable dynamic language which is a subset of ~JavaScript, so it is fully compatible with all of our existing ~JavaScript source code, tools, deployments, and programmer-brains.
* On the subject of [[Help Zooko Choose Books]], [[Myers Carpenter|http://icepick.info]] wrote in to say that he's reading [[Programming Erlang: Software For a Concurrent World|http://amazon.com/exec/obidos/ASIN/193435600X]], which he finds more interesting than Haskell because of the distributed, fault-tolerant, clustering, database features that ship with it.

* started reading [[The Transparent Society|http://books.google.com/books?id=brjIK1dnoYgC&dq=the+transparent+society&printsec=frontcover&source=bn&hl=en&sa=X&oi=book_result&resnum=4&ct=result]] by David Brin
* added [[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]] //Select default reactor based on platform and available libraries// to [[issue tickets]]
* added [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]] //closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.// to [[issue tickets]]

* resumed discussion of [[the TGPPL|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] on [[the OSI licence-review list|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:478:200901:ofpndmgcgmbhbmimpkpe]]; The contributors to that list have expressed great reluctance to put the OSI stamp on the TGPPL, not because there is any question about whether it is an open source licence, but because they don't want more open source licences to come into existence.
* added [[darcs #1217|http://bugs.darcs.net/issue1217]] //darcs put fails with 'darcs failed:  Malformed patch bundle: '{' is not 'Context:' '// to [[issue tickets]]

* Last week I said [[darcs is not trying to understand your source code]] and that eventually someone would steal or reinvent the simple idea of darcs merge and implement it in another revision control tool such as git.  Self-fulfilling prophecy -- Miklos Vajna, who is one of the git hackers, saw my post and is [[investigating how to make git do the right thing|http://article.gmane.org/gmane.comp.version-control.git/105748]] on my "badmerge" example.
* Whoo!  Josh gave me an [[OpenMoko Neo FreeRunner|http://wiki.openmoko.org/wiki/Neo_FreeRunner]] for Christmas!  I have no idea what I'm going to do with it.
* [[The $300 Million Button|http://www.uie.com/articles/three_hund_million_button]] as linked from [[Wes Felter's blog|http://wmf.editthispage.com]]
* added [[foolscap #109|http://foolscap.lothar.com/trac/ticket/109]] //make a "flogtool" executable that works on Windows// to [[issue tickets]]
* I updated the tags on this klog.  If you click on the word "Tags" over on the right-hand side you can browse entries by topic.  Also, of course, you can use the search box to find entries by text.  Let me know if it useful!
* This morning I posted to the darcs-users mailing list about [[how you can store a darcs repository on Tahoe|http://lists.osuosl.org/pipermail/darcs-users/2009-January/017199.html]].  :-)

* added //Michal Rjaško: [[Properties of Cryptographic Hash Functions|http://eprint.iacr.org/2008/527]]// to [[things to read]]
* added //Abhishek Parakh, Subhash Kak: [[A Recursive Threshold Visual Cryptography Scheme|http://eprint.iacr.org/2008/535]]// to [[things to read]]; (A good feature of visual cryptography papers is that they always come with pictures!)
* Yesterday I read //Peter Gutmann, [[The Crypto Gardening Guide and Planting Tips|http://www.cs.auckland.ac.nz/~pgut001/pubs/crypto_guide.txt]]//.
* Today I read //James Hamilton, [[The Case For Low-Cost, Low-Power Servers|http://perspectives.mvdirona.com/2009/01/15/TheCaseForLowCostLowPowerServers.aspx]]// (via [[Wes Felter's blog|http://wmf.editthispage.com/2009/01/16]]), and posted a question about it to James Hamilton's blog.
* added //Yehuda Lindell [[Adaptively Secure Two-Party Computation with Erasures|http://eprint.iacr.org/2009/031]]// to [[things to read]]
* added //[[foolscap #111|http://foolscap.lothar.com/trac/ticket/111]] -- timestamps of incident files -- TZ indicator please// to [[issue tickets]]
* added //[[foolscap #112|http://foolscap.lothar.com/trac/ticket/112]] -- timestamps of incident files -- ~ISO-8601'ish// to [[issue tickets]]
* added //[[foolscap #113|http://foolscap.lothar.com/trac/ticket/113]] -- timestamps of incident files -- UTC// to [[issue tickets]]
* added //Kevin D. Bowers, Ari Juels, and Alina Oprea [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]]// to [[things to read]]

* packaged darcs-2.2.0 for Windows and posted [[a note|http://lists.osuosl.org/pipermail/darcs-users/2009-January/017272.html]]; The darcs-2.2.0 packages for Windows are hosted on the Tahoe test grid.
* We're going to have a picnic in Martin Park around 15:00 today!
* still reading //[[The Transparent Society|http://books.google.com/books?id=brjIK1dnoYgC&dq=the+transparent+society&printsec=frontcover&source=bn&hl=en&sa=X&oi=book_result&resnum=4&ct=result]] by David Brin//

* posted [[another request|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:479:200901:ofpndmgcgmbhbmimpkpe]] to the Open Source Initiative about the [[Transitive Grace Period Public Licence|https://zooko.com/tgppl.html]]; I hope to get the stamp of Open Sourciness for the TGPPL as a Valentine's Day present!
* Tim O'Reilly says [[Work On Stuff That Matters|http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-fir.html]]: "Money is like gas in the car -- you need to pay attention or you'll end up on the side of the road -- but a well-lived life is not a tour of gas stations!"
* posted to distutils-sig about [[how Brian really wants his PYTHONPATH back|http://mail.python.org/pipermail/distutils-sig/2009-January/010755.html]] (let's fork setuptools)
* added the following to [[issue tickets]]: 
** //[[tahoe #424|http://allmydata.org/trac/tahoe/ticket/424]]: stdeb: push to upstream//
** //[[tahoe #423|http://allmydata.org/trac/tahoe/ticket/423]]: stdeb: use stdeb on tahoe itself//
** //[[tahoe #422|http://allmydata.org/trac/tahoe/ticket/422]]: stdeb: run from buildslaves//
** //[[setuptools #57|http://bugs.python.org/setuptools/issue57]]: {{{develop}}} doesn't create {{{.pth}}} files and {{{site.py}}} if {{{--multi-version}}}//
** //[[darcs #1255|http://bugs.darcs.net/issue1255]]: darcs put tries to convert to darcs-2-format?// and retired //darcs #1217// from [[issue tickets]] to [[issue tickets closed]]
* organized [[issue tickets]] a bit by adding some hierarchical layout, since it is growing so big
* retired //[[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]]: python-foolscap: should advertise [secure_connections] feature to setuptools// to [[issue tickets closed]] -- Debian hacker Stephan Peijnik quickly fixed it in response to Brian Warner's request. 
* finally got [[the tahoe-lafs buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]] back to working order; Now it is testing that the instructions in [["Install Tahoe"|http://allmydata.org/source/tahoe/trunk/docs/install.html]] work on all supported platforms, as well as a few other useful things.
* worked with Zandr on getting a Windows machine that JP Calderone can use for //[[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: package pyOpenSSL for easy_install on Windows//
* added //Gaetan Leurent and Phong Q. Nguyen: [[How Risky is the Random-Oracle Model?|http://eprint.iacr.org/2008/441]]// to [[things I have read]]
* added //Ewan Fleischmann, Christian Forler, and Michael Gorski: [[Classification of the SHA-3 Candidates (2009-09-19 edition)|http://eprint.iacr.org/2008/511]]// to [[things to read]]

yesterday:
* offered to help with //[[pyflakes #2720|http://divmod.org/trac/ticket/2720]]: Release Pyflakes//
* help with //[[buildbot #407|http://buildbot.net/trac/ticket/407]]: {{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better// (by pointing out that {{{utf-8}}} already seems to work)

* Oh, [[this week's EconTalk|http://econtalk.org]] is with Eric S. Raymond.  That's interesting.
* This week's [[LWN|http://lwn.net]] is good, as always.

* [[A Performance Evaluation and Examination of Open-Source Erasure Coding Libraries For Storage|http://www.cs.utk.edu/~plank/plank/papers/FAST-2009.html]], by Jim Plank, Jianqiang Luo, Catherine Schuman, Lihao Xu, and Zooko ~Wilcox-O'Hearn will be presented at [[FAST-2009: 7th USENIX Conference on File and Storage Technologies|http://www.usenix.org/events/fast09]].
* added to [[issue tickets]]:
** //[[nevow #2713|http://divmod.org/trac/ticket/2713]]: setup.py installs tests, but not documentation//
** //[[nevow #2830|http://divmod.org/trac/ticket/2830]]: setup.py incorrectly declares twisted.plugins to be a package//

* Update on my request for the stamp of ~OSD-conformance on [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] from the Open Source Initiative: I don't know who exactly is on the OSI's Board of Directors, but so far Russell Nelson and Bruce Perens have posted to [[the license-review list|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:iis:498]] saying that they will not certify the TGPPL as conformant, and questioning my motives.  Neither of them have actually suggested that the TGPPL is not ~OSD-conformant.  Obviously the TGPPL is an open source licence, and neither Nelson nor Perens have suggested otherwise.  Presumably if the OSI officially rejects the licence then it will have to state the reason for rejection.
* added to [[things to read]]:
** //George Bray: [[Obesity: a failure of homeostasis because of hedonic rewards: response to the letter from Gary Taubes|http://testgrid.allmydata.com:3567/file/URI:CHK:q3cwozbit4be2jxmxfre65wzdq:h6rm26yuvcjttexlwhdi6lfcgqoxkordv52xxkwyhxphkytcfuva:3:10:116345/@@named=/Bray_Taubes_rebuttal_rebuttal_rebuttal.pdf]]// (thanks to Gary Taubes for bringing it to my attention as I requested in [[rebuttals to "Good Calories, Bad Calories"]])
** //Bryan Cantrill, and Jeff Bonwick: [[Real-World CONCURRENCY|http://mags.acm.org/queue/200809/?folio=16&CFID=19616687&CFTOKEN=90220359]]// (thanks to Kragen Sitaker for bringing it to my attention)
* I've been trying to use Konqueror from KDE 4.  So far it is performing badly and behaving badly, compared to Firefox-3.
* added to [[issue tickets]]:
** //[[konqueror #321636|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321636]]: kioslave crashes when logging into my issue tracker//
** //[[konqueror #321656|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321656]]: iso-8859-1 and/or utf-8 character not decoded properly//
* [[Decentralized Money]]

* fixed a few bugs in {{{allmydata.test.test_runner}}}, most of which only affected Windows: [[[3482]|http://allmydata.org/trac/tahoe/changeset/20090127044046-92b7f-7f2a4a94dd1e18f7397f5ff2900136e1d19ddcb4]], [[[3483]|http://allmydata.org/trac/tahoe/changeset/20090127203245-92b7f-d573b92533ffde9c4b22eefb0e9d4b8eae2cc213]], [[[3484]|http://allmydata.org/trac/tahoe/changeset/20090127203505-92b7f-ee4a6bb8114e1717b62c818cbf08f27a52021415]], [[[3485]|http://allmydata.org/trac/tahoe/changeset/20090127203717-92b7f-4e916910ac3838630fc4c15697371c0d8534af82]]
* created //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* uploaded Twisted binary eggs for Linux to [[the tahoe dependencies repository|http://testgrid.allmydata.org:3567/uri/URI%3ADIR2-RO%3Asnrfwfxatrci35zdgjnzxxx2ke%3Aunarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a]]
* uploaded ~CGalvan's py 2.4 pyOpenSSL egg to the tahoe dependencies repository
* uploaded zope.interface bdist_eggs for Linux to the tahoe dependencies repository
* got JP Calderone access to a Windows machine (mostly for //[[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: package pyOpenSSL for easy_install on Windows / please provide binaries//) (thanks to Zandr)
* applied Eric Kow's patches to [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] to build darcs with [[the Haskell cabal tool|http://www.haskell.org/cabal]] instead of with autoconf and GNU make
* pushed on the distutils ticket about proceeding after failing to build extension modules: //[[distutils #4706|http://bugs.python.org/issue4706]]: try to build a C module, but don't worry if it doesn't work// (added to [[issue tickets]])
* added to [[things to read]]:
** Daniel J. Bernstein, and Adam Langley [[Crit-bit Trees|http://www.imperialviolet.org/binary/critbit.pdf]]
** Stanford Encyclopedia of Philosophy: [[Bayes' Theorem|http://plato.stanford.edu/entries/bayes-theorem]]

Things That I Could Do (unordered)
* post about hosting eggs on tahoe
* write to Shawn Willden on tahoe-dev about backup designs
* write to Shawn Willden on tahoe-dev about random access read of immutable file contents
* move reactor selection based on sys.platform from tahoe's setup.py to setuptools_trial and push on the Twisted ticket
* fix test failure of share corruption on Windows
* fix that problem with shutdown of tahoe node (?) on Windows (unit test failure)
* upload Twisted binary eggs for more platforms
* build and upload zope.interface bdist_eggs for more platforms
* see about http://www.mozilla.org/projects/netlib/dirindexformat.html and if tahoe wui/wapi should serve it up
* request more binary eggs from zope.interface upstream
* report nejucomo's and dreid's egg boos to distutils-sig
* fix the build failure on hardy-py2.6 due to the {{{#!/usr/bin/env python}}} differing from the Python used to build (seek Chris Galvan's advice about how to fix this)
* fix all those {{{SUCCESS!?!}}} and {{{TODO}}} regarding handling of randomly corrupted shares
* fix //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* enable large_share test on all platforms other than OSX
* enable runner tests on Windows
* fix the deb builders

To Do Next:
* fix the deb builders
Things Done So Far Today:
* posted about [[the limits of reliability estimates|http://allmydata.org/pipermail/tahoe-dev/2009-January/001058.html]] to tahoe-dev
* persuaded Jack Lloyd to let me use [[his code|http://www.randombit.net/bitbashing/programming/forward_error_correction_using_simd.html]] in zfec
* [[fixed a bug|http://allmydata.org/trac/darcsver/changeset/42]] in [[darcsver|http://pypi.python.org/pypi/darcsver]] that Nathan discovered when trying to install tahoe with [[the install.html|http://allmydata.org/source/tahoe/trunk/docs/install.html]], [[fixed another bug|http://allmydata.org/trac/darcsver/changeset/44]] in darcsver, released darcsver-1.1.8, bundled darcsver-1.1.8 with tahoe
* observed that {{{python ./setup.py darcsver}}} takes 55 seconds on this one darcs repository I have; observed that it takes 15 seconds even after running {{{darcs optimize}}} on that repository; submitted a copy of that repository to the darcs folks for their new project of optimizing darcs with automated benchmarking of real-live repositories
* helped the darcs hackers get [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] working better
* [[wrote to Shawn Willden on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001054.html]] about random access read of immutable file contents
* [[reported nejucomo's and dreid's egg boos to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2009-January/010816.html]]
* opened //[[ubuntu p7zip #322481|https://bugs.launchpad.net/ubuntu/+source/p7zip/+bug/322481]]: took an order of magnitude longer than expected to compress//
* updated the docs and metadata of setuptools_darcs (see [[its timeline|http://allmydata.org/trac/setuptools_darcs/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]) and uploaded [[setuptools_darcs-1.2.3|http://pypi.python.org/pypi/setuptools_darcs]]
* updated the docs and metadata of darcsver (see [[its timeline|http://allmydata.org/trac/darcsver/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]) and uploaded [[darcsver-1.1.9|http://pypi.python.org/pypi/darcsver]]
* made setuptools_trial specify the poll reactor on linux or cygwin, updated its metadata (see its [[timeline|http://allmydata.org/trac/setuptools_trial/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]), and uploaded [[setuptools_trial-0.5|http://pypi.python.org/pypi/setuptools_trial]]
* poked //[[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]]: Select default reactor based on platform and available libraries//
* enabled {{{allmydata.test.test_storage.test_large_share}}} on all platforms

Things That Ought To Be Done for the Tahoe-1.3 release:
* fix test failure of share corruption on Windows
* fix that problem with shutdown of tahoe node (?) on Windows (unit test failure)
* fix the build failure on hardy-py2.6 due to the {{{#!/usr/bin/env python}}} differing from the Python used to build (seek Chris Galvan's advice about how to fix this)
* fix //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* put this list and a few other open tickets onto [[The Roadmap for tahoe-1.3|http://allmydata.org/trac/tahoe/roadmap]]
Things That I Could Do (unordered):
* post about hosting eggs on tahoe
* write to Shawn Willden on tahoe-dev about backup designs
* move reactor selection based on sys.platform from tahoe's setup.py to setuptools_trial and push on the Twisted ticket
* upload Twisted binary eggs for more platforms
* build and upload zope.interface bdist_eggs for more platforms
* see about http://www.mozilla.org/projects/netlib/dirindexformat.html and if tahoe wui/wapi should serve it up
* request more binary eggs from zope.interface upstream
* enable runner tests on Windows
* fix all those {{{SUCCESS!?!}}} and {{{TODO}}} regarding handling of randomly corrupted shares

Good morning.

* [[This week's econtalk|http://www.econtalk.org/archives/2009/01/roberts_and_han.html]] is excellent so far.  Russ Roberts is having a crisis of faith. Enlisting Robin Hanson as the patient confessor, he turns his practice of economic analysis to focus on... his practice of economic analysis!  I love this kind of thing.  I've listened to only half of it so far.  Will Russ reaffirm his faith in the free market?  Will Robin say something which devastates the belief systems of all listeners in a single vast wave of disillusionment?  Tune in to find out.
* [[This week's LWN|http://lwn.net/Articles/316190]] reports on Simon Phipp's speech about "the third wave of free software".
* added to [[issue tickets]]:
** //[[buildbot #266|http://buildbot.net/trac/ticket/266]]: I wish to tell my buildmaster: "restart yourself the next time you quiesce"//

What I'm Doing Right Now:
* Debugging [[use setuptools's --multi-version mode|http://allmydata.org/trac/tahoe/ticket/530]] / [[develop doesn't create .pth files and site.py if --multi-version|http://bugs.python.org/setuptools/issue57]]
Argh, I give up on debugging that.  We'll just require users to uninstall any old conflicting packages before building Tahoe.

I'm working with Nathan and Brian and ~FriAM on Tahoe security architecture vs. ~JavaScript.  I'm still waiting for Collin Jackson, or Nathan Wilcox, or ''someone'' to demonstrate how the current tahoe security architecture's mismatch with the current web browser security architecture leads to an exploitable problem.
I think the best long-term strategy is:
* tell the web browsers that two web pages loaded from tahoe are from ''different'' origins
* have the tahoe web server do Caja verification/rewriting on all pages that it serves
Note that this "two pages from same server are actually of different origin" situation is exactly the situation with mash-ups.  You want to be able to serve up two different ~JavaScript applets from two different authors, from the same server and running on the same page, while telling the browser that these two applets are from different origins.  At least, if you're a capability-security person, that's what you want.
* fixed [[the deb-builders|http://allmydata.org/buildbot/waterfall?show_events=false&branch=&builder=edgy&builder=feisty2.5&builder=etch&builder=gutsy&builder=hardy&builder=deb-edgy&builder=deb-feisty&builder=deb-etch&builder=deb-gutsy&reload=none]] yesterday
* added to [[things to read]]:
** Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]]

Wow!  [[Mark Miller|http://research.google.com/pubs/author35958.html]] said he liked my blog!

* wrote a note to cap-talk about [[why it is important to publish "ACLs don't"|http://www.eros-os.org/pipermail/cap-talk/2009-January/012073.html]] 
* wrote a note to distutils-sig about how [[stdeb can be used to produce real Debian packages|http://mail.python.org/pipermail/distutils-sig/2009-January/010912.html]]
* I've been building binary Python eggs of many Tahoe dependencies on many platforms and uploading them to [[this directory|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:snrfwfxatrci35zdgjnzxxx2ke:unarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a/index.html]].
* wrote a note to distutils-sig about [[hosting Tahoe's dependency packages on a Tahoe grid|http://mail.python.org/pipermail/distutils-sig/2009-January/010915.html]]
* Heh heh heh.  This morning google was warning me that "This site might harm your computer.".  It was warning me about that for ''every'' site that it suggested, for ''every'' query that I asked about.  Truer words were never spoken, Mr. Google!  Here's a screen shot of one of the queries.  I got the same results when I searched for "sex" and for "frisbees".  (Click the image below for the full screen shot.)
[img[This site may harm your computer. -- cropped|/file/URI%3ACHK%3Asx2qslmo2lrjnaslhlr4t6rk7e%3Ajl6frjdvmfs2kuq7pfwov3t4iqcbraz6ey5ul7lrxakzlh6jqnba%3A3%3A10%3A66528
/@@named=/This_site_may_harm_your_computer-cropped.png][/file/URI:CHK:kry2elgkh42hzpfwkh4b3mhksi:xdcewsphgz5f4js3cf4hszbie4kcz4uuds4djbnztel2wkzbscbq:3:10:171851/@@named=/This_site_may_harm_your_computer.png]]
(P.S.  Here is [[a screenshot|http://folk.ntnu.no/brasetvi/google_considered_harmful.png]] sent in by an alert reader who googled for google.  I should have thought of that.)

What I'm Working On Right Now:
* figure out why my Windows Tahoe node doesn't connect to all the testgrid blockservers
* make green the rest of [[the tahoe buildbots|http://allmydata.org/buildbot/waterfall]] (cygwin and Windows)
Then:
* ask everyone on tahoe-dev to test out the current trunk in preparation for the 1.3.0 release
* close the rest of these tickets for the Tahoe-1.3.0 release: [[the Tahoe Roadmap|http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.3.0]]
Then:
* tag tahoe-1.3.0, upload tarballs and .debs, send out a release announcement far and wide letting people know about the existence of tahoe-1.3.0
Done:
* greenified the edgy and dapper buildslaves
* added to [[issue tickets]]
** //[[foolscap #107|http://foolscap.lothar.com/trac/ticket/107]]: exceptions.~KeyError: "unable to find reference for name//

* Here is [[the RSS feed of this blog|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]].
* I went to [[Boulder Open Coffee|http://boulder.me]] this morning.  The host was a funny venture capitalist named [[Jason Mendelson|http://jasonmendelson.com]].  The back room of the coffee shop was crowded with tech industry people.  I think I flubbed my 10-second introduction, describing [[tahoe|http://allmydata.org]] as a "secure decentralized filesystem", which did not elicit any detectable interest.  Perhaps I should have said that there exists a "decentralized web app" (this blog).  Oh well -- maybe I'll have another 10-second chance someday.<br>I was later pleasantly surprised when the fellow on my right, Manisch, asked me a few good questions (about naming and availability), quickly understand the architecture, and described some interesting related work.
* I finally finished listening to [[EconTalk 2009-01-26|http://www.econtalk.org/archives/2009/01/roberts_and_han.html]].  It was fascinating.  I wonder if thinkers like Hanson and Roberts will turn their current ideas about how science is practiced into an empirical scientific endeavour instead of merely a fascinating philosophical rumination.
* I've renewed [[my plea|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:512:200902:ofpndmgcgmbhbmimpkpe]] for the Open Source Initiative to certify that [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] is open source.  The discussion is valuable, although personally painful for me -- I get physically shaky when reading or writing on [[the thread|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:sss:512:200902:ofpndmgcgmbhbmimpkpe#b]].  One way in which the discussion is valuable is that it vividly reminds me of the fact that the TGPPL needs a careful FAQ and other such supporting material.  (Thanks also to Wes Felter for reminding me of that.)
* added to [[issue tickets]]:
** //[[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff//
** //[[tahoe #605|http://allmydata.org/trac/tahoe/ticket/605]]: delayed connection on Windows//
** //[[tahoe #556|http://allmydata.org/trac/tahoe/ticket/556]]: prepend 'application-version' with the name of this particular application//
** //[[tahoe #217|http://allmydata.org/trac/tahoe/ticket/217]]: ~DSA-based mutable files -- small ~URLs, fast file creation//
* Eric Shulman pointed out that all I needed to say to get the attention of the business people was "Tahoe is a cloud storage technology.".  Now I'm armed for next time I find myself in that position.

* added a comment to //[[setuptools #20|http://bugs.python.org/setuptools/issue20]]: package required at build time seems to be not fully present at install time?// explaining how the current attempt to fix it leads to a different problem for tahoe
Here's a joke, from [[Ian Grigg|http://financialcryptography.com]]:
<<<
Teacher: If there are ten sheep in the pen, and one sheep jumps out, then how many sheep are left in the pen?
Girl: None.
Teacher (sadly): You don't understand arithmetic.
Girl: No, you don't understand sheep!
<<<

Good morning!  The setup/packaging/installing problem on tahoe trunk appear to be fixed, judging by [[the mostly-green buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]].  In order to do this I had to create a toothpick of [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]].  A toothpick is like a fork, but smaller and best used for only one user.  The toothpick is being maintained under the whimsical name {{{zetuptoolz}}} on [[the allmydata trac|http://allmydata.org/trac/zetuptoolz]].  As soon as tahoe-1.3.0 is out, I'll try to merge my toothpick with upstream setuptools or one of its variants or alternatives.

I'm going to [[this talk by Peter Braam|http://www.cs.colorado.edu/events/colloquia/2008-2009/braam.html]] this afternoon.  So I'm reading [[the wikipedia about about Lustre|http://en.wikipedia.org/wiki/Lustre_(file_system)]].

Now that the OSI understand the intent of [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] as being a //transitive public licence// (i.e. like the GPL in this respect), [[they want to think about it more|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:518:200902:ofpndmgcgmbhbmimpkpe]].

add the following to the list of Things To Do After 1.3.0 release: write up the notes from the discussion with Brian about how to document standardizable explanation of tahoe:
* document 1: file formats: integrity, confidentiality, erasure-coding; immutable and mutable files (this would basically be my [[lafs.pdf|http://allmydata.org/~zooko/lafs.pdf]] with enough detail to facilitate compatible implementation)
* document 2: storage server protocol: how to connect to storage servers and upload/download files or parts thereof; foolscap protocol (extant), HTTP protocol (future work)
* document 3: peer selection ; This is the hard one.  We can document the "~TahoeTwo" system, which works okay for [[the allmydata.com use case|http://allmydata.org/trac/tahoe/wiki/UseCases]], but there are all sorts of use cases that are ill-served by the "~TahoeTwo" design -- more than a few hundred servers per grid, non-clique grids, secure decentralized introduction, different needs for reliability, availability, efficiency, etc. (some user wants Tahoe to store K+1 shares of each file in each of their several separate data centers, Shawn wants it to store exactly K shares of each file on his mom's computer, if that file is tagged as being a family photo), how to tell which grid or grids or set-of-servers a given cap can be found in.
* document 4: directories ; How to encode a set of caps, each annotated with metadata such as a name, timestamp, etc., in a file.  Traversal caps?
This project is very exciting because it could facilitate other people re-using tahoe, re-implementing it, or at least stealing its best ideas.  The walls of separation between these four documents are there to limit the damage from design mistakes, and by the same token to explicitly identify what each part requires of the other parts.
In particular I am eager to wall off the designs in document 3 from the others, as it is a field littered with the corpses of attractive designs.

Things to do for the tahoe-1.3.0 release; see also [[the Tahoe Roadmap|http://allmydata.org/trac/tahoe/roadmap]]:
* remove the cygwin buildbot from the list of supported platforms, move it to a new list of unsupported platforms, report this issue to twisted and cygwin via bug reports with minimal test cases ; //or// fix the cygwin buildslave.  (I contributed a small test script which shows the problem over on //** [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]]: closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.//.)
* remove Dan's ~ArchLinux buildbot from the list of supported platforms, //or// fix it.  (I sent Dan, Chris Galvan, and Brian mail about that behavior.)
* clean up the repairer tests -- probably comment many of them out since they are written to test a repairer that discriminates among servers on upload, probably add a verify run after the remaining ones so that it is judging more carefully the state of the shares after repair instead of just judging the repairer's report
* figure out why dreid and Andrzej Falout are both reporting bad performance from the prod grid
Things to do after the tahoe-1.3.0 release:
* //[[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail// (pushed out of the 1.3.0 milestone)
* write back on [[this thread on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001056.html]] and say "No, no, it really is almost as easy as Shawn originally thought.  Go, Shawn, go."
* figure out why setuptools/zetuptoolz is rebuilding things when {{{python ./setup.py test}}} after it just built them when {{{python ./setup.py build}}}
* pycryptopp improvements -- link against system libcryptopp.so for Debian and Fedora packagers, add new improved ECDSA, build out more buildbots, etc.
* accounting/garbage-collection/quotas/etc.
* write up all those documents described in [[2009-02-06]]
* write up my new idea for immutable crypto caps
* start backing up all my personal files with tahoe
* one zillion other things
Added to [[things to read]]:
* new IETF group [[on Massively Multiplayer Online interop|http://trac.tools.ietf.org/bof/trac/wiki/MmoxCharter]]
* Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]] and [[the ensuing conversation on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2009-January/012030.html]]
* Nate Foster, Benjamin C. Pierce, and Michael Greenberg, et al.: [[Harmony/Boomerang|http://www.seas.upenn.edu/~harmony]], A bidirectional programming language for ad-hoc, textual data

Thanks especially to Brian Warner, we got a lot closer to the tahoe-1.3.0 release today!  See [[the Roadmap|http://allmydata.org/trac/tahoe/roadmap]] and [[the Timeline|http://allmydata.org/trac/tahoe/timeline]] for details, and [[the tahoe-dev list|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]] for social context.

added //[[twisted #3649|http://twistedmatrix.com/trac/ticket/3649]]: more specific warning about plugin cache// to [[issue tickets]]

Today is the day that we'll close all [[open tickets for tahoe-1.3.0|http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.3.0]].  I'll also send out a plea to [[the tahoe-dev mailing list|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]] to test it and report bugs.  Hopefully tomorrow is the day that we tag it and send out the release announcement far and wide.

added to [[things to read]] a couple of things that Nathan just put on //his// "things to read":
* Ken Thompson: [[Reflections on Trusting Trust|http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf]]
* Helen J. Wang, Xiaofeng Fan, Jon Howell, and Collin Jackson: [[Protection and Communication Abstractions for Web Browsers in MashupOS|http://research.microsoft.com/en-us/um/people/helenw/papers/sosp07mashupos.pdf]]

* helped Nathan with his newly announced [[tahoewapi.js|http://allmydata.org/pipermail/tahoe-dev/2009-February/001158.html]]
* got some help from other people on packaging Tahoe for Fedora and Debian -- [[working around subtle C++ linking issues|http://allmydata.org/pipermail/tahoe-dev/2009-February/001159.html]]
* posted [[a beta-test of the tahoe-1.3.0 Release Announcement|http://allmydata.org/pipermail/tahoe-dev/2009-February/001151.html]] :-)
* posted about [[tahoe RelatedProjects|http://allmydata.org/pipermail/tahoe-dev/2009-February/001157.html]]
* added to [[issue tickets]]: //[[tahoe #615|http://allmydata.org/trac/tahoe/ticket/615]]: Can ~JavaScript loaded from Tahoe access all your content which is loaded from Tahoe?//
* made a new tiddler on this blog: [[things to do after the tahoe-1.3.0 release]]

Hello, world!  Tahoe-1.3.0 looks to be just about done.  I'm editing docs and release notes and changing the format of the "application version string" and running extra tests on repairer, but basically what is in trunk now is going to be released as Tahoe-1.3.0 today.  So hurry up and test trunk for me and report bugs!

added to [[issue tickets]]:
* //[[konqueror #184157|http://bugs.kde.org/show_bug.cgi?id=184157]]: crashed (I might have just hit the "back" button)//
* //[[tiddly_on_tahoe #9|http://allmydata.org/trac/tiddly_on_tahoe/ticket/9]]: can't save from Konqueror-4.2.0//
* //[[darcsver #2|http://allmydata.org/trac/darcsver/ticket/2]]: use "darcs query" to get count of patches faster//

* I updated [[Help Zooko Choose Books]] to reflect Myers's recommendation.  I'm increasingly interested in Erlang for similar reasons as Myers is -- I'm very interested in the "crash-only" design for high availability and correctness.  Also I'm told that Erlang is an iota away from being a capability-secure programming language.
* Okay [[Tahoe-1.3.0|http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=20090214000500-92b7f-26d07f519c69a3e086ee4599620f6d2c0c9b2fcd]] is out!  Now I'm looking at a huge backlog of tasks.  Last night I submitted a proposal to present Tahoe at [[CodeCon 2009|http://codecon.org]].  (I offered to suddenly and violently destroy a hard drive or two on stage.)  Many of the tasks on this huge backlog involve writing mail to [[tahoe-dev|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]].
* submitted a couple of patches to fix a couple of bugs in [[stdeb|http://github.com/astraw/stdeb/tree/master]]
* helped Nils Durner host his new {{{libtahoeclient_webapi}}} project on http://allmydata.org/trac
* had a nice chat at a local coffeeshop with Peter Braam

I got audio working on my linux workstation.  Hey, that was painless!
* working with François Deppierraz, ~Jan-Benedict Glaw, Shawn Willden and perhaps others on making tahoe handle the tangled mess that is character-set encoding (//[[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue//, added to [[issue tickets]])
* posted about how to automatically produce .deb packages from Python source code [[on distutils-sig|http://mail.python.org/pipermail/distutils-sig/2009-February/010998.html]]
* re-iterated my [[Request For Approval As Open-Source-Definition-Conformant|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:529:200902:ofpndmgcgmbhbmimpkpe]] for [[The Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]]
* //update!// and then I [[challenged the Open Source Initiative to sue me|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:531:200902:ofpndmgcgmbhbmimpkpe]]
* I submitted the Transitive Grace Period Public Licence to [[the debian-legal mailing list for review|http://lists.debian.org/debian-legal/2009/02/msg00047.html]].
added to [[issue tickets]]:
* //[[twisted #1956|http://twistedmatrix.com/trac/ticket/1956]]: Make a less sucky producer/consumer API//

[[The Open Source Initiative Stumbles]]

Tonight I'm going to see [[Children of Men at the International Film Series|http://internationalfilmseries.com/event_detail.php?event_id=9509]].  Meet me in the square with the giant buffalo at 21:00.

I want to make it possible to use the [[Brainpool ECC curves in Crypto++|http://groups.google.com/group/cryptopp-users/browse_thread/thread/b04fd594c5fc833]], but where are the test vectors?  I'm writing letters to Hal Finney and Niels Ferguson to ask if they have some ideas.

added to [[issue tickets]]
* //[[amarok #184834|https://bugs.kde.org/show_bug.cgi?id=184834]]: amarok crashes on "Quit"//

//(My mom, and my friend Nat and others, have pointed out that my blog is chock full of acronymy goodness and they can get only the vaguest idea of what I'm on about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.)//

I've decided to attend [[PyCon|http://us.pycon.org/2009/about/]] this year!  Whoo!  //''Mom'': that means a giant party in Chicago where we learn, write programs, and engage in "professional networking", such as by consuming intoxicants.//

[[Bespin+Tahoe]]

Irby and Elliot are really enjoying their [[OLPC XOs|http://en.wikipedia.org/wiki/OLPC_XO-1]], which a generous relative gave to them a year ago for Christmas.  They play [[Battle for Wesnoth|http://wesnoth.org]] quite a lot, but they also explore the more explicitly educational activities.  A couple of days ago I browsed the [[list of Activities|http://wiki.laptop.org/go/Activities/All]] and chose this one named [[Physics (Activity)|http://wiki.laptop.org/go/Physics_(activity)]].  It is great!  Irby has spent many hours now configuring different 2-D objects to fall and bump and bend and scrape each other.

Good morning, world!

A bunch of Free Software/Open Source hackers from around the world have volunteered their efforts to the Tahoe project.  Come join the fun!  http://allmydata.org .  The [[tahoe-dev mailing list|http://allmydata.org/pipermail/tahoe-dev/2009-February/date.html]] is the center.

(This offer does not apply if contributing to a Free/Open Source project to create a secure, decentralized storage grid doesn't sound like "fun" to you.)


things to do (I'm taking notes here so I don't forget any of them):
* //[[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff// == //[[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff// (next step: separate out my patches to embed C++ objects directly into Python objects and save it aside for later)
* review [[François Deppierraz's latest patch|http://allmydata.org/trac/tahoe/ticket/534]] to fix handling of non-ascii characters on all platforms
* respond to Nils Durner's wiki entry on [[building tahoe for Wndows|http://allmydata.org/trac/tahoe/wiki/WindowsBuild]]
* compare Nils Durner's summary of how [[GNUnet|http://gnunet.org]] finds its local state files on Windows with how tahoe currently does it (which was implemented by ~RobK)
* post my note about Jack Lloyd's attack on rsync
* fix Eugen Leitl's contributed buildslave so it builds Debian packages for Lenny/amd64
* figure out why yukyuk is getting signal 11 errors sometimes in test_backupdb
* fix buildbot breakage due to recent patch to improve the checker/repairer unit tests
* post about buildbots to tahoe-dev
* set up an introducer on nooxie for the nascent Community Grid
* review [[DarKnesS_WolF's patch|http://allmydata.org/trac/tahoe/ticket/638]] to build .deb's for Intrepid


//''Mom:'' Just read the part about people volunteering.//

Here's my reply to Jack Lloyd's attack on rsync: [[collision-resistance vs. second-preimage-resistance|http://allmydata.org/pipermail/tahoe-dev/2009-February/001309.html]].

//''Mom:'' I'm trying to figure out how to make ~URLs which are nice and short but which are secure in the sense that nobody can make you get a different page than the page intended by the person who gave you the URL.  Other people might be able to use the underlying cryptographic principles to make other things.  Is that a good enough translation?//

Hello folks!  Sorry for the blog outage.  I got [[laid off|http://allmydata.org/pipermail/tahoe-dev/2009-March/001461.html]], Amber is pregnant, and the Tahoe test grid that this blog is hosted on [[suffered a bug|http://allmydata.org/trac/tahoe/ticket/651]].  But I'm back!  Hope you missed me!  Don't forget to subcribe to [[my RSS feed|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]].

All Hackers who are going to be near Boulder, Colorado at the beginning of April, please write to [[zooko@zooko.com|mailto:zooko@zooko.com]]!

Also, I'm going to [[PyCon 2009|http://us.pycon.org/2009/about/]]!  Whoo!

[[PyCon|http://us.pycon.org/2009/about]] was great fun.  It was a pleasure to find out that [[Tahoe|http://allmydata.org]] has become a widely known project within the Python community.  Most people have heard of it, and many people find it interesting.

NEWSFLASH!  Tahoe is eligible for Google Summer of Code sponsorship, thanks to the generous Python community in the form of the [[Python Software Foundation|http://www.python.org/psf]] umbrella organization!  If you are a student, or you know a student, who wants to hack on Tahoe under google's sponsorship this summer, then [[read this now|http://allmydata.org/pipermail/tahoe-dev/2009-April/001516.html]]!  :-)

(P.S.  This is not Fake News as in the Internet tradition of April first.  It is true.  And you have less than 48 hours to sign up, so get on with it.)

//~CodeCon!  Whoo!//  I'm going to [[CodeCon|http://codecon.org]]!  I'll be presenting [[Tahoe|http://allmydata.org]] there.

(//''Mom:'' Major hacker party in San Francisco.//)

//[[Nicholas Nassim Taleb returns to econtalk|http://www.econtalk.org/archives/2009/03/taleb_on_the_fi.html]]// It has been two years since [[Nicholas Nassim Taleb's first appearance on econtalk.org|http://www.econtalk.org/archives/2007/04/taleb_on_black.html]]. Now, he gets introduced by the host, Russ Roberts, as "one of the few thoughtful people alive today who can say 'I told you so.'".

I got interested in [[Nicholas Nassim Taleb|http://www.fooledbyrandomness.com]] because of his first appearance on econtalk. (I got interested in econtalk because of [[Robin Hanson's first appearance|http://www.econtalk.org/archives/2007/05/hanson_on_healt.html]].) I've subsequently read Taleb's book //[[The Black Swan|http://www.randomhouse.com/catalog/display.pperl/9781400063512.html]]// and found it to be rife with errors and ridiculousities, and yet delightful, profound and overwhelmingly important.

This reprise does not disappoint. Far from simply saying 'I told you so.', Taleb and Roberts provide important insights into why things like this happen and what to do about it.

(//''Mom:'' That all makes sense, right?  Listen to the podcasts.//)

Hello I am at ~CodeCon.

I am showing Zack Weinberg how I update my blog.

My personal server which runs http://zooko.com is unreachable at the moment.  It'll probably be back on-line later today, but in the meantime if you were looking for my résumé.html, here it is on the Tahoe decentralized storage grid: [[résumé.html|../../file/URI%3ACHK%3Ano2l46woyeri6xmhcrhhomgr5a%3A5p7cxw7ofacblmctmjtgmhi6jq7g5wf77tx6befn2rjsfpedzkia%3A3%3A10%3A8328/@@named@@/r%C3%A9sum%C3%A9.html]].  :-)

* ~SHA-1 broken?  [[These slides from Eurocrypt 2009 rump session|http://eurocrypt2009rump.cr.yp.to/837a0a8086fa6ca714249409ddfae43d.pdf]] by Cameron ~McDonald, Philip Hawkes, and Josef Pieprzyk, presented by Greg Rose, seem to say that they figured out how to compute ~SHA-1 collisions with 2^^52^^ cost.  The previous best-known technique required 2^^63^^ computations, so if this is right then it puts ~SHA-1 squarely in the category of things that [[some people|http://ioerror.livejournal.com]] will abuse and exploit just to make a point.  [[Here's my take|http://allmydata.org/pipermail/tahoe-dev/2009-April/001663.html]] in a message to tahoe-dev.
* Matt Mackall has written a new tool to give a [[useful measurement of how much memory|http://lwn.net/SubscriberLink/329458/d28c2d45a663045a]] your Linux program is using.  Hooray!
* Added to [[things I have read]]:
** Stefan Tillich: [[Hardware Implementation of the SHA-3 Candidate Skein|http://eprint.iacr.org/2009/159]];  Comparing this to [[the EnRUPT SHA-3 proposal|http://www.enrupt.com/SHA3/Supporting_Documentation/EnRUPT_Specification.pdf]] (page 24) suggests that ~EnRUPT is probably better than Skein both ASIC and FPGA, in both area and speed.  This is true even after remembering that ~EnRUPT as it was proposed was broken, and that the current proposal is to double (or so) the number of rounds, so I probably need to mentally double all of the "How many seconds to process 1 Gbit" in the ~EnRUPT results.
* Okay, http://zooko.com is back, thanks to the heroic efforts of Brian Warner and rootard.  I am very pleased that my blog remained fully functional (for both read and write) while my server was off-line.  (Because this blog is on [[Tahoe-LAFS|http://allmydata.org]].)

I picked up the third book in //[[The Golden Age|http://www.amazon.com/dp/0765349086]]// from the library today, and I've read more than half of it already. I crowed with pleasure when the opening pages started with an intense space battle including previously unknown laws of physics, mind virusses, nanotech, and tactical personality implantation, and then the survivors sat down to argue strategy using, among other ideas, detailed renditions of Ayn Rand's "fallacy of the stolen concept" and David Ricardo's "comparative advantage". (Those names were not used, but I recognized the ideas.)

finished //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// and re-organized my [[things to read]] tiddler;   //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// was satisfying both as an adventure novel with an admirable hero (I always like those) and as a thought-provoking exploration of many ideas.

Hooray!  I finally have an intuition for Bayes' Theorem thanks to [[this visualization|http://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem]].  Thank you, Oscar Bonilla!

 * added //Kevin Bauer, Damon ~McCoy, Douglas Sicker, Dirk Grunwald: [[BitBlender: Light-Weight Anonymity for BitTorrent|http://systems.cs.colorado.edu/~bauerk/papers/alpaca2008.pdf]]// to [[things to read]]
 * I'm told that [[this German radio program|http://c-radar.ccc.de]] has described [[Tahoe-LAFS|http://allmydata.org]] in very positive terms.  However, I can't tell what they are saying.  :-)

 * I'm speaking about [[Tahoe|http://allmydata.org]] at [[Boulder Linux Users Group|http://lug.boulder.co.us]] next Thursday.
 * [[Brian Warner|http://www.lothar.com/blog]] explained [[the Sphinx remailer crypto idea|http://research.microsoft.com/en-us/um/people/gdane/papers/sphinx-eprint.pdf]] to me.
 * [[Sunah Cherwin|http://sunahweb.com]] sent me a nice gift: a dead-tree copy of //[[What The Dormouse Said|http://www.amazon.com/dp/0670033820]]//.
 * Come over to my house at 20:00 tonight to play games such as Go and Dominion.

I read Vasudevan et al.: [[FAWNdamentally Power-efficient Clusters|http://www.cs.cmu.edu/~vrv/papers/hotos2009.pdf]] which was [[recommended by Wes Felter|http://wmf.editthispage.com/2009/05/07]].  They say by using "embedded" chips like ARM or Atom we can build servers which cost about a third as much per gigabyte of storage as the current commodity servers cost.  Last year I investigated such an approach to cheap storage servers for http://allmydata.com, but couldn't find such machines with SATA interfaces.  Oh, I see: "costs for FAWN machines were based on private communication with a large manufacturer".  So it isn't actually for sale yet? 
Update: thanks to Zandr Milewski and to J Silverman from the #edev channel on irc.freenode.net I see that I can buy (retail) a Buffalo Linkstation ~LS-CHL with a 1.5 TB drive in it for about $265.  (1.5 TB hard drives are currently a better deal per TB than 2.0 TB [[according to pricegrabber.com|http://storagereview.pricegrabber.com/search_attrib.php/vendorIds%255B%255D=1643/page_id=11/popup10%255B%255D=12:393/popup10%255B%255D=14:393/popup20%255B%255D=1:394/popup30%255B%255D=8:144/]].)
If I plug in my numbers for a Linkstation and a 1.5 TB drive in place of their FAWN device, I get $0.20 TCO/GB instead of their $0.19.  If I plug in some numbers for a 2U server that can run 6 SATA drives and that is currently listed with a price on web sites it works out to $0.29 TCO/GB.
In fact, if I plug in the paper's own numbers for the "Traditional five 2TB HD" I don't get the same result they do.  Shouldn't the TCO/GB column in Table 2 work out to $0.265 TCO/GB instead of $0.53?  They mentioned that they divided by two for "even larger bulk discounts", so maybe they forgot to divide that field by two.  That would be a major flaw; for my purposes comparing those fields -- "Traditional five 2TB HD x TCO/GB" to the "FAWN 2TB Disk x TCO/GB" -- is the central measurement.
Still, I'm interested to learn that I could buy a Buffalo Linkstation for each 1.5 TB hard drive that I want to deploy instead of buying a 2U server for each six 1.5 TB hard drives and reduce overall cost from $0.29 TCO/GB to $0.24 TCO/GB.  Too bad that you can't shoehorn six of them into a 2U rack space.  :-)  If someone actually made servers with guts like that but a rackable form factor I would be interested to see what price they ask for them.

//''Mom:'' We're trying to figure out how to make it cheaper to buy and operate servers -- the computers that store the data and produce the results when you visit web sites, check your e-mail, backup your files to [[allmydata.com|http://allmydata.com]], etc..//

I wrote to the primary author of that paper that I mentioned [[yesterday|2009-05-09]], Vijay Vasudevan, and he confirmed that he forgot to divide that field by half.  He also suggested [[this mini-itx Atom motherboard|http://www.newegg.com/Product/Product.aspx?Item=N82E16813121360]] as a basis for a cheap little server.

I'm off to speak about [[Tahoe-LAFS|http://allmydata.org]] at [[Boulder Linux Users Group (BLUG)|http://lug.boulder.co.us]].

My presentation at BLUG went pretty well.  I came with only one slide, and took a lot of questions, and only one person fell asleep.  My memories are fuzzy -- I think I was already sick with the flu which has kept me down from then until now.

Wow, that was a bad flu.  I was incapacitated and miserable -- fever, chills, aches, sweats.  My time sense was distorted so I didn't know if it was morning or evening and it seemed to drag on for day after day, although now that I'm out of it I can see that it was only about 60 hours.  I lost five pounds.  I sure hope my boys and my pregnant wife don't get it.

Update: Oh no, it turns out that I'm still not 100% over it yet, and I think Amber is coming down with it now.  Poor Amber.

Oh wow, after I wrote [[that last post|2009-05-17]] I started feeling worse again and went back to bed.  I'm still not sure that I'm entirely over it.

I'm alive!

[[weaknesses discovered in AES!?]]

Elliot has decided that, since he is graduating from pre-school and becoming a Big Boy, that it is time to learn how to swim and how to read.

* I wrote a rant about [[longevity and dementia, drugs and diet]].
* added to [[issue tickets]]:
** //[[cryptopp #?+0|http://groups.google.com/group/cryptopp-users/browse_thread/thread/6d37437aa40fc135]]: patch: build libcryptopp.so//
** //[[cryptopp #?+1|http://groups.google.com/group/cryptopp-users/browse_thread/thread/97570197f77cc80c]]: patch: initialize randpool with zeroes//

* The [[duplicity|http://duplicity.nongnu.org]] open source backup tool [[now supports Tahoe-LAFS|http://allmydata.org/pipermail/tahoe-announce/2009-May/000009.html]]!
* added to [[things I have read]]:
** //Henry Harpending: [[Henry And The Cape Buffalo|http://the10000yearexplosion.com/henry-and-the-cape-buffalo]]// -- An anthropologist begins to wonder how hunter-gatherers ever managed to kill large game animals with pointy sticks... as he is running for his life.
** {{multiline{//Gardner Dozois (ed.): [[The Year's Best Science Fiction: Twenty-Second Annual Collection|http://www.amazon.com/exec/obidos/ASIN/0312336608]]// -- It didn't seem as consistently strong as the old Gardner Dozois-edited tomes that I used to read, perhaps partly because I read almost the whole thing while sick with the flu.
''SPOILER ALERT'' -- the following includes minor spoilers for //Rainbows End// by Vernor Vinge

I was surprised to find that there was a Vernor Vinge story in here named "Synthetic Serendipity", which is the first chapter of //[[Rainbows End|http://www.amazon.com/exec/obidos/ASIN/0812536363]]// with a few characters changed.  I think this explains one of the weaknesses of the book -- the disjointedness.  Louise Chumlig is a dominant character in "Synthetic Serendipity" and also in the first chapter of //Rainbows End//, but then she silently fades out.  Of course, part of the theme of //Rainbows End// is that the world is too complex, evolves too fast, and your viewpoint is too limited (even if you are the most omniscient, computationally-enhanced and organizationally-enhanced superman that modern technology and the political power of a nation-state can produce).  But the full novel explores that theme in better and more satisfying ways.  I attribute the transience of Louise Chumlig to the inevitable disjointedness of extending a short story into a novel.

(P.S.  Amber points out that the above is not a very strong recommendation of //Rainbows End//.  I've read it twice so far and I intend to read it again.  I suspect that it is the most important work Vernor Vinge has ever produced, although not the most entertaining.  I have the strong suspicion that most of //Rainbows End// doesn't make sense unless you figure out certain mysteries which are ''never'' revealed explicitly.)
}}}
* {{multiline{I simplified the tags on this blog.  All entries that used to be marked with either ''//software engineering tools//'' or ''//software engineering automation//'' are now marked with [[software engineering]].  Likewise, all entries the used to be marked with ''//secure hash functions//'', which were also of course marked with [[cryptography]], are now merely marked with [[cryptography]].

I wonder if I should do some further coalescing of tags.  Should there be separate tags for [[cryptography]] and for [[computer security and reliability]], or should they all just be marked as [[computer security and reliability]]?  What about [[politics and economics]], [[freedom and privacy]], and [[Freedom-Compatible technology]]?  Maybe those are all the same thing.

Opinions from readers are welcome!}}}

Last night Amber and I watched this video of [[Daniel Suarez speaking at The Long Now Foundation|http://fora.tv/2008/08/08/Daniel_Suarez_Daemon_Bot-Mediated_Reality]].  I like the way he thinks and I share his concerns and interests, but he got a few facts wrong and I think there are a lot of interesting possible strategies besides the ones he focusses on at the end.

Still, it felt good to see other people thinking about the kinds of issues that bedevil me.

I finally read //George Bray: [[Obesity: a failure of homeostasis because of hedonic rewards: response to the letter from Gary Taubes|http://testgrid.allmydata.com:3567/file/URI:CHK:q3cwozbit4be2jxmxfre65wzdq:h6rm26yuvcjttexlwhdi6lfcgqoxkordv52xxkwyhxphkytcfuva:3:10:116345/@@named=/Bray_Taubes_rebuttal_rebuttal_rebuttal.pdf]]//.

What a disappointment!  Dr. Bray asserts the obviousness of his own hypothesis and then issues a series of statements which you can easily disprove by comparing against [[his original book review|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/bray-review-of-gcbc.pdf]] and [[Taubes's rebuttal|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/taubes-response-to-bray-ob-reviews.pdf]].  Dr. Bray attempts to cover up the way that, in the initial book review, he claimed that the book omitted mention of low-density lipoproteins and high-density lipoproteins, when in fact the book describes them in extensive detail, including a comprehensive history of how they were discovered and how scientific perception of their importance changed through the 20th and early 21st centuries.

(I have a hypothesis of my own as to why Dr. Bray originally claimed that the book omitted those concepts.  I happened to be looking for those concepts in the index of the paperback version of the book myself the other day, and I noticed that the index doesn't link to them under "HDL" or "high-density lipoproteins".  This is an error in the index.  It does link to them under "LDL" and it links to them extensively under "lipoproteins".  My hypothesis is that Dr. Bray didn't read Part Two or Part Three of the book, which describe the kinds of lipoproteins in great detail, and then he looked in the index under "HDL", found no index entry, and concluded that the book missed this important aspect of the picture.)

He also bizarrely misquotes Taubes's rebuttal on the topic of Bray's conflict of interest.  Taubes's rebuttal ended with:
<<<
Finally, I would like to identify one potential conflict of interest on Bray's part that he neglected to mention. In the 1970s, as I discuss in GCBC, the hormonal/enzymatic regulation of fat tissue was deemed irrelevant to the cause, cure and prevention of human obesity. I identify Bray as one of two individuals most responsible for this dubious accomplishment, and 'for effectively removing the [century-old] concept of the fattening carbohydrate from the nutritional canon . . .' (GCBC, p. 417). Thus, Bray's critique of GCBC may be as much a defense of his own career as it is an unbiased assessment of the book. Readers should be aware of this possibility. It would be a shame if obesity researchers based their opinions on Bray's review, rather than the book itself.
<<<
In the new rebuttal-rebuttal by Bray, he says:
<<<
Near the end of the letter, Mr Taubes suggests that my review of his book may be a ‘conflict of interest’. He says ‘I [Bray] may be defending what my scientific research has led me to believe’. If this is a conflict of interest, then all scientists have a conflict of interest.
<<<
What?  Did he not understand the potential conflict of interest issue?  The book singles out Dr. Bray (among others) as an historical figure and criticizes his career.  When he then publishes a review of that book, this forms a potential conflict of interest which should be disclosed.  This is entirely different from someone arguing against hypotheses that you prefer -- that is the sort of "conflict of interest" which everyone knows is going on all the time and does not need to be pointed out.  And why did he make it look like he was quoting Taubes with those quote marks and insertion of his own name in square brackets?  I can't find any quote like that from Taubes.  Perhaps Bray didn't actually know that the book included criticism of him when he published the original book review -- his name doesn't appear in Part One of the book, and perhaps he didn't read the other two parts, nor look for his own name in the index.

I'm honestly surprised at this level misrepresentation and/or misunderstanding, because I've been told that Dr. Bray is one of the leading lights of human nutrition research both in the second half of the 20th century and today.  The leading scientists in fields that I am more familiar with (cryptography, computer science, philosophy, linguistics) are a lot more precise and conscientious with their words.

I'm sure that there are important errors and incompleteness in Gary Taubes's theory.  We need careful analysis, and above all good experiments, to learn more.  We need much more careful and fair examination than is shown here.  Is this indicative of the state of the art of nutrition research?

* Come over to my house on Sunday to play and celebrate!
* added to [[things to read]]:
** Sachin Katti, Hariharan Rahul, Wenjun Hu, Dina Katabi, Muriel Medard, Jon Crowcroft: [[XORs in The Air: Practical Wireless Network Coding|http://127.0.0.1:8123/file/URI%3ACHK%3A5v6bgzvcdovykxluylk5omyp5a%3Anzf5v2wmv2ox3ajb5gvd7f7photy43eb2zp724ls5qz6oaphqhhq%3A3%3A10%3A452699/@@named=/XORs_in_the_air-practical_wireless_network_coding.pdf]]

* //"Tahoe rules"//
This is the kind of thing that fuels much open source development -- gratitude and compliments.  And offers to pitch in and help!
<<<
From:   //email address withheld//
Subject: Tahoe rules
Date: May 29, 2009 22:58:33 PM MDT
To:   zooko@zooko.com

Hi Zooko, I am placing this e-mail just to let you know that I like Tahoe!

I am planning to use it to create a shared-space between my team mates workstations, as a test project,
and then to try it on production in some of our systems in the enterprise I work.

Thanks for the awesome LAFS. I would like to contribute, send patches, if I find/implement something.

Regards,
-- 
//Name Withheld//
<<<
* added to [[things to read]]:
** //Yvonne Hitchcock, Paul Montague, Gary Carter, Ed Dawson: [[The efficiency of solving multiple discrete logarithm problems and the implications for the security of fixed elliptic curves|/file/URI:CHK:qxugzb4esi6b6clupozci7x6hu:jscs6fjjtm5jmqh3rchurubi5qmz7pxc2bf54ixt7vn64zmy5abq:3:10:305839/@@named=/multipledlog.pdf]]//
* added to [[things I have read]]:
** //Gautham Sekar, Souradyuti Paul, Bart Preneel: [[Related-key Attacks on the Py-family of Ciphers and an Approach to Repair the Weaknesses (2007)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.92.3976]]// //~RCR32// and //~RCR64// are fast stream ciphers and the latest in the long line of the Py family.  2.7 cycles per byte.  Wei Dai's [[Crypto++ benchmarks|http://cryptopp.com/benchmarks.html]] show 2.0 cycles per byte for Salsa20/8.  The authors mention at the end that it could be interesting to build a secure hash function from it.  I guess nobody did that for the ~SHA-3 process. Of the three authors, Preneel worked on the //LANE// candidate for ~SHA-3.  (See also my comments about //LANE//: [[today's crypto education]], [[2008-11-26]], [[2008-12-17]].)

* added to [[issue tickets]]:
** [[tailor #693|http://bugs.darcs.net/issue693]]: rmfile patch created without the actual content removal
** [[darcs #1477|http://bugs.darcs.net/issue1477]]: please speed up "darcs show contents"
* moved from [[issue tickets]] to [[tickets closed]]:
** [[cryptopp #?+1|http://groups.google.com/group/cryptopp-users/browse_thread/thread/97570197f77cc80c]]: patch: initialize randpool with zeroes @@fixed@@
** [[pyflakes #2720|http://divmod.org/trac/ticket/2720]]: Release Pyflakes @@fixed@@
** [[konqueror #321656|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321656]]: iso-8859-1 and/or utf-8 character not decoded properly @@fixed@@
** [[pycryptopp #9|http://allmydata.org/trac/pycryptopp/ticket/9]]: link against existing (system) libcrypto++.so @@fixed@@
** [[twisted #3568|http://twistedmatrix.com/trac/ticket/3568]]: ERROR from conch test when pycrypto is not installed @@fixed@@
** [[pycryptopp #11|http://allmydata.org/trac/pycryptopp/ticket/11]]: submit patch for Brainpool ECC to Crypto++ @@fixed@@
** [[cryptopp #?+0|http://groups.google.com/group/cryptopp-users/browse_thread/thread/6d37437aa40fc135]]: patch: build libcryptopp.so @@invalid@@ rejected in [[this mailing list discussion|http://groups.google.com/group/cryptopp-users/browse_thread/thread/6d37437aa40fc135]]

Every year Amber attends the annual conference on computational linguistics.  This year it was in Boulder!  I snuck in so that I could see a few talks, like 

Tae Yano, William W. Cohen and Noah A. Smith: //Predicting Response to Political Blog Posts with Topic Models//
Weifu Du and Songbo Tan: //An Iterative Reinforcement Approach for ~Fine-Grained Opinion Mining//
Luciano Barbosa, Ravi Kumar, Bo Pang and Andrew Tomkins: //For a few dollars less: Identifying review pages sans human labels//
Stephan Greene and Philip Resnik: //More than Words: Syntactic Packaging and Implicit Sentiment//

That last one was measuring the use of "framing", such as the difference between "The soldier's jeep swerved into a crowd, and five people were killed." vs. "The soldier swerved his jeep into a crowd, killing five people.".  The scientists developed algorithms to analyze how you construct your sentences and evaluated the success of their algorithms by how well they predicted which side of a controversial political issue a writer was on.  (The issues they used to test to algorithm included the death penalty in the USA and the ~Palestinian-Israeli conflict.)

Unfortunately, this research wasn't measuring the ''efficacy'' of framing -- to what degree does framing succeed at directing the listener's attention or prejudicing them to an interpretation or a moral judgment?  It does provide groundwork for future research to do so.

* going to talk about [[Tahoe-LAFS|http://allmydata.org]] at [[Northern Colorado Linux Users Group|http://www.nclug.org]] tomorrow
* added to [[issue tickets]]:
** [[twisted #3238|http://twistedmatrix.com/trac/ticket/3238]]: patch: declare that twisted requires pywin32 if it is to offer process management or iocp reactor on Windows
* moved to [[issue tickets closed]]:
** [[nevow #2830|http://divmod.org/trac/ticket/2830]]: setup.py incorrectly declares twisted.plugins to be a package @@fixed@@

* Hooray!  Launchpad.net now lists the [[Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] as an open source licence.  You can see it on the [[Launchpad page for Tahoe-LAFS|https://launchpad.net/allmydata.org]].
* Irby got Amber and me started on //The Warriors// series.  He owns the first set.  We're now on [[The Warriors book 4: Rising Storm|http://www.amazon.com/Rising-Storm-Warriors-Book-4/dp/0060525630/ref=sr_1_1?ie=UTF8&s=books&qid=1244570985&sr=8-1]].  It's exciting and dramatic.

I'm reading this magazine article: [[Bullion and Bandits: The Improbable Rise and Fall of E-Gold|http://www.wired.com/threatlevel/2009/06/e-gold]], [[as recommended by Ian Grigg|https://financialcryptography.com/mt/archives/001169.html]].
* added to [[issue tickets]]:
** [[twisted #3878|http://twistedmatrix.com/trac/ticket/3878]]: build executables for trial, twistd, manhole, etc. on Windows (if setuptools is installed)
* Sigh... Building modules for Python 2.6 for Windows using only Free Software build tools [[sometimes fails|http://allmydata.org/pipermail/tahoe-dev/2009-June/001974.html]].  For Python 2.5, it works fine.  I guess my current work-around is to recommend that my Windows users use Python 2.5.  (Hm, I wonder if this is another case where [[ctypes|http://docs.python.org/library/ctypes.html]] would be a more portable way to access native code from Python than the ~CPython API.)
* Off I go to the [[Boulder Linux Users Group|http://lug.boulder.co.us]].
* BLUG was a treat!  [[Rob Savoye|http://www.welcomehome.org/rob.html]]'s talk about [[Cygnal|http://www.gnashdev.org/?q=node/5]], [[Gnash|http://www.gnashdev.org]], and a dozen other fun topics was exciting.  I'm delighted to find out that there is a [[hippie Free Software hero|http://www.welcomehome.org/rob/rob.jpg]] [[living deep in the mountains|http://senecass.com]] above my home town.

//''Hack Fest!''//

* added to [[issue tickets]]:
** [[pycryptopp #23|http://allmydata.org/trac/pycryptopp/ticket/23]]: can't build Python module for Python 2.6 with ~MinGW
** [[python #6280|http://bugs.python.org/issue6280]]: calendar.timegm() belongs in time module, next to time.gmtime()

It makes me happy that Amber keeps tabs on me by examining the status of my buildbots: [[Tahoe-LAFS buildbot|http://allmydata.org/buildbot/waterfall]], [[pycryptopp buildbot|http://allmydata.org/buildbot-pycryptopp/waterfall]], and the new arrival, [[zfec buildbot|http://allmydata.org/buildbot-zfec/waterfall]].

* Josh is here!  Nathan is coming to visit for a while!  And my Dad is coming for the weekend, too!
* "Hash functions are the nails of cryptography; they hold the world together." -Niels Ferguson speaking at the first ~SHA-3 conference
* Help!  I need a Debian Developer to help package [[Tahoe-LAFS|http://allmydata.org]] for Debian! //Update: Thanks, POX!//
* added to [[things I have read]]:
** [[the slides and transcripts from the first SHA-3 conference|http://csrc.nist.gov/groups/ST/hash/sha-3/Round1/Feb2009/program.html]].
** Hal Finney's [[summary of Craig Gentry's deep homomorphic encryption|http://www.mail-archive.com/cryptography@metzdowd.com/msg10571.html]]

* Ada Trouble is growing up! She's beginning to look a bit like her mom. http://www.flickr.com/photos/quinn/3634423983/
* added to [[things to read]]:
** Lera Boroditsky: [[How does our language shape the way we think?|http://edge.org/3rd_culture/boroditsky09/boroditsky09_index.html]] as recommended  by [[Kris Nuttycombe|http://nuttycombe.gaia.com/blog]]

I added links to the ~Tahoe-LAFS, pycryptopp, and zfec buildbots on the "side stuff" bar over there to the right. ====>

This is for my convenience, so I can easily load those three web pages, but you are welcome to use it too!  That way you can tell how well those three projects are passing their tests.

Type the text for '2009-06-22'
* added to [[issue tickets]]:
** [[ubuntu #389865|https://bugs.launchpad.net/ubuntu/+source/python-setuptools/+bug/390965]]: Ubuntu package of setuptools has no version number in the filename of the {{{.egg-info}}} file

* added to [[issue tickets]]:
** [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory
* moved from [[issue tickets closed]] back to [[issue tickets]::
** [[pyopenssl#238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: please provide binaries

We're off to Harlow Platts Park for a neighborhood parade and picnic.

It's Game Day!

* added to [[things to read]]:
** w3schools.com: [[XML Schema Tutorial|http://www.w3schools.com/Schema/default.asp]]
** Dan Reverri: [[Writing an AMQP connector with Python, Twisted, Trial and txAMQP|http://app.arat.us/blog/?p=112]]
* added to blogroll:
** [[Duncan McGreggor|http://oubiwann.blogspot.com]]

* added to blogroll:
** [[Ben Hyde|http://enthusiasm.cozy.org]]
* added to [[things to read]]:
** AMQP standards writers: [[Advanced Message Queueing Protocol spec, v0-10|http://jira.amqp.org/confluence/download/attachments/720900/amqp.0-10.pdf?version=1]]

* added to [[things to read]]:
** Ursula K. ~LeGuin: [[The Dispossessed|http://www.sfsite.com/01b/dis73.htm]] (as recommended by Jake Appelbaum) (I've already read most of this book at least once.)

~TahoeLAFS: a good tool for anarchist file swappers and for giant governments.

I had an interesting conversation this morning with a friend who worked for the U.S. National Security Agency (NSA) for a while and now works for ~Booz-Allen Hamilton (which is an arm of NSA).  He said that cloud computing security is a big issue over there in the military-industrial complex.  He reports that the new administration of the US Federal Government "knows what a computer is".

He pointed out that [[TahoeLAFS|http://allmydata.org]] is, of course, the cloud storage tool that already comes with effective security built in.  (This statement implies more than it says, because to really have ''effective'' security you have to address vexing problems of user interface and human behavior -- most purported security solutions simply ignore these issues.)

So, I think ~TahoeLAFS is going get a lot more attention from a lot more people after the upcoming v1.5 release.

I posted a letter to the tahoe-dev mailing list arguing that ~TahoeLAFS can't (at least in the short term) offer the sort of "filesystem shape privacy" that we might want: [[simple authority taxonomy versus a kind of privacy|http://allmydata.org/pipermail/tahoe-dev/2009-July/002302.html]].


* added to [[issue tickets]]:
** [[bzr #294709|https://bugs.launchpad.net/bzr/+bug/294709]]: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" @@This will make bzr work on top of ~TahoeLAFS!@@
* added to blogroll:
** [[Nick Szabo|http://unenumerated.blogspot.com]]
* moved from [[things to read]] to [[things I have read]]:
** //[[The Warriors book 4: Rising Storm|http://www.amazon.com/Rising-Storm-Warriors-Book-4/dp/0060525630/ref=sr_1_1?ie=UTF8&s=books&qid=1244570985&sr=8-1]]// by Erin Hunter
** Leslie Lamport: [[Paxos Made Simple|http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf]]
** Tushar Chandra, Robert Griesemer, Joshua Redstone: [[Paxos Made Live -- An Engineering Perspective|http://www.chandrakin.com/paper2.pdf]]
* added to currently reading:
** //[[The Warriors book 5: Dangerous Path|http://www.amazon.com/Dangerous-Path-Warriors-Book/dp/0060525657/ref=pd_sim_b_1]]// by Erin Hunter

post to tahoe-dev: [[two reasons not to use semi-private keys in our new cap design|http://allmydata.org/pipermail/tahoe-dev/2009-July/002314.html]]

I invented a new public key technique which is elegant.  But we probably shouldn't use it, because I invented it and it is new.  Also because elliptic curve public keys are twice as big as one might wish them to be.

The last three [[issues blocking the TahoeLAFS v1.5 release|http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.5.0]] are on the verge of being fixed, thanks to the volunteer efforts of a loosely organized team of open source hackers.

Tomorrow I'll write up a Release Announcement advertising ~TahoeLAFS as the only secure cloud storage technology.

* added to //currently reading//: //[[JavaScript: The Good Parts|http://oreilly.com/catalog/9780596517748/]]// by [[Doug Crockford|http://www.crockford.com]]

* added to blogroll:
** [[Nate Lawson|http://rdist.root.org]]
** [[Danny O'Brien|http://oblomovka.com]]
 Danny O'Brien has an intriguing article: [[What if the Firm is The Wrong Size?|http://www.oblomovka.com/wp/2009/07/12/#returns]].  It is a book review of //[[Organization Theory|http://www.abebooks.com/Organization-Theory-Kevin-A-Carson-BookSurge/1252202295/bd]]// by [[Kevin Carson|http://mutualist.blogspot.com]].  I was going to add it to my amazon.com wishlist, but amazon.com appears to be unreachable to me at the moment -- I get //"Connection Interrupted / The connection to the server was reset while the page was loading. / The network link was interrupted while negotiating a connection. Please try again."// when I point my web browser at amazon.com.  Perhaps this symbolic of one of the themes under discussion -- that giant, centralized corporations might not be as inherently efficient as they appear.  (Somehow doubly ironic, since amazon.com, in addition to being a giant centralized corporation, is also home to pioneering engineering in decentralized computing).  Anyway, I guess I'll just add that book to my [[things to read]].
I do hope it isn't still the case that Kevin Carson builds on the Labor Theory of Value, [[as alleged by Walter Block|http://en.wikipedia.org/wiki/Kevin_Carson#Criticism]].
* things Elliot wants for his next birthday:
** a remote control cake that has candles that light up or turn off by remote control; Also, it has wheels.
** a remote control gecko flashlight; You can move it around and it can go up on the ceiling and it is shaped like a gecko and it is a flashlight.
** a remote control snake; It can move and change color and stick its tongue in and out.
** a remote control Darth Vader with a shiny red nose, and you can remotely open its mouth, and it has teeth that shine a little bit, and it has spikes on its back that are gold.  No, silver.
* I drew a logo for ~TahoeLAFS.  It isn't "official" yet -- first let's see if the other Tahoe hackers like it.
[img[TahoeLAFS-logo|/uri/URI%3ACHK%3Alztifk7me3pnijhvgmhzemlg24%3Agshelvesidaax5yo44eaurvt5seqlad3vsr5eefmpap3kxkzwrlq%3A3%3A10%3A5977]]

* Now that's what I like to hear:
{{{
<midnightmagic> the more I use it, the happier I am with my tahoe grid.
* midnightmagic is a happy user.
<midnightmagic> happy.. happy.. don't want to go on the cart.
}}}
* (Cart?  I think that was a reference to a Monty Python skit.)
* moved from [[issue tickets]] to [[issue tickets closed]]:
** [[bzr #294709|https://bugs.launchpad.net/bzr/+bug/294709]]: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" @@Fixed -- this makes bzr work on top of ~TahoeLAFS!  Thank you, Nils Durner!@@

* Cleversafe says that encryption is overrated. Their arguments are confused and include an unhealthy dose of Fear Uncertainty and Doubt: http://allmydata.org/pipermail/tahoe-dev/2009-July/002383.html .
* added to [[things I have read]]:
** Tillich and Martin Feldhofer, Wolfgang Issovits, Thomas Kern, Hermann Kureck, Michael Mühlberghuber, Georg Neubauer, Andreas Reiter, Armin  Köfler, Mathias Mayrhofer: [[Compact Hardware Implementations of the SHA-3 Candidates ARIRANG, BLAKE, Grøstl, and Skein|http://eprint.iacr.org/2009/349]]; Comparing this to [[the EnRUPT SHA-3 proposal|http://www.enrupt.com/SHA3/Supporting_Documentation/EnRUPT_Specification.pdf]] (page 24) shows that ~EnRUPT is better than Skein, ARIRANG, BLAKE, and Grøstl in ASIC in both area and speed.  This is true even remembering that ~EnRUPT as it was proposed was broken, and that the current proposal is to double the number of rounds, so you need to halve all of the throughput measurements in the ~EnRUPT results.
UPDATE:  NIST has announced the fourteen ~SHA-3 candidates to advance to round 2 of the ~SHA-3 contest!  Much to my disappointment, ~EnRUPT, ~Edon-R, and LANE are out.  Of the ones that have been advanced to round 2, the only ones that I like are ~CubeHash, Skein, and BLAKE.  I'm disappointed that NIST chose eleven other candidates that perform worse than ~EnRUPT and ~Edon-R.  Oh well.

* I just re-read [[Tim Bray's introduction to the Sun Cloud|http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud]]. William Vambenepe's essay (linked below) led me to re-read it.
* I just read [[Ivan Krstić|http://radian.org/notebook/nonsense-omelet]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/07/24]])
* added to [[things I have read]]:
** William Vambenepe: [[REST in practice for IT and Cloud management (part 1: Cloud APIs)|http://stage.vambenepe.com/archives/863]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/07/16]])
* added to [[things to read]]:
** Dave Neary: [[Open Source Community Building – Barriers to Entry|http://dneary.free.fr/articles/Community_barriers_to_entry_checklist.pdf]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/07/24]])
* NEWFLASH!  Amber is eating breakfast at the South Side Walnut, all by herself.  Go hang out with her.  2009-07-25 10:20 Mountain Standard Time ~UTC-6
* recipe:
//Elliot's Mixture//
* 1/4 tablespoon vanilla
* 1 & 1/2 tablespoon turbinado sugar
* 1 & 1/2 tablespoon splenda
* 2 tablespoons peanut butter
* 1/2 cup cream
taste after every addition
stir together and eat

* added to [[things I have read]]:
** Norm Hardy: [[The Confused Deputy (or why capabilities might have been invented)|http://www.cis.upenn.edu/~KeyKOS/ConfusedDeputy.html]] (I just now re-re-read this paper.  It is short, informal, and ancient, but it is the best summary of a critical modern security issue.)
** The Economist: [[Financial economics: Efficiency and beyond|http://www.economist.com/displaystory.cfm?story_id=14030296]] (note: fragile URL; I should copy the document onto a ~Tahoe-LAFS grid and link to the copy.)
* added to [[things to read]]:
** Ron Rivest: [[All-Or-Nothing Encryption and The Package Transform (1997)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.1059]] (because of [[this controversy about cleversafe encryption|http://allmydata.org/pipermail/tahoe-dev/2009-July/002383.html]])
* removed from "currently reading":
** Amber and I gave up on reading //[[The Warriors book 5: Dangerous Path|http://www.amazon.com/Dangerous-Path-Warriors-Book/dp/0060525657/ref=pd_sim_b_1]]// by Erin Hunter, because it is too frustrating that Fireheart doesn't tell people what he knows.  I might finish it myself, though, because Fireheart //is// a brave warrior and Tigerclaw //is// a terrible villain.
* added to blogroll:
** [[Mike Eades|http://proteinpower.com/drmike]]

* I'm helping my brother, Nathan Wilcox, practice his presentation for ~BlackHat tomorrow on [[cloud security|http://www.blackhat.com/html/bh-usa-09/bh-usa-09-speakers.html#Stamos]].
* added to blogroll:
** [[J.P. Calderone|http://jcalderone.livejournal.com]]
* added to [[things to read]]:
** Paul Crowley: [[Squaring Zooko’s Triangle, part two|http://www.lshift.net/blog/2007/11/21/squaring-zookos-triangle-part-two]] (blog entry)
I suspect Paul Crowley might have fixed [[Zooko's Triangle|http://en.wikipedia.org/wiki/Zooko%27s_triangle]], a fundamental theorem in distributed systems that [[Mark Miller|http://caplet.com]] and I came up with, but which I've never quite understood.  Paul Crowley's idea is that there is actually a tetrahedron of which the original Zooko's Triangle is one face.  That would explain why my understanding of Zooko's Triangle varies and why it sometimes doesn't fit -- I'm probably switching between faces of a tetrahedron without realizing it.
* I read this blog entry: Cloudera: [[File Appends in HDFS|http://www.cloudera.com/blog/2009/07/17/file-appends-in-hdfs]].  It is interesting to think about how ~Tahoe-LAFS can avoid the problems that they encountered while adding a file append feature.  Also we might not want to spend the engineering time to add file-append first, but instead jump straight to medium-sized distributed mutable files, or even large distributed mutable files, or something.  I suspect that by the time we are ready to start such a project (at least a couple of months from now), we'll be driven more and more by user demand and less by our own creativity.
* added to [[things to read]]:
** The Economist: [[The state of economics: The other-worldly philosophers|http://www.economist.com/displaystory.cfm?story_id=14030288]] (note: fragile URL; I should copy the document onto a ~Tahoe-LAFS grid and link to the copy.)
* Last night I went to bed with Ron Rivest: [[All-Or-Nothing Encryption and The Package Transform (1997)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.1059]] (because of [[this controversy about cleversafe encryption|http://allmydata.org/pipermail/tahoe-dev/2009-July/002383.html]]).  It is indeed a strange algorithm, with interesting properties but probably useless.  One of the motivations for AONT is that your government might forbid you to use long crypto keys, and AONT makes slightly better use of short ones!  I'm actually interested in the topic of short crypto keys, because key size is an important performance factor in ~Tahoe-LAFS, but AONT seems to be a worse way to strengthen keys that the normal techniques.  The most interesting part of the article is the part in "Discussion" where he compares AONT to OAEP, to secret-sharing schemes, and to authenticated encryption.  I feel like I ought to go back and ponder those relationships more.
* added to [[things to read]]:
** Fabian Kuhn and René Struik: [[Random Walks Revisited: Extensions of Pollard’s Rho Algorithm for Computing Multiple Discrete Logarithms|../../named/URI:CHK:wu2kjacvelvgzula6ojafumrru:ynj4xnlwfp35obni7ydxjcc6winzdyc4tlyj5e4ooh5prn5oh2ca:3:10:195703/Random_Walks_Revisited.pdf]] (like a lot of entries in [[things to read]], I've already had a good long look at this one, but I don't yet feel that I understand it well enough to move it into [[things I have read]].)
(Also I often re-read things.  The ~All-Or-Nothing Transform, for example, I read or tried to read in around the year 2000, when it was new and when George Danezis wanted to use it for something in ~MixMinion.)

~Tahoe-LAFS v1.5.0 is done, except that the Debian packages aren't hosted properly in the http://allmydata.org apt repositories and the install process on Windows isn't nicely documented.  Can I wait until these things are fixed before announcing the existence of ~Tahoe-LAFS v1.5.0?  Argh -- we've been waiting for this release for so long!  Here is my post to the tahoe-dev mailing list: //[[v1.5.0 -- now with less apt-get!|http://allmydata.org/pipermail/tahoe-dev/2009-July/002473.html]]//

Wow!  Cryptographers have devised even more effective ways to crack a weakened variant of ~AES-256: [[Schneier's blog entry|http://www.schneier.com/blog/archives/2009/07/another_new_aes.html]]. This doesn't mean that anyone who is current relying on AES is vulnerable, but it does increase the likelihood that in the future someone will come up with a way to crack the full-strength AES.  This means that for long-term storage (such as in [[the Tahoe-LAFS storage system|http://allmydata.org]]), it might be better to encrypt with a stronger cipher such as Salsa20 (actually ~XSalsa20, which is just Salsa20 with a larger initialization vector) or, as Bruce Schneier suggests, AES with extra rounds.

It is ironic that ~AES-256 is the only algorithm approved for TOP SECRET by the U.S. government (~AES-128 is approved for SECRET but not for TOP SECRET). ~AES-256 has now been revealed as being weaker than ~AES-128. The other issue is that large-scale practical quantum computers (if they existed) could crack any cipher with a mere 128-bit key, but not a good cipher with a 256-bit key. This might mean that ~AES-256 would be vulnerable if there were a sufficiently powerful quantum computer.

That would mean there is now no encryption algorithm which is both secure against quantum computers and approved by the U.S. government for TOP SECRET.

I was recently pondering whether the next iteration of ~Tahoe-LAFS should switch from ~AES-256 to ~XSalsa20.  The benefits I was considering were that ~XSalsa20 is probably more secure than ~AES-256 (see [[the Tahoe-LAFS Bibliography|http://allmydata.org/trac/tahoe/wiki/Bibliography]], especially the practical issue of side-channel attacks) and is certainly much faster.  The drawbacks were that ~XSalsa20 is newer and less widely studied and that it wasn't approved for U.S. government usage.  This new attack on ~AES-256 makes my dilemma all the more pointed.

Rabbits rabbits!

Off to breakfast at the South Side Walnut.  If you're in the neighborhood, come around.

We finally released [[Tahoe-LAFS v1.5.0|http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=20090802031003-92b7f-289b017e81538b436d2aff2586888a91a0e67598]]!  ~Tahoe-LAFS is now a solid, mature tool.  I'm proud of it.

* added to [[things I have read]]:
** Alex Biryukov, Orr Dunkelman, Nathan Keller, Dmitry Khovratovich, Adi Shamir: [[Key Recovery Attacks of Practical Complexity on AES Variants With Up To 10 Rounds|http://eprint.iacr.org/2009/374]]
** Christopher Soghoian: [[Manipulation and abuse of the consumer credit reporting agencies|http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2583/2246]]

I've been posting to twitter recently.  You can follow along here: http://twitter.com/zooko .  Here are some of my recent tweets:
* And, just to be clear, this makes me happy, because I think it is a good thing that this law is being widely violated. http://bit.ly/jYA2G
* Free Software/Open Source hackers now routinely ignore US crypto export laws. Good! http://bit.ly/HgJ6E
* Heh heh... "This approach to cloud storage might be more appropriately described as 'crowd' storage." http://bit.ly/Lodg2
* .@aaroncordova Whoa! I hadn't seen that! ~Tahoe-LAFS reviewed on Ars Technica! http://bit.ly/Lodg2
* People: if there is a Free Software/Open Source developer in your life, and they put out a new release of their project, congratulate them.
* .@ioerror The word "darknet" suggests illegality, immorality, etc. That's not relevant to the topology, which is a friendnet topology.
* Don't call it a "darknet". My friends are moral people. http://bit.ly/Ofr7H

I wrote a post to the ~SHA-3 mailing list today:

[someone else wrote]
> The cost for people  to move away from ~MD5 has been negligible.
What?! Please tell me more about your observations!

As far as I know, the only people who successfully moved away from ~MD5 are the designers of crypto protocols such as SSH and SSL.  All the *other* uses of ~MD5 that I am aware of continued to use ~MD5 until there was a catastrophic demonstration of their failure (the ~RapidSSL root cert) or, in fact are still using ~MD5 (along with other algorithms) today (the NIST National Software Reference Library).  (See my notes from previous postings to this mailing list, appended, for further examples.)

The entire field of decentralized revision control tools would be using ~MD5 today (since the first version of git was a clone of ~BitKeeper and used ~MD5) but for the lucky fact that the second version of git was a clone of Monotone and used ~SHA1, so now all decentralized revision control tools use ~SHA1.  When Linus Torvalds and company were told that ~SHA1 was vulnerable to collisions, they retorted that such collisions couldn't be used to exploit their system, thus putting them in approximately the same position that ~RapidSSL was in a few years ago.

My guess is that there are still a lot of people out there who are going to continue to rely on ~MD5 and ~SHA1 until they really get burned.

Regards,

Zooko O'Whielacronx

begin appended message re-post
From: zooko <zooko@zooko.com>
Date: Monday, December,2008-12-22 15:05:40 0 MST
To: Multiple recipients of list <hash-forum@nist.gov>
Subject: will ~SHA-3 replace the current standard secure hash algorithm -- ~MD5?


Folks:

Below, I re-post a letter that I wrote to this list last February.  I think that this letter, which some of you may not have seen, casts light on why we have different assumptions about valid security/performance trade-offs.  It is because secure hashes are used today for more than just their original purpose.

Since I wrote this letter, I did some more snooping around, and I learned that the situation is even more extreme than I thought -- for some areas of endeavour, ~MD5 is actually the standard secure hash algorithm in 2008.

I chatted with a couple of friends who are information security consultants -- they get paid big bucks by household-name corporations to audit source code and systems for security flaws.  I asked them what kinds of secure hash functions they see used in the wild.  They answered that ~MD5 was the most common, occasionally ~SHA-1, in large part because it is a default value on the Java Cryptography Extensions, and they have never seen any other secure hash functions in client systems.

I chatted with a friend who works at the Internet Archive -- all files stored at the Internet Archive are identified by their ~MD5 hashes.

I noticed that there was a new release of the Haskell compiler GHC.  One of the new features is that it uses ~MD5 to identify code modules.

I learned more about the "computer forensics" field.  ~MD5 appears to be the standard mechanism to identify files in that field.  I read discussion forums in which computer forensics practitioners asked each other whether the cryptographic attacks on ~MD5 that they had heard about meant that they needed to change their practice.  The consensus seemed to be that they could continue using ~MD5 for now.

Finally, I was intrigued to see that NIST, of all organizations, uses and recommends the use of ~MD5 (in addition to ~SHA-1), as part of its "National Software Reference Library", which supports digital forensics.  This document explaining why NIST believes that this is safe is fascinating:

http://www.nsrl.nist.gov/Documents/analysis/draft-060530.pdf


The wide gap between the performance needs of using a secure hash function for public key cryptography versus using it for bulk data identification and integrity checking (which is what I use it for at my day job), make me wonder if ~SHA-3 should include variants or officially recommended tuning parameters so that people identifying large files can use a ~SHA-3 which is at least as fast as ~MD5 or Tiger, while people who are signing thousand-year documents can use a ~SHA-3 which is more expensive but safer.  (By the way, I tend to think that HMAC shouldn't be weighted heavily as a use case for ~SHA-3 simply because people should stop using HMAC and start using ~Carter-Wegman ~MACs instead such as Poly1305 or VMAC.)


Regards,

Zooko O'Whielacronx

begin appended message re-post
From: zooko
Date: February 5, 2008 12:15:53 PM MST
To: Multiple recipients of list
Subject: bulk data use cases -- ~SHA-256 is too slow
Folks:

Cryptographic hash functions were invented for hashing small variable-length strings, such as human-readable text documents, public keys, or certificates, into tiny fixed-length strings in order to sign them. When considering such usage, the inputs to the hash function are short -- often only hundreds or thousands of bytes, rarely as much as a million bytes. Also, the computational cost of the hash function is likely to be swamped by the computational cost of the public key operation.

Later, hash functions were pressed into service in ~MACs as exemplified by HMAC. In that usage, the inputs to the hash function tend to be small -- typically hundreds of bytes in a network packet. Also, the network is often the limiting factor on performance, in which case the time to compute the MAC is not the performance bottleneck.

I would like to draw your attention to another way that cryptographic hash functions have been pressed into service -- as core security mechanisms in a myriad of bulk data systems. Examples include local filesystems (e.g. ZFS [1]), decentralized filesystems (e.g. a project that I hack on: allmydata.org [2]), p2p file-sharing tools (e.g. ~BitTorrent [3], Bitzi [4]), decentralized revision control tools (e.g. monotone [5], git [6], mercurial [7], darcs [8]), intrusion detection systems (e.g. Samhain [9]), and software package tools (e.g. Microsoft CLR strong names [10], Python setuptools [11], Debian control files [12], Ubuntu system-integrity-check [13]).

Commonly in this third category of uses the size of the data being hashed can be large -- millions, billions or even trillions of bytes at once -- and there is no public key operation or network delay to hide the cost of the hash function. The hash function typically sits squarely on the critical path of certain operations, and the speed of the hash function is the limiting factor for the speed of those operations.

Something else common about these applications are that the designers are cryptographically unsophisticated, compared to designers in the earlier two use cases. It is not uncommon within those communities for the designers to believe that hash collisions are not a problem as long as second pre-image attacks are impossible, or to believe that the natural redundancy and structure of their formats protect them ("only meaningless files can have hash collisions", they say).

A consequence of these conditions is that raw speed of a hash function is very important for adoption in these systems. If you browse the references I've given above, you'll find that ~SHA-1, Tiger, and ~MD5 (!!) are commonly used, and ~SHA-256 is rare. In fact, of all the examples listed above, ~SHA-256 is used only in my own project -- allmydata.org. It is available in ZFS, but it is never turned on because it is too slow compared to the alternative non-cryptographic checksum.

I should emphasize that this is not just a matter of legacy -- it is not just that these older hash functions have been "grandfathered in". Legacy is certainly a very important part of it, but newly designed and deployed systems often use ~SHA-1. Linus Torvalds chose to use ~SHA-1 in his newly designed "git" decentralized revision control tool, *after* the original 2005-02-15 Wang et al. attack was announced, and roundly mocked people who suggested that he choose a more secure alternative [6]. I recently plead with the developers of the "darcs" revision control tool that they should not use ~SHA-1 for their new, backwards-incompatible design. (The issue currently hangs on whether I can find a sufficiently fast implementation of ~SHA-256 or Tiger with Haskell bindings.)

Because of my exposure to these systems, I was surprised to see a few comments recently on this mailing list that ~SHA-256 is fast enough. My surprise abated when I decided that the commentors are coming from a background where the first two use cases -- public key signatures and ~MACs -- are common, and they may not be aware that ~SHA-256 is potentially too slow for some other use cases.

Regards,

Zooko O'Whielacronx

[1] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Tuning_ZFS_Checksums
[2] http://allmydata.org
[3] http://en.wikipedia.org/wiki/BitTorrent_%28protocol%29
[4] http://bitzi.com/developer/bitprint
[5] http://www.venge.net/mtn-wiki/FutureCryptography
[6] http://www.gelato.unsw.edu.au/archives/git/0506/5299.html
[7] http://www.selenic.com/pipermail/mercurial/2005-August/003832.html
[8] http://www.nabble.com/announcing-darcs-2.0.0pre3-tt15027931.html#a15048993
[9] http://la-samhna.de/samhain/manual/hash-function.html
[10] http://blogs.msdn.com/shawnfa/archive/2005/02/28/382027.aspx
[11] http://peak.telecommunity.com/DevCenter/setuptools
[12] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Files
[13] https://wiki.ubuntu.com/IntegrityCheck

I met Robert Scoble.  I'd been intending to talk his ear off about [[Tahoe-LAFS|http://allmydata.org]] in order to get publicity with [[his 97,792 twitter followers|http://twitter.com/scobleizer]], but instead we got drunk with Rob L, Rocky, Lucretia, Jed, Tiffany, Kevin, and Julie and chatted loudly about a variety of things until Tiffany led the diehards off to another bar/club and I realized that I was sober enough to drive home.  It feels like it was good for my brain to loosen up.  I'm usually always wound up tight around some problem or another.

The rooftop patio of The Foundry had a beautiful view of The Flatirons and beautiful weather.  I learned that Jed and his family moved to Boulder, Colorado from Australia after considering New York City and Silicon Valley.  They were looking for good places to live while founding a startup company.  Their children go to Bear Creek Elementary.  Rob L was a bit of an ornery drunk -- the most exciting moment of the evening was when he tried to set Rocky's beard on fire with his lighter.  He also urged me to apply (reapply) for a job at his company, Rackspace, even if I still refuse to move my family out of Boulder, Colorado.  Let's see if he remembers when he sobers up.

So, I didn't elicit massive publicity for ~Tahoe-LAFS.  Oh well -- I'm not sure Scoble-level publicity is what ~Tahoe-LAFS needs right now anyway.  We've recently gotten a burst of activity -- people contributing to the project itself or building new things on top of it.  That makes me happy.  My desire would be to have a community of hackers large enough to be self-sustaining and small enough to be comfortable.  But I know that I don't really have control over that.

* added to [[things I have read]]:
** Hugo Krawczyk: [[On Extract-then-Expand Key Derivation Functions and an HMAC-based KDF|http://webee.technion.ac.il/~hugo/kdf/]]
This looks like a good candidate for the KDF to use in the next version of ~Tahoe-LAFS caps.  It proposes a two-step design for ~KDFs called "~Extract-then-Expand", and it proposes a (heuristically) stronger expand step than the current KDF tradition.  Also it proposes using HMAC everywhere that it possibly can.  ;-)
We don't actually need the extraction step, because the inputs to our ~KDFs are already high-quality unguessable keys.  However: it doesn't hurt, the extraction step is where the salt goes, and we'll be able to tell people "We just use {{{HKDF-sha-2-512t256-sha-2-256}}}." in order to explain our key-derivation function.  (This is assuming that HKDF becomes widely used, studied, and standardized.)
I confused at first because at the beginning of the paper (page 1) it says that HKDF's expansion step looks like this:
{{{
KM = PRF(K, “1” || info) || PRF(K, “2” || info) ||  . . . || PRF(K, “t” || info) 
}}}
Which is not that different from "the traditional approach" that the author is arguing against.  However, he says that this is "a rough sketch", and later explains that his expansion step is ''really'' this:
{{{L}}} is bits to produce, {{{k}}} is size of the output of the hash function in bits, {{{t = L/k}}} and {{{d = L mod k}}}:
{{{
PRF∗ (PRK, x, L) = K(1) || K(2) || . . . || K(t) : d 
}}}
where
{{{
K(1) = PRF(PRK, 0 : k || x || 0 : 32), 
K(i + 1) = PRF(PRK, K(i) || x || i : 32), 1 ≤ i < t
}}}
* added to [[things to read]]:
** ~Jean-Sébastien Coron, Thomas Icart: [[A Random Oracle into Elliptic Curves|http://eprint.iacr.org/2009/340]]
** Thomas Icart: [[How to Hash into Elliptic Curves|http://eprint.iacr.org/2009/226]]
Perhaps these ideas will help build a strong and/or provably secure implementation of [[semi-private keys|http://www.mail-archive.com/cryptography@metzdowd.com/msg10660.html]].

I posted about this to tahoe-dev: [[proposal for the Key-Derivation Function for the next generation of caps|http://allmydata.org/pipermail/tahoe-dev/2009-August/002598.html]].

I've been commuting to work on the bus.  It's only a few minutes, plus a few minutes of waiting for the bus (or thirty minutes waiting if I miss the one I'm aiming to catch), but even though it is such a short ride, I love commuting on public transport because then I get regular, uninterrupted time to read.  There's no wifi on these buses, and I don't have a cellphone.  Nothing to do but chat with strangers and read.  My reading has been outstripping my updating of the [[things I have read]] tiddler on this klog.  :-)
* added to [[things I have read]]:
** ~David-Sarah Hopwood: //[[Jacaranda programming language specification, draft v0.42|http://jacaranda.org/jacaranda-spec-0.44.txt]]//; Exciting!  A secure subset of ~JavaScript, which is also potentially more elegant and useful than the whole hog, and which can be simply and quickly verified. ~David-Sarah's rationales for the design decisions sound compelling.  Their writing is excellent.  I'm eager to try this language out.  The only problem is that there is no implementation yet.  You can check [[David-Sarah's blog|http://davidsarah.livejournal.com]] to see if they've announced any new implementation.  Here are [[some slides from a presentation about Jacaranda|http://ses.json.org/Jacaranda.ppt]].
** [[Doug Crockford|http://www.crockford.com]]: //[[JavaScript: The Good Parts|http://oreilly.com/catalog/9780596517748/]]//; This book complements the Jacaranda spec.  Doug Crockford's writing is a pleasure to read: precise, concise, and witty.  His ideas are often excellent.  I'm eager to try out the languages, such as [[cajita|http://code.google.com/p/google-caja/wiki/SubsetRelationships]] which have descended from the ideas he expresses in this book.
* added to ''currently reading'':
** //[[The Omnivore's Dilemma|http://www.michaelpollan.com/omnivore.php]]// by Michael Pollan

* added to ''currently reading'':
** [[Cryptographic Hash Function Basics: Definitions, Implications and Separations for Preimage Resistance, Second-Preimage Resistance, and Collision Resistance|http://eprint.iacr.org/2004/035]] by Phillip Rogaway and Thomas Shrimpton
Ever since I started my new job (a little more than a month ago -- the beginning of July), I've had the feeling that there's been something missing from my life.  Today I brought headphones into the office, and now I know what it was: music while hacking!  Now I'm listening to Frank Zappa [["You Are What You Is"|http://en.wikipedia.org/wiki/You_Are_What_You_Is]] and feeling much better.
* added to [[things to read]]:
** Emily Yoffe: [[Seeking: How the brain hard-wires us to love Google, Twitter, and texting. And why that's dangerous.|http://slate.com/toolbar.aspx?action=print&id=2224932]]
** NIH "Research Matters" newsletter: [[Low-Fat Diet May Cut Ovarian Cancer Risk|http://www.nih.gov/news/research_matters/october2007/10222007diet.htm]]
** Dhananjay Nene: [[CRUD is not only good for, but is the only consistent way to build REST over HTTP |http://blog.dhananjaynene.com/2009/08/crud-is-not-only-good-for-but-is-the-only-consistent-way-to-build-rest-over-http/]]
** Mauricio Fernández: [[Introducing extprot: extensible binary protocols for cross-language communication and long-term serialization|http://eigenclass.org/R2/writings/extprot-extensible-protocols-intro]] (recommended by Paul Snively)

Hooray, [[my pycryptopp software|http://allmydata.org/trac/pycryptopp]] is now [[included in Debian|http://packages.debian.org/sid/python-pycryptopp]]!  I'm proud.  Also, this is a big step toward getting [[Tahoe-LAFS|http://allmydata.org]] included in Debian.
* added to [[things to read]]:
** Daniel R. L. Brown and Robert P. Gallant: [[The Static Diffie-Hellman Problem|http://eprint.iacr.org/2004/306]] (recommended by Dcoder)
** Aniket Kate and Ian Goldberg: [[Asynchronous Distributed Private-Key Generators for Identity-Based Cryptography|http://eprint.iacr.org/2009/355]]
** Dimitrios Poulakis: [[Small Solutions of Bivariant Modular Equations and the security of DSA and ECDSA|http://eprint.iacr.org/2009/363]]
** Nishanth Chandran and Vipul Goyal and Ryan Moriarty and Rafail Ostrovsky: [[Position Based Cryptography|http://eprint.iacr.org/2009/364]]
** Abhishek Parakh and Subhash Kak: [[Space Efficient Secret Sharing: A Recursive Approach|http://eprint.iacr.org/2009/365]]
** Boris Skoric: [[Quantum readout of Physical Unclonable Functions: Remote authentication without trusted readers and authenticated Quantum Key Exchange without initial shared secrets|http://eprint.iacr.org/2009/369]]
** Rosario Gennaro and Shai Halevi: [[More on Key Wrapping|http://eprint.iacr.org/2009/372]]
** Masao KASAHARA: [[Forgotten Secret Recovering Scheme and Fuzzy Vault Scheme Constructed Based on Systematic Error-Correcting Codes|http://eprint.iacr.org/2009/375]]
** Manoj Kumar: [[A Registration Scheme to Allocate a Unique Identification Number|http://eprint.iacr.org/2009/383]]
** Joppe W. Bos and Marcelo E. Kaihara and Thorsten Kleinjung and Arjen K. Lenstra and Peter L. Montgomery: [[On the Security of 1024-bit RSA and 160-bit Elliptic Curve Cryptography|http://eprint.iacr.org/2009/389]]
** Qian Wang and Cong Wang and Jin Li and Kui Ren and Wenjing Lou: [[Enabling Public Verifiability and Data Dynamics for Storage Security|http://eprint.iacr.org/2009/281]]
** Ueli Maurer and Stefano Tessaro: [[Computational Indistinguishability Amplification: Tight Product Theorems for System Composition|http://eprint.iacr.org/2009/396]]
** Francesco Davì and Stefan Dziembowski: [[Leakage-Resilient Storage|http://eprint.iacr.org/2009/399]]
(No, there's no way I'm going to read, much less grok, all of those.  The life so short, the craft so long to learn.)

* added to [[things to read]]:
** Brian Baldwin and Andrew Byrne and Mark Hamilton and Neil Hanley and Robert P. ~McEvoy and Weibo Pan and William P. Marnane: [[FPGA Implementations of SHA-3 Candidates:CubeHash, Grøstl, LANE, Shabal and Spectral Hash|http://eprint.iacr.org/2009/342]]
** Alexandra Boldyreva and Serge Fehr and Adam O'Neill: [[On Notions of Security for Deterministic Encryption, and Efficient Constructions without Random Oracles|http://eprint.iacr.org/2008/352]] (it is directly relevant to an issue that we addressed in ~Tahoe-LAFS: [[convergent encryption reconsidered|http://hacktahoe.org/drew_perttula.html]])
** Algirdas Avizienis, ~Jean-Claude Laprie, Brian Randell, and Carl Landwehr: [[Basic Concepts and Taxonomy of Dependable and Secure Computing|http://www.cs.umass.edu/~ransford/srg/papers/avizienis--dependable-secure.pdf]] (recommended by Ludovic Courtès)
My wife encouragingly tells me that I ''could'' read and grok all these papers in one semester, if I had a couple of study buddies to meet and discuss with once a week.

* [[Tahoe-LAFS is like encrypted git|http://allmydata.org/pipermail/tahoe-dev/2009-August/002656.html]]
* Is there something better than ECDSA (the Elliptic Curve based Digital Signature Algorithm standard)? [[Paul Crowley writes|http://allmydata.org/pipermail/tahoe-dev/2009-August/002661.html]] that other digital signature algorithms come with proofs that they are at least a secure as ECDSA (and maybe more secure).  [[I reply|http://allmydata.org/pipermail/tahoe-dev/2009-August/002665.html]] that ~Tahoe-LAFS has unusual performance requirements.  [[Jack Lloyd argues|http://allmydata.org/pipermail/tahoe-dev/2009-August/002666.html]] that a proof is not necessarily worth the parchment it is inscribed on.

Something I love about the computer security industry is how it has a surfeit of "fail".

I was just trying to submit a proposal to speak at [[The RSA Conference|http://www.rsaconference.com/2010/usa/index.htm]].  The submission deadline was fast approaching, and whenever I clicked the "Submit" button I got an error message:
{{{
Special characters such as < and > and system commands are not allowed in your submission for security reasons.
Please check your submission and try again.
}}}
So by careful trial and error I eventually figured out which line of my submission triggered this:
{{{
high-level language such as Python, Ruby, or JavaScript. Finally, we demonstrate the
}}}
The answer is, I can't say "~JavaScript" in my submission.  :-)

added to ''currently reading'':
* //[[The Steel Remains|http://books.google.com/books?id=AsRF3s1MK08C&dq=the+steel+remains&printsec=frontcover&source=bl&ots=keAI_8SMcl&sig=DKVlGa63-Q0r9pI5PLsQKGNQnLI&hl=en&ei=mrOSSs2xAonuswOUo4wM&sa=X&oi=book_result&ct=result&resnum=8#v=onepage&q=&f=false]]// by Richard K. Morgan, a gift from Tom Christiansen.  //The Steel Remains// is by noted new science fiction author Richard Morgan, but it is a fantasy novel about swords, monsters, and sorcery.  Or is it?  I started reading it last night as a way to escape from reality and not have to go to sleep, and I ended up staying awake til midnight and read about 150 pages.  On the bus to work this morning I was reading it and a woman named Una who works at the same company asked what I was reading.  I showed it to her and she hadn't heard of it, so I explained "It is a science-fiction/fantasy/gay-erotica novel!".  She was amused and/or embarassed.

* [[I aint afraid of no patents]]
* [[not currently following on twitter]]
* [[book review: The Steel Remains]]

I opened a ticket asking the Ubuntu folks to include [[Tahoe-LAFS|http://allmydata.org]] in the upcoming [[Ubuntu 9.10|https://wiki.ubuntu.com/KarmicKoala]] (codenamed Karmic Koala): [[Feature Freeze Exception: Tahoe-LAFS|https://bugs.launchpad.net/ubuntu/+bug/421802]].  Karmic has several other features intended to enhance Cloud Computing, so ~Tahoe-LAFS is a good fit for Karmic.
* added to [[things I have read]]:
** Apple Computer: [[Grand Central Dispatch: A better way to do multicore|http://images.apple.com/macosx/technology/docs/GrandCentral_TB_brief_20090608.pdf]]; Good engineering. I guess in the future applications really will work better on multicore Macintoshes.  I wonder if other operating systems and programming languages will be allowed copy the Grand Central Dispatch API and runtime so that other systems will gain the same benefits without having to write platform-specific tweaks to the applications, or whether there is some legal barrier to that.
** Steven R. Hetzler: [[The Storage Chasm: Implications for the Future of HDD and Solid State Storage|http://www.caiss.org/docs/DinnerSeminar/TheStorageChasm20090205.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/08/28]]); If I understand these slides correctly, it says that the current paradigm of SATA drives plus DRAM is likely to remain the market leading storage architecture for the forseeable future.  Wes Felter sounds disappointed that the technologically superior solid state disk isn't likely to take off, and I can sympathize -- I like the idea of storage which is faster and more robust than hard drive while being non-volatile.  On the other hand, [[Tahoe-LAFS|http://allmydata.org]] is engineered to make good use of super cost-efficient commodity SATA drives and DRAM, so I'm happy to imagine that those will continue to be dominant.
* added to [[things to read]]:
** Gilles Dubochet: [[Computer Code as a Medium for Human Communication: Are Programming Languages Improving?|http://infoscience.epfl.ch/record/138586/files/dubochet2009coco.pdf]] (as recommended by Bryan O'Sullivan); At last!  A real psychology paper studying programming languages!  Such research is about thirty years overdue.

* added to [[things to read]]:
** Tim Nufire of Backblaze: [[Petabytes on a budget: How to build cheap cloud storage|http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/]] (blog entry)

* ~WHOO-HOO!  [[Tahoe-LAFS|http://allmydata.org]] will be included in the next release of Ubuntu -- Karmic Koala, which will be released on October 29.  Here is [[the ticket|https://bugs.launchpad.net/ubuntu/+bug/421802]] showing that ~Tahoe-LAFS is accepted into Ubuntu.
Thanks to Brian Warner, Iulian Udrea, Piotr Ożarowski, Dustin Kirkland, Scott Kitterman, Nicola Ferralis, Fernando Ribeiro, Stephan Peijnick, Siegfried Gevatter, Artur Rona, Duncan ~McGreggor, Micah Anderson, and Stefan Potyra.  Wow, that's a lot of people who helped out -- Debian/Ubuntu packaging is a highly collaborative effort.
* I posted a message about ~Tahoe-LAFS's crypto-capability design: [[Key Management Without Keys|http://allmydata.org/pipermail/tahoe-dev/2009-September/002783.html]].
* added to [[things to read]]:
** Tom Knox: [[Do these mysterious stones mark the site of the Garden of Eden?|http://www.dailymail.co.uk/sciencetech/article-1157784/Do-mysterious-stones-mark-site-Garden-Eden.html]] (as recommended by Kris Nuttycombe)

* added to [[things to read]]:
** Michael Nielsen: [[Doing science in the open|http://physicsworld.com/cws/article/print/38904]]
** Peter Seibel: [[Coders at Work|http://codersatwork.com]]
** J. Lacan, V. Roca, J. Peltotalo, and S. Peltotalo: [[RFC 5510: Reed-Solomon Forward Error Correction (FEC) Schemes|http://tools.ietf.org/html/rfc5510]]

* added to [[things to read]]:
** John Timmer, writing in arstechnica.com: [[Anti-"publication bias" efforts not panning out for science|http://arstechnica.com/science/news/2009/09/for-clinical-trials-design-and-results-dont-always-match.ars]]
** Natalie Angier, writing in The New York Times: [[Skipping Spouse to Spouse Isn’t Just a Man’s Game|../../file/URI%3ACHK%3Ala6cyuvjlmk7jf4e42ywvakfxe%3Acdol7iz3bytmglqtympwrre6jyrbgqowyzj5piogl56jzvqrsxoq%3A3%3A10%3A21737/@@named=/01angi.html]]
* currently reading:
** a draft of the next edition of //[[Practical Cryptography|http://www.schneier.com/book-practical.html]]// by Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno

* posted [[how to encrypt and integrity-check with only one value|http://allmydata.org/pipermail/tahoe-dev/2009-September/002796.html]] (it comes with a pretty picture -- click on picture for full-size version)
[img[How to encrypt and integrity-check with only one value|/uri/URI:CHK:zunvg56akpjs2rtwdwmfjuhchu:jvkkhx57l6qk47glmdmsrulquniunzu4bfybyuq246ai77xnpx7q:3:10:28292][/file/URI:CHK:j3mnnlj7ttcumz6m46gwpa766y:56plr5eqlybe4gzcowxhbx2tztscrlibi7owlnivvylbm6crsuiq:3:10:24311/@@named=/imm-short-readcap-simple-drawing.svg]]
* added to [[things to read]]:
** Nik Cubrilovic (in ~TechCrunch): [[The Anatomy of the Twitter Attack|/file/URI:CHK:nm72blax6oqt3fui3dnrhahszq:wcpjaneyqzf4bw752izfey44abql6ywync2vweejsmnohyiwkkia:3:10:275196/@@named=/anatomy_of_the_twitter_attack.html]]
* added to [[things I have read]]:
** Daniel J. Bernstein: [[Understanding brute force|http://cr.yp.to/snuffle/bruteforce-20050425.pdf]] (short answer: use 256-bit keys! Too bad that following this advice would make ~Tahoe-LAFS read-caps unwieldy.)

[[Ideas for future versions of Tahoe-LAFS caps|http://allmydata.org/pipermail/tahoe-dev/2009-September/002812.html]] (in response to Kevin Easton's queries about how ~Tahoe-LAFS caps work)

We might be able to make file capabilities that are as short as this example: //''<nowiki>lafs:R_3AeIEJs52QXFMGYqalmuxSYx0</nowiki>''// .  Here's my [[recent post to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-September/002829.html]] explaining what the underlying crypto schemes would be.

//''Mom:'' I want people to use links which are secure, in the sense that if you send me a link nobody else can redirect me to a different page than the one you intended.  The current research is all about making those links ''short enough'' so that people will use them.  The current links look something like this: //''<nowiki>http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa</nowiki>''// and I'm hoping to squeeze them down to something like this: //''<nowiki>http://testgrid.allmydata.org:3567/uri/R_3AeIEJs52QXFMGYqalmuxSYx0</nowiki>''// or //''<nowiki>http://tahoelafs.org/c/R_3AeIEJs52QXFMGYqalmuxSYx0</nowiki>''// or even eventually //''<nowiki>lafs:R_3AeIEJs52QXFMGYqalmuxSYx0</nowiki>''// .  My hypothesis is that the shorter they are the more likely you would use them to share files with friends and family.  What do you think?//

Thinking about small crypto caps and secure "streaming upload" for ~Tahoe-LAFS: [[mailing list thread|http://allmydata.org/pipermail/tahoe-dev/2009-September/002842.html]] on tahoe-dev.

Great computer scientist Mark Miller is currently engaged in working out [[how to improve the security of the Web|http://www.eros-os.org/pipermail/cap-talk/2009-June/012926.html]] with (possible future great computer scientist) Adam Barth.  Mark Miller referred to an [[essay that I wrote years ago|http://www.eros-os.org/pipermail/cap-talk/2004-May/001814.html]].  That makes me feel good.  I hope my idea helps.

In computer science, we are privileged to rub elbows with the giants on whose shoulders we stand.

(A quote I got from Ping Yee, which I guess was derived from //"In the sciences, we are now uniquely privileged to sit side by side with the giants on whose shoulders we stand."// -- Gerald Holton.)

I read the text of a speech that Ian Bicking gave: [[Toward a new self-definition for open source|http://blog.ianbicking.org/2009/09/10/a-new-self-definition-for-foss]], and I as read it I became excited and emotional.  He's onto something.  There are big changes in the air.  The Hacker movement, the Free Software movement, the Open Source movement, these have always been tiny movements.  Our accomplishments -- even our proudest ones -- are really only a modest contribution to humanity.  The scale and scope of humanity's values -- its loves and its arts and crafts and sciences -- dwarf all the contributions of the hackers.
But, as Ian Bicking points out, there is something special going on here, and it could take root in other fields and mutate and grow.  Perhaps the wonders in humanity's future will outshine even the magnificent accomplishments of today, and if so, this may be partially due to an outgrowth of the Hacker movement.
The part of Ian's speech that I might disagree with is the word "new" in the title -- the spirit behind his words is much like the one that he inherited from the GNU project (as he properly credits) and the Hacker Ethic.  However, it might be new to many in his audience of Django professionals.  So perhaps the Open Source movement, originally designed as a cloak or a dampening of the Free Software spirit, and having reached and influenced a wider group of engineers and businesspeople, wikipedians and netizens, is ready to rekindle.

The ~Tahoe-LAFS Test Grid -- the grid that hosts the blog you are now reading -- is having problems.  It is composed mostly of servers owned by allmydata.com and offered as a free service to ~Tahoe-LAFS hackers to test out their experiments.  Also various people connect their ~Tahoe-LAFS nodes to the Test Grid while doing experiments.  There is a publicly writable directory linked from the front page of http://allmydata.org .  There is also a page of measurements of the test grid: http://allmydata.org/trac/tahoe/wiki/TestGrid .

The Test Grid really isn't the ideal grid to host a blog.  Allmydata.com explicitly stated when they set it up that they might delete the data or take away the storage servers at any time.  It was strictly for testing.

I've been persisting in hosting my blog on the ~Tahoe-LAFS Test Grid partly as an experiment -- can a grid that is completely unmaintained (nobody from allmydata.com has looked at any of the donated servers in many months) and includes transient servers from anonymous people still be stable enough to use for my blog?  The answer is "barely", and I'm probably going to end the experiment soon and move my blog to a reliable grid such as The Volunteer Grid, which is composed of servers owned and operated by contributors to the ~Tahoe-LAFS project.  :-)

Elliot's first soccer game was this morning.  It was great fun to watch and cheer for the kindergarteners.

My brother wants to set up a klog like this one.  :-)

Oh, contrary to what I said on [[2009-09-16]], I think I'll keep using the Test Grid to host my klog for now.  I found some interesting bugs in ~Tahoe-LAFS and/or Ubuntu ([[Tahoe-LAFS #806|http://allmydata.org/trac/tahoe/ticket/806]], [[Tahoe-LAFS #807|http://allmydata.org/trac/tahoe/ticket/807]]), but once I upgraded the ~Tahoe-LAFS storage servers on the Test Grid from an old version deployed almost a year ago to the current release, everything started performing much better.

Thanks to a comment on a [[darcs|http://darcs.net]] issue ticket, I discovered [[RFC3339|http://www.ietf.org/rfc/rfc3339.txt]].  All hail ~RFC3339!  Here are some good things about it:
* an unambiguous, international, simple date and time format
* a subset of ~ISO-8601
* IETF ~RFCs are more respected in some quarters than ISO standards, so hopefully ~RFC3339 will pull more hackers into using a common de facto standard (in particular, the existence of ~RFC3339 may encourage my programming partner Brian to start using this format in more places, since he is an Internet Engineer and respects ~RFCs)
* it slyly blesses the common cheat of writing pseudo-standard "2009-09-21 10:07:21" instead of the real standard "2009-09-21T10:07:21".  I personally always use this cheat, because the former is human-friendly and the latter isn't.  ~RFC3339 slyly blesses this hack by pointing out that while "2009-09-21 10:07:21" is not an standard ~ISO-8601 date-and-time stamp, it ''is'' a standard ~ISO-8601 date stamp followed by a standard ~ISO-8601 time stamp.  ;-)

~WHOO-HOO!  [[Tahoe-LAFS is in Ubuntu!]]

* added to [[things to read]]:
** Paul Tough (in The New York Times): [[Can the Right Kinds of Play Teach Self-Control?|../../uri/URI%3ADIR2%3An33p7lilkk4heaz2twmtmjjaz4%3Atdon63znadd5yu53kagt7d4xqbl466m4cnknjhmifdjbeim4jrsa/27tools-t.html]]

Rabbits, rabbits!  And Happy GNU ~MailMan Membership Reminder Day, everyone!

I finally started reading my printout of the [["Elk Point" distributed crypto-cap design|http://allmydata.org/pipermail/tahoe-dev/2009-September/002848.html]].  Great stuff!  It is going to lead to a faster and more efficient ~Tahoe-LAFS with more human-friendly caps and added features such as the ability to revoke your write-cap.

Also, it is great intellectual fun.  ~David-Sarah Hopwood, Brian Warner, Shawn Willden and Hal Finney are excellent engineer/cryptographers and I'm grateful that I get a chance to work with them on something as intellectually challenging and valuable as this.

In the coming days and weeks I'm going to prioritize writing up my contributions to that conversation.

I've learned that someone with whom I occasionally collaborate has [[ALS|http://www.alsa.org/als/what.cfm]].  This means he is very likely to die within two to five years, and he is likely to become increasingly crippled during that time. This reminds me that opportunities to work with such people are precious.

Good news!  Amyotrophic lateral sclerosis (a.k.a. Lou Gehrig's disease a.k.a. Stephen Hawking's disease) doesn't actually kill most people. What happens is that most victims actually choose to die by not going on an artificial respirator or by having theirs turned off.  Since Hal Finney is planning to choose to live, even if that means depending on an artificial respirator, then his life expectancy is much greater!  [[Dying Outside|http://lesswrong.com/lw/1ab/dying_outside/]], by Hal Finney.

Hal is the fellow that I mentioned on [[2009-10-01]] who was recently diagnosed with ALS.  Also, he has a contract with a cryonics company which will freeze his body in liquid nitrogen upon his death, with the hope that someday technology will advance to the point that he can be resurrected from that frozen state.  That's, as Hal puts it "a long shot", though, so I'm glad to learn that artificial respiration and modern user interfaces might prolong his life.  

[[hash function combiners]]

Is this "Hacker Dojo" comparable to ~NoiseBridge? San Jose Mercury News: [[Hacker Dojo in Mountain View sparks ideas and tinkering|http://www.mercurynews.com/breaking-news/ci_13573588?nclick_check=1]]

[[It's a Boy!]]
I have little G.D. cuddled up in my left arm.  Sometimes he asks to suck on my pinkie the way his big brothers used to do.  My hurt leg is propped up on the dining room table (I've now strained its patience thrice by dropping my crutches to pick up each of my three sons).  I'm exhausted!
So, shall I try to contribute to [[buildbot|http://buildbot.net]] [[working around|http://buildbot.net/trac/ticket/377]] [[this Windows brain damage|http://thesystemguard.com/TheGuardBook/CCS-Ext/helptext/Cmd-K3.txt.htm]] or study [[some really cool cryptography|hash function combiners]]?  I think I just convinced myself.  I'm going to bed with a printout of [[Robust Multi-Property Combiners for Hash Functions Revisited|http://homepages.cwi.nl/~pietrzak/publications/FLP08.pdf]].  (Interested in cryptography?  See also [[this message|http://www.ietf.org/mail-archive/web/cfrg/current/msg02651.html]] and [[that one|http://www.ietf.org/mail-archive/web/cfrg/current/msg02652.html]].)

[["If that happens, then I hope Tahoe-LAFS at least leaves a legacy of demonstrating that provider-independent security is possible."|http://allmydata.org/pipermail/tahoe-dev/2009-October/003045.html]]

Someone gets it!  Greg Wilson: [[Bits of Evidence|http://www.slideshare.net/gvwilson/bits-of-evidence-2338367]]
Is the time ripe for "evidence-based software engineering"?
(Ref: [["evidence based medicine"|http://en.wikipedia.org/wiki/Evidence-based_medicine]] which, you may be surprised to learn, was invented in 1990, and hasn't yet had much of an effect on the medical care that you get from your doctor.)

Whoo-hoo!  Brian Warner and I are going to present at the [[RSA Conference 2010|http://www.rsaconference.com/2010/usa/index.htm]] which is the biggest conference in the computer security industry.  Here's our abstract:

//Session Title:// ''~Tahoe-LAFS: Your Cloud Storage Provider Does Not Need Access To Your Data''
//Session Abstract:// Cloud storage offers cost-savings, scalability, and convenience. If you store your data in the cloud you have to rely on your cloud storage provider for the availability of your data. But do you also have to rely on them for its confidentiality and integrity? No, you do not. [[Tahoe-LAFS|http://allmydata.org]] is an open source storage platform that transparently encrypts and integrity-checks all data. The user gains "provider-independent security" without sacrificing cost-savings, scalability, or convenience.


//Speakers:// 
Zooko ~Wilcox-O'Hearn, 
Brian Warner
//Session Classification:// Advanced
Session
//Keywords:// application_security, cloud_computing, encryption_key_management

//Scheduled Date:// 2010-03-05

//Scheduled Time:// 10:10 AM - 11:00 AM


(I wonder if the conference organizers are hoping for an encore of [[Laptop Versus Axe]].)

I read Jonathan Ellis's article [[NoSQL Ecosystem|http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosystem]].  I am belatedly realizing that [[Tahoe-LAFS|http://allmydata.org]] is actually a key-value distributed database, like [[Project Voldemort|http://project-voldemort.com]] or [[Tokyo Cabinet|http://1978th.net/tokyocabinet]].  I assume that it is much slower than either of those two for some common work-loads, but on the other hand [[Aaron Cordova's presentation at HadoopWorld|http://www.cloudera.com/sites/all/themes/cloudera/static/hw09/3%20%20-%202-30%20Aaron%20Cordova,%20BAH,%20HadoopWorldComplete.pdf]] shows that ~MapReduce over ~Tahoe-LAFS is about as fast as ~MapReduce over HDFS for a read-intensive work-load which is, as he puts it "not bad".
~Tahoe-LAFS is probably much more reliable than most other distributed key-value stores, not only because of its erasure-coding architecture, which makes it so that you can shrug off even the catastrophic failure of many of your servers simultaneously, but also because we've carefully engineered ~Tahoe-LAFS for reliability. We have a strong tradition of test-driven development and careful handling of edge cases, and ~Tahoe-LAFS has been deployed at scale under monitoring for years.  Hundreds of users have entrusted millions of files totaling tens of TB of data, and so far it has never lost any data.  Not even that time that the cheap 1U ~SuperMicro server went haywire and destroyed all four of its 1 TB hard drives at once.
Unfortunately, it appears that the people who are interested in the modern "~NoSQL" databases have never heard of ~Tahoe-LAFS.  I suppose that is because it has been branded as a "secure distributed filesystem" instead of as a "distributed key-value store".  For example, [[the wikipedia page on NoSQL|http://en.wikipedia.org/wiki/NoSQL]] doesn't mention ~Tahoe-LAFS.  I wonder if the people who are currently using such tools would like using ~Tahoe-LAFS.
(By the way, this blog is a //decentralized web app//.  All of the computation -- such as when you use the search box or the tags -- is done on the client side in your browser using ~JavaScript.  All of the storage is done on a ~Tahoe-LAFS distributed storage grid. For more details see [[about this klog]].)

[[the business model of the web versus cloud computing]]

* added to [[things to read]]:
** Rohit Khare, Richard N. Taylor: [[Extending the REpresentational State Transfer (REST) Architectural Style for Decentralized Systems|http://www.ics.uci.edu/~rohit/ARRESTED-ICSE.pdf]] ([[indexed on CiteSeerX|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.8033]]) ([[copy hosted on Tahoe-LAFS decentralized secure storage grid|http://../../file/URI%3ACHK%3Alwd3qsbxzrnqkaabdrzunshuqe%3A6gipjscuop2r3hrcb47svxc3xehj6c2ptcaoplhflv7smn3vcx4q%3A3%3A10%3A4550069/@@named=/ARRESTED-ICSE.pdf]]) (as recommended [[by James Urquhart|http://twitter.com/jamesurquhart/status/5738731163]] and [[by Mike Kelly|http://twitter.com/mike__kelly/status/5689070113]])

* added to [[things I have read]]:
** Jen Lexmond, Richard Reeves: [[Building Character|../../file/URI%3ACHK%3Arev3ooozvxyxcm3ek56pxe5qza%3Amzwkmziexzwjyrly3p2wvzxu7hy6bxevwumitz32hem3ux3y4meq%3A3%3A10%3A749145/@@named=/Building_Character_Web.pdf]]; This is written to inform and influence policy of the government of the United Kingdom. The first half of it is a research survey about what factors influence the quality of the character of children, focussing on parenting styles.  Then there is a chapter advising the government of the U.K. on what policies it should adopt, then the second half is about the statistical techniques used to produce the first half.  I just read the first half.  I also read the introduction, which pleased me greatly by advocating an Aristotelian theory of what is character and why does it matter.
** Anil Dash: [[The Web in Danger|../../file/URI%3ACHK%3Aymeendyhlglzbqomu7c2ppo2re%3Azaigwylxh3gyt6lrfzhwm4n3v534nn6zl3z5hnefybklikopep5a%3A3%3A10%3A63751/@@named=/the-web-in-danger.html]]; Tim O'Reilly hits the nail on the head -- centralized services are natural monopolies. This is why I wrote in [[Decentralized Money]]: //A plethora of centralized services is not the same as a decentralized service.//  Tim O'Reilly rightly points out that a plethora of centralized services will naturally evolve to a situation in which one is dominant.

Hey this looks like it could be useful; added to [[things to read]]:
* Sean Eron Anderson: [[Bit Twiddling Hacks|http://graphics.stanford.edu/~seander/bithacks.html]] (mentioned by Brad Neuberg)

* added to [[things I have read]]:
** [[HAIL: A High-Availability and Integrity Layer for Cloud Storage]]
** Vadim Lyubashevsky, Adriana Palacio, Gil Segev: [[Public-Key Cryptographic Primitives Provably as Secure as Subset Sum|http://eprint.iacr.org/2009/576]]; Exciting! Cryptographers are getting close to practical, efficient public-key crypto which is proven to be as difficult as NP. This one also has the wonderful property of being simple enough that even I can partially understand it.
** Tom Knox: [[Do these mysterious stones mark the site of the Garden of Eden?|http://www.dailymail.co.uk/sciencetech/article-1157784/Do-mysterious-stones-mark-site-Garden-Eden.html]] (as recommended by Kris Nuttycombe); Huh, that was really interesting!  Too bad I don't trust the author of that story to know a lot of other relevant facts and to mention all the relevant facts that he knows.  Maybe I'll find a more detailed story about it by an archaeologist...

It turns out that Boulder, Colorado is an excellent place to have broken your anterior cruciate ligament. They have a great deal of expertise and practice and fancy equipment here all dedicated to the job of reconstructing anterior cruciate ligaments.

I read or re-read Eduardo Pinheiro, ~Wolf-Dietrich Weber, and Luiz André Barroso (all of Google Inc.): [[Failure Trends in a Large Disk Drive Population|http://labs.google.com/papers/disk_failures.pdf]]. Too bad they couldn't or wouldn't tell us what brands and models of drives failed at a rate of 1.7% per year and which failed at a rate of 8.6% per year.

 * Hey look!  [[I'm in TechCrunch.|http://www.techcrunch.com/2009/12/14/simple-geo-beta-keys]]  ~TechCrunch is, I think, the most important magazine/blog for the Internet Startup scene.  I'm really excited to be joining [[SimpleGeo|http://simplegeo.com]].  I can't even imagine all the great new things that location-based technology is going to bring to our lives.  ~SimpleGeo's business plan is to make easier for people to invent those things.  Awesome!
 * Here's an interesting story of how a well-known writer turns out to be a woman. James Chartrand: [[Why James Chartrand Wears Women’s Underpants|http://www.copyblogger.com/james-chartrand-underpants]]. Maybe all women starting businesses on the Internet should use male pseudonyms.

The [[p2p-hackers mailing list|http://lists.zooko.com/mailman/listinfo/p2p-hackers]] is experiencing technical difficulties.  We apologize for the inconvenience.  I will investigate

[[U.S. Drones transmit video in cleartext]]
[[Crypto in JavaScript]]

Duane "spacedoc" Graveline [[opines|http://www.spacedoc.net/cholesterol_relevant_heart_disease]] that the catastrophic results of a trial of a new cholesterol-fixing drug which induced 60% greater mortality in the test subjects and killed some thirty people and had to be abruptly cancelled (Torcetrapib [[1|http://en.wikipedia.org/wiki/Torcetrapib]], [[2|http://pipeline.corante.com/archives/2006/12/03/the_torcetrapib_catastrophe.php]], [[3|http://pipeline.corante.com/archives/2008/12/02/torcetrapib_what_was_the_problem_and_does_it_matter.php]]), casts doubt on the entire hypothesis that blood cholesterol levels cause heart disease.  If you've read [[Good Calories, Bad Calories]], this will come as no surprise. More specifically this trial casts doubt on whether HDL causes healthier hearts. (I got the link to this from [[Mike Eades|http://www.proteinpower.com/drmike/]]'s twitter stream.)

added to [[things I have read]]:
* Chris Williams writing for The Register: [[Software fraudster 'fooled CIA' into terror alert|http://www.theregister.co.uk/2009/12/24/cia_montgomery/]]
Wow!  This guy apparently conned the CIA and the Department of Homeland Security out of a lot of money and prompted them to issue nationwide terror alerts and evacuate 5000 people from the Metropolitan Museum of Art by claiming that his video compression software could detect secret terrorist barcodes in Al Jazeera broadcasts.  Beautiful!  The Register story concludes with this valiant attempt to pass the buck:
> Frances Townsend, a homeland security adviser to Bush, said she did not regret having relied on Montgomery's mysterious intelligence. "It didn't seem beyond the realm of possibility. We were relying on technical people to tell us whether or not it was feasible," she said.

added to [[things I have read]]:
* Chris Palmer's [[Open Letter to the Free Software Community|http://noncombatant.org/blog/2009-12-29-01:40.html]]
Ouch! Sad but true. "Usability and security are so closely intertwined that they are almost the same thing, and both are absolutely non-negotiable features.  Computers users are not free if they cannot understand how to use their software... Similarly, they cannot be free if their computers can be pwned by any random maniac on the internet." ... "I can’t recommend exclusively free software..."
Chris Palmer is a security expert and is the author of the first open source project which attempts to beat [[Tahoe-LAFS|http://allmydata.org]] at its own game: [[Octavia|http://www.mail-archive.com/tahoe-dev@allmydata.org/msg02485.html]].
* Trygve Tollefsbol, Gerald Weissmann, et al.: [[Calorie Restriction: Scientists Take Important Step Toward 'Fountain of Youth'|http://www.sciencedaily.com/releases/2009/12/091222105219.htm]] (press release) (as recommended by [[Dr. Mike Eades|http://www.proteinpower.com/drmike/]] on his twitter stream)
Very exciting! This seems to be consistent with Taubes's speculations in [[Good Calories, Bad Calories]] that eating refined carbohydrates (i.e. sugars) is a leading cause of cancer and of aging.

added to [[things to read]]:
* Oscar ~Fernandez-Capetillo: [[Intrauterine programming of ageing|http://www.nature.com/embor/journal/v11/n1/full/embor2009262.html]]
(as recommended by [[Dr. Mike Eades|http://www.proteinpower.com/drmike/]] on his twitter stream)

Peter Secor (CEO of http://allmydata.com) showed me this article: //Patrick Munsey, staff writer of a local newspaper: [[Long arm of law reaches into World of Warcraft|http://kokomoperspective.com/news/local_news/article_15a0a546-f574-11de-ab22-001cc4c03286.html]]//. It makes me angry! An innocent man (a dealer of marijuana and LSD) fled from the US to Canada and settled down. Years later the local cops from his former hometown were able to track him down and have him arrested by the Royal Canadian Mounted Police and delivered back to Howard County, Indiana for punishment. This is in part thanks to Blizzard Entertainment's policy of informing police in ''different countries'' about the current location of their players. I wonder if they do this only for USA cops (Blizzard is a USA company), or if they also give out the locations of people living in the USA to cops from other countries.

added to [[things I have read]]:
* [[Patri Friedman|http://patrifriedman.com/]]: [[Beyond Folk Activism|http://www.cato-unbound.org/2009/04/06/patri-friedman/beyond-folk-activism/]]; This article argues that trying to persuade other people of your political convictions is a waste of time. Patri Friedman writes energetically and persuasively. He ends with a call to action: don't sit around and whine! If you value freedom then act like a proper hacker and go make some. (The analogy that he uses, of the obesity epidemic being caused by a mismatch between modern abundance and paleolithic eating instincts, is wrong. Obesity is caused by eating carbohydrates.)
moved to [[things I have read]] from the side bar (over there on the right-hand side of this page):
* Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno: a draft of the next edition of //[[Practical Cryptography|http://www.schneier.com/book-practical.html]]// (I actually stopped reading this a few months ago when the deadline for submitting editing suggestions passed.)
* Michael Pollan: //[[The Omnivore's Dilemma|http://www.michaelpollan.com/omnivore.php]]// (2006); Absorbing, pleasurable, packed with informative anecdotes and stories, and about one of my favorite topics of late: diet. Unfortunately its substance is not as good for your mind as it is pleasurable.  The science in this book is mostly a tossed salad of commonplaces about diet and health, many of which were refuted by [[Good Calories, Bad Calories]] a year later.  The economics in this book is an all-too-common example of a smart intellectual presuming that he can see how the values and choices of the multitude are wrong and advocating that they should adopt his instead.

added to [[things I have read]]: Jonathan Lehrer writing in Wired Magazine: [[Accept Defeat: The Neuroscience of Screwing Up|http://www.wired.com/magazine/2009/12/fail_accept_defeat/all/1]]; A fun and interesting article about [[Kevin Dunbar|http://utoronto.academia.edu/KevinDunbar]], who studies how scientists behave when they are doing science. Many of the lessons sketched out in this article seem to be applicable to other problem-solving tasks, such as software engineering. (recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]] on his twitter stream)
----
Nate Lawson (a good security researcher) wrote [[a nice article summarizing side-channel attacks|http://rdist.root.org/2009/12/30/side-channel-attacks-on-cryptographic-software/]].  It includes this discouraging line: "Unfortunately, owing to the structure of AES, there appears to be no way to build a high-performance implementation on a general-purpose CPU while avoiding cache side channels.".  This is why I want to switch from AES to [[XSalsa20|http://cr.yp.to/snuffle.html#security]].  (Also because ~XSalsa20 [[uses much less power and/or time|http://bench.cr.yp.to/results-stream.html]].) Unfortunately, "AES" is a successful brand -- it is the only crypto brand that matters and it appears on all crypto products -- and if we offer our users ~XSalsa20 instead, even though they will probably be safer, they will probably think that we are making them less safe.  :-(

''UPDATE'': chatting with folks on twitter and writing a note to tahoe-dev spurred me to realize that we can make a construction which is strong as the strongest of the two ciphers, even if AES leaks secrets by side-channels (as long as ~XSalsa20 doesn't): [[my note to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2010-January/003476.html]].  Hooray!  Problem solved!  (Aside from the regrettable waste of CPU cycles, which means shorter battery life, higher energy bills, and sometimes slower performance.)
----
[[NIST-certified USB Flash drives with hardware encryption cracked|http://www.h-online.com/security/news/item/NIST-certified-USB-Flash-drives-with-hardware-encryption-cracked-895308.html]] (referenced by Sébastien Martini); "The real question, however, remains unanswered – how could USB Flash drives that exhibit such a serious security hole be given one of the highest certificates for crypto devices? Even more importantly, perhaps – what is the value of a certification that fails to detect such holes?"
----
Nick Szabo blogged that the [[U.S. Supreme Court may be about to strike down software and business method patents|http://unenumerated.blogspot.com/2009/11/software-and-business-method-patents-at.html]]. That would be interesting! :-)

"Yesterday I was ~THinking about ~THings that make the world better, and I came across the Internet, but I realized that was already there. But anyway I was amazed by how wonderful it is, and I was thinking, why can't ''every'' game be networked?" —Irby
----
Wanted: someone to either port Poly1305 to Crypto++ or else convince me that VMAC is fast enough and peer-reviewed enough: http://allmydata.org/trac/pycryptopp/ticket/33 .

I just set up a new development environment -- Ubuntu 9.10 Karmic in virtualbox on a new Macbook Pro.  I want a terminal emulator that is good-looking and fast.  There are three terminal emulators that I was able to easily apt-get and configure to use my favorite font and colors: konsole (the KDE terminal emulator), gnome-terminal (the Gnome one) and xfce4-terminal (the XFCE one).  (I tried to get uxterm to use my preferred font but couldn't figure out how to do it so I gave up after a few minutes.)  I made a file which contained 27 copies of Markus Kuhn's [[UTF-8-demo.txt|http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt]] and catted it to stdout.  I did this ten times in a row with each terminal.  The best and worst times to cat this file were:
* konsole
** best: 0.534s
** worst: 0.764s
* gnome-terminal:
** best: 0.978s
** worst: 1.122s
* xfce4-terminal:
** best: 0.970s
** worst: 1.098s
The winner is konsole!  Hm, gnome-terminal and xfce4-terminal look very alike one another visually. I guess they are probably built on the same underlying implementation.

UPDATE! 21:40 ~UTC-7 The current soundtrack for ~Tahoe-LAFS Hacking Weekend here in my house is the [[live Ogg Vorbis stream from Radio1190|http://radio1190.colorado.edu:8000/high.ogg]].
----
It is ~Tahoe-LAFS Hacking Weekend! Read [[the invitation|http://allmydata.org/pipermail/tahoe-dev/2010-January/003515.html]] and join us.  Newbies welcome!

~Tahoe-LAFS Hacking Weekend is a good time to listen to [[Kraftwerk|http://en.wikipedia.org/wiki/Kraftwerk]]'s [[Computer World|http://en.wikipedia.org/wiki/Computer_World]] album on Amber's awesome new hand-me-down stereo system.

added to [[things I have read]]:
 * Kazuyuki Kobayashi, Jun Ikegami, Shin’ichiro Matsuo, Kazuo Sakiyama and Kazuo Ohta: [[Evaluation of Hardware Performance for the SHA-3 Candidates Using SASEBO-GII|http://eprint.iacr.org/2010/010.pdf]]; a careful evaluation of ~SHA-3 candidates on a Xilinx Vertex 5. The good ones in my opinion are in descending order: ~CubeHash, ~SHA-256, Hamsi, Luffa, Skein, Shabal, and Blake.  The high-throughput implementations reported here for those algorithms each fit into less than 2000 slices, leaving only ECHO and Groestl as using more than 3000 slices.  (Note the implementations were optimized for throughput, not size.)
----
[[Performance comparison of Amazon EC2 vs. Rackspace Cloud Servers|http://www.thebitsource.com/2010/01/11/rackspace-cloud-servers-versus-amazon-ec2-performance-analysis/]] (recommended by Jonathan Ellis on twitter) includes this:
|!~CloudServer instance|!price to compile kernel (in cents)|
|256MB CS|0.13¢|
|512MB CS|0.17¢|
|1 GB CS|0.3¢|
|2 GB CS|0.67¢|
|4 GB CS|1.38¢|
|8GB CS|2.62¢|
|15.5GB CS|5.21¢|
|!Amazon ~EC2 instance|!price to compile kernel (in cents)|
|~EC2 Small (1.7G)|3.5¢|
|~EC2 Medium (High CPU, 1.7G)|1.94¢|
|~EC2 Large (7.5 G)|2.93¢|
|~EC2 XL (High CPU, 7.5 G)|1.67¢|
|~EC2 XL (15G)|3.28¢|
----
I can get you $200 discount off of the registration fee for RSA Conference 2010, but only if you haven't already registered. Ask me!
----
Today I set up a bunch more tools. I'll use these tools to set up some more tools. Then we'll use those tools to make awesome tools!
----
Ahh!  I ran //find Music -name '*.flac'// and rediscovered the sweet and catchy sounds of Surfjan's Steven's "(Come On Feel The) Illinoise".  All songs on this album are about the state of Illinois.  Also, listening to music soothes my soul and helps me to focus instead of frantically switching tasks and worrying about what ''else'' I ought to be doing.
----
The Python maintainers need to accept that the word for "a bundle of Python functionality that you can download and use in your code" is "a Python package": http://mail.python.org/pipermail/distutils-sig/2010-January/015332.html

added to [[things I have read]]:
* Jian Guo, San Ling, Christian Rechberger, and Huaxiong Wang: [[Advanced Meet-in-the-Middle Preimage Attacks: First Results on Full Tiger, and Improved Results on MD4 and SHA-2|http://eprint.iacr.org/2010/016.pdf]]; a "certificational" pre-image attack (requiring 2^^184^^ computations) on the full Tiger hash function, a slightly better attack on ~SHA-256 reduced to 42 rounds (from its actual 64), and the promise of future applications to ~SHA-3 candidates.  I look forward to that last part!
----
I updated [[can we build a crypto system to last for a hundred years?]], appending a couple of paragraphs about how to cascade together two digital signature schemes.
----
I stayed up way, way too late last night writing [[can we build a crypto system to last for a hundred years?]].
----
Uh-oh! The anti-aging drug company that ~GlaxoSmithKline paid $720 million for might turn out to be full of hot air instead of the elixir of youth.  Derek Lowe: [[The Sirtris Compounds: Worthless? Really?|http://pipeline.corante.com/archives/2010/01/12/the_sirtris_compounds_worthless_really.php]]
I enjoy this story, not only because I love reading about failures, mistakes, and mismanagement, but also because I really hope that the fountain of youth will turn out to be cheap, unpatented, and impossible to centrally control.  It is plausible at this point in time that an extremely low-carb diet will add years or even decades to the life of ''most'' people -- something that can't be said of any drug to date and probably not of any drug on the horizon.

On the #cassandra IRC channel:
<zooko> "idempotent" is a good word.
<asl2> you can say that again.
----
We're going to have a ~NoSQL meetup in Boulder, Colorado!  "~NoSQL" is the name for an array of new distributed database-ish systems: http://en.wikipedia.org/wiki/NoSQL . Write to [[zooko@zooko.com|mailto:zooko@zooko.com]] and say when you can attend, what you want to say or hear, and if you have any robotic toys you can bring.

(What do robotic toys have to do with ~NoSQL?  Come on -- give me a break.  Robotic toys are awesome.)

Oh dear I stayed home from [[work|http://simplegeo.com]] today to help Brian make slides for our RSA Conference 2010 talk: [[Your Cloud Storage Provider Does Not Need Access To Your Data|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=4522]], and to review Kevan's patch for [[Tahoe-LAFS ticket #778|http://allmydata.org/trac/tahoe/ticket/778]], but instead I spent almost all day writing these blog entries: [[the human cost of bad science]] and [[my notes on Skeaff and Miller 2009]].

added to [[things to read]]:
* The Memory Bank: [[Building economic democracy with community currencies|http://thememorybank.co.uk/2010/01/16/building-economic-democracy-with-community-currencies/]] (blog post) (recommended by [["openmoney" on twitter|http://twitter.com/openmoney]])
added to [[studies on low-carb diet]]:
* [[Würzburg Diet for Cancer Patients|http://comfort-eaters-diet.blogspot.com/2010/01/wurzburg-diet-for-cancer-patients.html]]; Not a study but a blog post about a low-carb diet programme for cancer patients published by the University of Würzburg.
----
It is time for the Very Boring ~Tahoe-LAFS Hacking Weekend!
Join us on IRC or the mailing list and help with:
* documentation -- improve the user docs, developer docs, wiki, help text, UI text
* manual testing -- upload and download files, share them with your friends, destroy hard drives with weaponry
* automated testing -- write unit tests, win plaudits from your peers
* packaging -- get ~Tahoe-LAFS compiled and packaged for your favorite operating system or packaging tool
* publicity -- write a post about ~Tahoe-LAFS in your favorite user group newsletter or bulletin board system
* code review -- study Brian's patches and learn software engineering from a master
* Related Projects -- there are a ton of great hacks that build on top of ~Tahoe-LAFS or integrate ~Tahoe-LAFS with other tools.  See [[the RelatedProjects page|http://allmydata.org/trac/tahoe/wiki/RelatedProjects]] for a full list, but they include fuse, Hadoop, Android, bzr, ~TiddlyWiki and more. Join us and play with or hack on some of those! We will mention some of the most interesting Related Project in the ~Tahoe-LAFS v1.6 release announcement.
irc.freenode.net #tahoe-lafs
([[original announcement|http://allmydata.org/pipermail/tahoe-dev/2010-January/003564.html]] of the Very Boring ~Tahoe-LAFS Hacking Weekend)

posted to the microblogging services: (identi.ca and twitter):
* The CEO of [[simplegeo|http://simplegeo.com]] just came by and gave us all homework: to read a science fiction book. Awesome!
* A coworker tried to buy [["Rainbows End"|http://en.wikipedia.org/wiki/Rainbows_End]] for Kindle. Not there! B&N: nope. ~BitTorrent a PDF: yes. Beautiful irony! (Read "Rainbows End"!)
* The same coworker casually described [[simplegeo|http://simplegeo.com]] as "a Singularity-oriented company".
----
added to [[studies on low-carb diet]]:
* David N. Ruskin, Masahito Kawamura, Jr, Susan A. Masino: [[Reduced Pain and Inflammation in Juvenile and Adult Rats Fed a Ketogenic Diet|http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0008349]] (recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]] who posted on twitter: "Ketogenic diet reduces pain and inflammation in rodents. My experience is that it does in humans, too.")

I can walk without crutches! Walking is awesome! You have no idea how great it is to WALK.
----
[[pyutil.jsonutil]]

It never ceases to amaze me how only a few lines of code can conceal bugs. And how those bugs can evade inspection and static type checking but not testing. http://allmydata.org/trac/tahoe/ticket/928#comment:12

* [[FIPS and Common Criteria: don't rely on them]]
* I've been reading a lot about the Inuit people, especially about their diet and health. I'm keeping a few references and notes in [[studies on low-carb diet]]

It is time for Game Day! Come to Chez O'Whielacronx Sunday 2010-02-07 at 14:00 ~UTC-7 and play board games, card games, mind games, dice games, role-playing games, computer games, or other sorts of games!

added to [[things I have read]]:
* Jennifer B Keogh, Grant D Brinkworth, Manny Noakes, Damien P Belobrajdic, Jonathan D Buckley, and Peter M Clifton: [[Effects of weight loss from a very-low-carbohydrate diet on endothelial function and markers of cardiovascular disease risk in subjects with abdominal obesity|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Keogh-2008-Effects_of_weight_loss_from_a_very-low-carbohydrate_diet_on_endothelial_function_and_markers_of_cardiovascular_disease_risk_in_subjects_with_abdominal_obesity.pdf]] <-- that link is a copy of the article stored on this ~Tahoe-LAFS grid; Here is the page for the article [[at the American Journal of Clinical Nutrition|http://www.ajcn.org/cgi/content/abstract/87/3/567]]
This was a short (8 week) randomized controlled trial of a reduced-calorie (1400 kcalorie) very low carb diet (20 grams of carb per day) with lots of saturated fat versus a reduced-calorie low fat diet. The authors started with the hypothesis that the saturated fat in the first diet would cause a worsening of flow-mediated dilation, which is a good measure of cardio-vascular health. This turned out not to be the case: the low-carb dieters didn't suffer worse flow-mediated dilation and in fact did better than the low-fat dieters on (almost) all of the numerous and precise measures that the authors took.

I just listened to [[the twit.tv FLOSS weekly show about MongoDB|http://twit.tv/floss105]], because I've been invited to appear on this show to talk about [[Tahoe-LAFS|http://allmydata.org/trac/tahoe-lafs]] and because I'm interested in ~MongoDB. I was horrified at the first ten minutes of the show—the hosts made juvenile jokes and crudely insulted some wikipedian who was trying to get [[their wikipedia page|http://en.wikipedia.org/wiki/FLOSS_Weekly]] removed. It sounded to me like they were trying to emulate the style of shock-jock morning ~DJs on radio who use low-brow humor to titilate the audience.
I said to Amber "Well, I guess I don't want to appear on ''this''!". However, once their guest, Michael Diroff of the ~MongoDB project, joined them they went into a fun and stimulating discussion with plenty of technical depth and honest enthusiasm.
So now I have mixed feelings. Maybe I'll listen to some more episodes—they mentioned that they covered ~CouchDB a year ago—and see if the incivility of this episode was a rarity. Or maybe they'll happen to see this blog entry, and if they do they might uninvite me. Who knows.
----
P.S. Yep, [[they uninvited me|http://twitter.com/merlyn/status/8996248573]].
----
I just want to point out something about the diet study that I reported on [[a couple of days ago|2010-02-09]]. This is the sort of diet research that you should pay attention to because:
1. It was a randomized controlled trial, not just an observational report. Ex post facto observations are far too easy to explain with some story that fits your preconceived theory. In particular, it is far too easy to //assume an arrow of causation// when you see a correlation in your ex post facto observations, which can lead you to incorrect conclusions. This mistake is absolutely ubiquitous in diet research. Randomized controlled trials are a great way to prevent yourself from incorrectly assuming causation.
2. The authors didn't get the result that they expected. This is always a signal that real learning is happening. Confirmation hardly teaches you anything new—no matter how many times your theory has been confirmed, it might still fail the next test and turn out to be wrong. ''Disconfirmation'' is when you really take a great step forward! In this case, the scientists who designed the experiment thought that eating plenty of saturated fat would lead to worse cardiovascular health, as evaluated by measurements such as flow-mediated dilation (which I think means your arteries are nice and stretchy instead of "hardened") and LDL cholesterol. The results clearly showed that this wasn't the case—eating a diet high in saturated fat did ''not'' induce worse flow-mediated dilation and did ''not'' increase LDL cholesterol levels.
//The first principle is that you must not fool yourself—and you are the easiest person to fool.// —Richard Feynman [[Cargo Cult Science|http://www.lhup.edu/~DSIMANEK/cargocul.htm]]
See also [[The Black Swan|http://en.wikipedia.org/wiki/The_Black_Swan_%28Taleb_book%29]] by Nicholas Nassim Taleb.

I posted to the tahoe-dev mailing list: [[The Future of Tahoe-LAFS|http://allmydata.org/pipermail/tahoe-dev/2010-February/003891.html]].

 * Irby, Munga, and I just watched this video: [[200 years that changed the world|http://www.gapminder.org/videos/200-years-that-changed-the-world/]] starring Hans Rosling (recommended by [[Dirk Loss|http://dirk-loss.de/]]).
 * Amber and I just watched this video about [[flattr|http://flattr.com]]. It is called "social micropayments". The video emphasizes two points: 1. it shows giving money to creators as a way of expressing love. 2. the donors pay only a flat monthly fee, which means the marginal cost of clicking the "give this creator some love" button is zero. flattr.com is from the creators of [[The Pirate Bay|http://en.wikipedia.org/wiki/The_Pirate_Bay]]. This could be huge! It was one of the dreams we tried to create at Mojo Nation. I'm glad it is finally happening.
 * [[FIPS and Common Criteria: don't rely on them, part 2]]

 * I went back and edited my blog entry from [[2010-02-11]] to say this: "Confirmation hardly teaches you anything new—no matter how many times your theory has been confirmed, it might still fail the next test and turn out to be wrong. ''Disconfirmation'' is when you really take a great step forward!"
 * We're probably going to put out a bug-fix release of ~Tahoe-LAFS next weekend. [[[tahoe-dev] calling all bugs! Please report to v1.6.1|http://allmydata.org/pipermail/tahoe-dev/2010-February/003903.html]]
 * [[[tahoe-dev] governance note: David-Sarah Hopwood now has write authority to trunk|http://allmydata.org/pipermail/tahoe-dev/2010-February/003902.html]]

//(Sorry, but it appears that [[you have to click here|https://cm.rsaconference.com/US10/catalog/public.jsp]] before the rsaconference.com web site will allow the hyperlinks to it below to work. So click there and then come back and read this journal entry and you can follow the links to sessions.)//

Brian's and my presentation at the RSA conference in San Francisco is going to be Friday, March 5 at 10:10 AM. Mark your calendars! That's late enough that you can stumble out of bed and find food and coffee and Orange Room 310 in time! [[Tahoe-LAFS: Your Cloud Storage Provider Does Not Need Access To Your Data|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=4522]]
----
Suppose you get up and find food and coffee earlier. What presentation should you watch while waiting for ours to start? Looking at the schedule of things that start at 9:00, I would be interested in:
* Richard Howard: [[2010 Cyber Security Trends and Future Cyber Security Disruptors|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=3780]], or possibly:
* panel: [[Digital Forensics vs. Security & Encryption|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=3567]] (in which technologies like ~Tahoe-LAFS are presumably considered part of the problem :-)), or of course if you are into serious crypto:
* Martin Schlaeffer, Gaetan Leurent: [[Analysis of Hash Functions|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=6214]].
And of course after you've seen our talk you might be interested in the following things that start at 11:20:
* Ken Grewal, Marc Miller: [[Next Generation Scalable, Cost-Effective E2E Security|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=3509]], or:
* Phillip ~Hallam-Baker: [[DNSSEC – A Right Answer to the Wrong Question|https://cm.rsaconference.com/US10/catalog/profile.do?SESSION_ID=3493]]
But, I don't know if I'm overlooking something. Are some of the speakers that I didn't list exceptionally good, or is there exceptionally interesting new material in some of those talks? If you know, please tell me.
----
I'm watching this [[talk by Chris Wanstrath|http://thechangelog.com/post/391445075/rippin-off-python-w-chris-wanstrath]] in which he is addressing an audience of Rubyists and telling them what the Python community does better than the Ruby community. One thing I like about his presentation style is that there are no sentences or phrases on his slides which he could read from. Each slide is only a noun or a picture.
I would really like to see a presentation like this (possibly by this same guy) telling Pythonistas like me what the Ruby community does better.

added to [[things to read]]:
* R. Neil A. Black, Michelle Spence, Ross O. ~McMahon, Geraldine J. Cuskelly, Cieran N. Ennis, David R. ~McCance,Ian S. Young, Patrick M. Bell, and Steven J. Hunter: [[Effect of Eucaloric High- and Low-Sucrose Diets With Identical Macronutrient Profile on Insulin Resistance and Vascular Risk|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Black_et_al-2006-Effect_of_Eucaloric_High-_and_Low-Sucrose_Diets_With_Identical_Macronutrient_Profile_on_Insulin_Resistance_and_Vascular_Risk.pdf]] (as recommended by Brendan O'Connor)

Heh. I spent all evening writing up [[my notes on Black 2006]], and so I didn't get around to writing up the much more interesting paper that I read today:
* Amisha Patel, Paula L. Pyzik, Zahava Turner, James E. Rubenstein, and Eric H. Kossoff: [[Long-term outcomes of children treated with the ketogenic diet in the past|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Patel_et_al-Long-term_outcomes_of_children_treated_with_the_ketogenic_diet_in_the_past.pdf]]

added [[Stephan Guyenet|http://wholehealthsource.blogspot.com]] to by blogroll (over there on the right-hand side)
Stephan Guyenet blogs about health. As I hope you know by now, I wouldn't be linking to his blog if he didn't practice "evidence-based thinking"—he seeks out and pays attention to disconfirmatory evidence.

* added to [[things to read]]:
** [[Steve Solomon|http://www.soilandhealth.org/05steve%27sfolder/05aboutmeindex.html]]: [[Longevity and Nutrition Library|http://www.soilandhealth.org/02/0203CAT/0203longevitylibcat.html]]; Wow! It is a treasure trove of out-of-print books about human health and longevity, curated by some kind of radical Tasmanian farmer philosopher!
* added to [[things I have read]]:
** [[Stephan Guyenet|http://wholehealthsource.blogspot.com/]]: [[Magnesium and Insulin Sensitivity|http://wholehealthsource.blogspot.com/2010/02/magnesium-and-insulin-sensitivity.html]] (blog entry)
** Inna Slutsky, Nashat Abumaria, ~Long-Jun Wu, Chao Huang, Ling Zhang, Bo Li, Xiang Zhao, Arvind Govindarajan, ~Ming-Gao Zhao, Min Zhuo, Susumu Tonegawa, and Guosong Liu: [[Enhancement of Learning and Memory by Elevating Brain Magnesium|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Slutsky-2010-Enhancement_of_Learning_and_Memory_by_Elevating_Brain_Magnesium.pdf]] and [[the press release about it|http://www.sciencedaily.com/releases/2010/02/100222162011.htm]] (as recommended by some commenters on that blog entry of Stephan Guyenet's)
Wow, that is interesting. Stephan Guyenet says that manipulating magnesium levels in human test subjects has a dramatic effect on insulin sensitivity and blood lipid levels.
See also that article by Slutsky et al. 2010. Receiving a dose of a special compound of magnesium that can cross the blood-brain barrier improved memory and learning in rats. Okay, I am convinced that intake of magnesium is important! However, none of the compounds of magnesium that you can buy in pills today crosses the blood-brain barrier, and the commercially-available ones they tested in this study failed to make rats any smarter. The press release says: "Before the new compound becomes commercially available, Dr. Slutsky advises people to get their magnesium the old-fashioned way -- by eating lots of green leaves, broccoli, almonds, cashews and fruit." Ah, no thanks! According to [[USDA National Nutrient Database for Standard Reference|http://www.nal.usda.gov/fnic/foodcomp/search/]] 100g of broccoli contains 21mg of magnesium (and there is reason to suspect that this number was true of broccoli grown in the U.S. fifty years ago but the amount of magnesium in modern commercial broccoli is probably much lower). 100g of almonds allegedly contains 268mg magnesium, and 100g of espresso 80mg magnesium. US recommended daily allowance is 420mg.
I've asked Amber to feed me more bone broth.

* added to [[things I have read]]:
** ~Yun-Jung Bae, ~Mi-Hyun Kim, ~Mi-Kyeong Choi: [[Analysis of Magnesium Contents in Commonly Consumed Foods and Evaluation of its Daily Intake in Korean Independent-Living Subjects|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Bae-2009-Analysis-of-Magnesium-Contents-in-Commonly-Consumed-Foods-and-Evaluation-of-its-Daily-Intake-in-Korean-Independent-Living-Subjects.pdf]] (as provided by a completely anonymous and untraceable friend known by his //nom de guerre// "Running Squirrel")
They measured the magnesium content of 366 different food items and compared them to the Korean national food authority's statements of the magnesium contents of those foods. I would like to see the USDA values added to this table for comparison. Then they interviewed more than two hundred people and asked them what they'd eaten over the preceding 24 hours, and calculated their approximate magnesium intake. Chinese broccoli apparently has 35mg of magnesium per 100g. Mackerel has a mere 14mg. (If you scan these results looking for foods which are high in magnesium then read Stephan Guyenet's [[blog entry on the topic|http://wholehealthsource.blogspot.com/2010/02/magnesium-and-insulin-sensitivity.html]] which says that if they also contain [[phytic acid|http://en.wikipedia.org/wiki/Phytic_acid]] then this will prevent your body from absorbing magnesium.)

I see that Maciej Fijalkowski has posted [[shiny new benchmarks|http://morepypy.blogspot.com/2010/03/hello.html]] of [[PyPy|http://codespeak.net/pypy/dist/pypy/doc/]] running [[Twisted|http://twistedmatrix.com]]. One of the tasks is measured as 285% faster than ~CPython! (And also much faster than the widely touted [[Unladen Swallow|http://code.google.com/p/unladen-swallow/]] project.)
That is an exciting result! For starters, the fact that ~PyPy is mature and stable enough to run Twisted at all is an achievement, and following that the fact that the ~PyPy ~Just-in-Time compiler is successful at optimizing Twisted code is very exciting.
However, there is something confusing about this: the task in question (labelled //iterations//) is an empty loop—it is just measuring the overhead of Twisted's scheduling and socket management. So Twisted can do nothing 285% faster on ~PyPy than on ~CPython! Does that matter? The answer is yes it does matter, and we should start reporting performance benchmarks in terms of CPU cycles used instead of in terms of "speed".
CPU cycles used is (for a ~CPU-bound process) basically the inverse of speed. So another way to phrase this same result is that ~PyPy used about 1/4 of the CPU cycles to run an empty loop in Twisted, compared to ~CPython. That means that your application written on top of Twisted has more CPU cycles available to it during the same time period, and it means that less power is wasted by the Twisted overhead itself. Somebody pays for the power usage. This is particularly acute on a battery-powered device where power usage means shorter battery life.
However, this rarely has any effect on "speed", because your process is rarely ~CPU-bound. Usually your process is waiting on the network or the disk or the user. A better way to think about computational performance is that you can do the same job with less CPU usage, which means having a longer battery life, or fitting more server processes onto the same CPU, or leaving more CPU available for other processes that are running at the same time as yours.
* added to [[things to read]]:
** John Jacob Cannell MD: [[Vitamin D, Vitamin A, and Cancer|http://www.vitamindcouncil.org/newsletter/vitamin-d-vitamin-a-and-cancer.shtml]] (as recommended [[by Dr Mike Eades|http://twitter.com/DrEades/status/9825372497]] and [[by Amber|http://twitter.com/ambimorph/status/9831705925]])
** Joshua Wolf Shenk: [[What Makes Us Happy?|http://www.theatlantic.com/magazine/archive/2009/06/what-makes-us-happy/7439/]] ([[as recommended|http://twitter.com/bos31337/status/2162906886]] by [[Bryan O'Sullivan|http://serpentine.com]])
* added to [[things I have read]]:
** Leila Gray, Dr. Philippe P. Hujoel: [[If A Diet Is Bad For The Teeth It Is Also Bad For The Body|http://www.medicalnewstoday.com/articles/157105.php]] (press release) ([[as recommended|http://twitter.com/DrEades/status/9692921690]] by [[Dr. Mike Eades|http://www.proteinpower.com/drmike/]]); I'm not sure if I'll get around to posting notes about that, but I did find it interesting. I'm just very busy right now visiting San Francisco for the RSA conference.

* added to [[things I have read]]:
** Ellen Ferrante, Bobbie Mixon: [[Breakthrough in Electron Spin Control Brings Quantum Computers Closer to Reality|http://www.nsf.gov/discoveries/disc_summ.jsp?cntn_id=116456]] (press release); Hopefully [[Tahoe-LAFS|http://tahoe-lafs.org]] v2 will be invulnerable to quantum computers—it will have use "post-quantum" cryptography as described in [[can we build a crypto system to last for a hundred years?]].

* Okay folks, this is your last warning! Head to Orange room 310, RSA Conference 2010, San Francisco, California, tomorrow morning at 10:10!
* Got home from a pleasurable evening of discussing ~Tahoe-LAFS and rehearsing our Security Theatre act, and found a musician named Boss ~McGravity practicing his catchy, slightly sad songs in Nathan's living room. Life is good.
* Had a nice breakfast with D.C., a founding father of open cryptography and "Freedom Technologies". We talked about [[Tahoe-LAFS|http://tahoe-lafs.org]], zero-carb diets, [[One Hundred Year Security|can we build a crypto system to last for a hundred years?]], his crazy new optics inventions, voter-verifiable elections, and personal stuff. It was a great pleasure.
* added to [[things to read]]:
** Amir Herzberg: [[Folklore, Practice and Theory of Robust Combiners|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Herzberg_2008-Folklore,_Practice_and_Theory_of_Robust_Combiners.pdf]]

added to [[things to read]] and to [[studies on low-carb diet]]:
* Vilhalmur Stefannson: //[[The Fat of the Land|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Stefansson_1956-The_Fat_of_the_Land.pdf]]// (book, 1956) (<-- PDF version hosted on ~Tahoe-LAFS)
* Stefania Piconi, Daria Trabattoni, Cristina Luraghi, Edoardo Perilli, Manuela Borelli, Michela Pacei, Giuliano Rizzardini, Antonella Lattuada, Dorothy H. Bray, Mariella Catalano, Antonella Sparaco and Mario Clerici: [[Treatment of periodontal disease results in improvements in endothelial dysfunction and reduction of the carotid intima-media thickness|http://www.fasebj.org/cgi/content/abstract/23/4/1196]] ([[as recommended|http://twitter.com/DrEades/status/10185869017]] by Dr. Mike Eades)
----
[[Tahoe-LAFS was a hit at the RSA conference|http://allmydata.org/pipermail/tahoe-dev/2010-March/004058.html]] (mailing list message)

Amber built a jerky drier following [[$10 Jerky Drier instructions|../URI:DIR2-RO:ju2g6y7ttynge6pl7wgv4cqteu:ipmwu7g6p5loa76ov32kokmnri3mzj4q2p2f4ngruoehsliot75a/JerkyDrierInstructions.pdf]]. The instructions claim that it is illegal to sell real jerky in the USA—jerky that has been dried without being cooked—and that the only way to get real jerky is to dry it yourself. I don't know about that, but I do know that the jerky Amber has been making is by far the best jerky I've ever tasted.
Before she built this jerky drier she would make jerky by putting strips of meat in the oven and leaving it on low for a few days. Those were okay, but they weren't nearly as good as these.
The only problem is that we're eating the jerky as fast as she can make it, so there's none left to make pemmican. I guess she needs to build a couple more jerky driers!

* Interesting fact: you can maintain your integrity and unforgeability while secure hash algorithms and digital signature algorithms come and go, but you can't maintain your confidentiality while ciphers come and go: [[message to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2010-March/004102.html]].
* "Using unguessable names is a good way to build a cap system on top of an existing singleton namespace system.  But if you can avoid ever having a global namespace in the first place, that's even better." —Kenton Varda

* Published a Python data structure for processing data which arrives and departs in many chunks: [[StringChain|http://tahoe-lafs.org/trac/stringchain]].
* [[Tahoe-LAFS|http://tahoe-lafs.org]] is applying to be part of the [[Google Summer of Code 2010|http://code.google.com/soc/]]! Read the mailing list message to tahoe-dev: [[Google Summer of Code 2010 -- Ideas Needed!|http://allmydata.org/pipermail/tahoe-dev/2010-March/004117.html]]

Big day today! [[SimpleGeo|http://simplegeo.com]] launched public beta by [[admitting three thousand more developers|http://twitter.com/simplegeoinc/status/10687388719]] to beta test our web service. [[Tahoe-LAFS|http://tahoe-lafs.org]] was [[selected for the Google Summer of Code|http://allmydata.org/pipermail/tahoe-dev/2010-March/004184.html]].
Now it is almost eleven o'clock and I still haven't gotten to sleep.
But at least I'm keeping my new policy of not drinking a pot of coffee in the evening.
Added to [[things I have read]]:
* Daniel Suarez: //[[Daemon|http://thedaemon.com/]]// (book)
* Jonathan Lange: [[Your Code Sucks and I Hate You: The Social Dynamics of Code Reviews|http://mumak.net/stuff/your-code-sucks.html]] ([[as recommended|http://twitter.com/dreid/status/10690810313]] by David Reid)
Oh wow //Daemon// has left me pensive. The writing is amateurish and every time the premise was reiterated (which was frequently) I was reminded of how unbelievable I found the premise to be. ''But'' this future or something like it is all too possible, and I can't stop thinking about it, and I can't make up my mind whether it is good or evil. Also the book ends in media res and now I have to read the sequel! Fortunately my boss gave me a hardcopy of the sequel along with the original. :-)

God has been speaking to me and telling me to study agile software engineering practices. And by "God" I mean my twitter stream:
* added to [[things I have read]]:
** Jonathan Lange: [[Your Code Sucks and I Hate You: The Social Dynamics of Code Reviews|http://mumak.net/stuff/your-code-sucks.html]] ([[as recommended|http://twitter.com/dreid/status/10690810313]] by David Reid); I linked to this from the ~Tahoe-LAFS [[How To Review Patches|http://tahoe-lafs.org/trac/tahoe-lafs/wiki/PatchReviewProcess]] page.
* added to [[things to read]]:
** Jason Yip: [[It's Not Just Standing Up: Patterns of Daily Stand-up Meetings|http://martinfowler.com/articles/itsNotJustStandingUp.html]]
** Mike Cohn: [[An Overview of Scrum for Agile Software Development|http://www.mountaingoatsoftware.com/scrum/overview]]
** Agile Manifesto Authors: [[Principles behind the Agile Manifesto|http://agilemanifesto.org/principles.html]]
** Abby Fichter: [[Are We Agile Yet?|http://www.thehackerchickblog.com/2010/02/are-we-agile-yet.html]]
** [[The Agile Skills Project|http://sites.google.com/site/agileskillsprojectwiki/]] (via Jack Lloyd)

''Hack Fest!''
By popular demand, we are organizing a Hack Fest next Saturday, the 27th of March, in Boulder, Colorado and also on The Internet. Read the [[thread on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2010-March/004187.html]].

Josh's friend John pointed out this science paper as evidence that consuming saturated fat or all fat might be bad for your heart. I wrote an e-mail reply, a slightly edited version of which is below.

<<<
On Saturday, 2010-02-20, at 23:00 , Josh wrote:

> So I mentioned to my friend John that I thought high fat diets might not cause coronary heart disease.  He said he thought maybe it did.   He found this article.  It rings bells, so I think it might be discussed in Gary Taubes's "Good Calories, Bad Calories" but I'm at John's house right now and don't have access to it.

Unfortunately I don't have a copy of "Good Calories, Bad Calories" anymore -- I've given them all away! (The most recent copy I gave to D.C.)

Here are my quick notes about this paper -- Hu et al. [[Dietary Fat Intake and the Risk of Coronary Heart Disease in Women|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Hu-1997-Dietary_Fat_Intake_and_the_Risk_of_Coronary_Heart_Disease_in_Women.pdf]] (1997):

"Each increase of 5 percent of energy intake from saturated fat, as compared with equivalent energy intake from carbohydrates, was associated with a 17 percent increase in the risk of coronary disease (relative risk, 1.17; 95 percent confidence interval, 0.97 to 1.41; P=0.10)."

Note P > 0.05.

I don't understand what it means to have both a confidence interval and a P value. I barely understand what a confidence interval is in the first place. Wouldn't they be able to choose a wider confidence interval and then get a smaller P value with the same data?

Anyway, wouldn't the standard way to report a P value of 0.1 be something like "There was no statistically significant association between increased energy intake from saturated and risk of coronary disease."?

(Josh is fond of mentioning that the "0.05" cutoff for P is arbitrary. I'm sure I agree, but if everyone would obey the same arbitrary rules we could at least compare papers to each other.)

The letters to the editor at the end raised good reasons for doubt in my mind. The most telling to me are (a) food-frequency questionnaires suffer decorrelation from actual food intake and at the same time have many interesting correlations with other confounding factors, and (b) nurses aren't necessarily like the rest of us, for example they died of coronary heart disease less than half as often as the rest of USA women during this period. The authors of the paper retorted that the "metabolic effect" of saturated fat on blood lipids and heart health should affect nurses the same as they affect everyone else. No kidding -- if their theory of that metabolic effect were //true//. But there is no evidence (that I can find) that it is.

Okay next I looked at a recent meta-analysis that I have read which cites this Hu 1997 Nurses Health Study. Here is my blog entry about this meta-analysis:

[[my notes on Skeaff and Miller 2009]]

That meta-analysis (Skeaff & Miller 1009) cites Hu 1997 only a little bit. It mostly cites the follow-up, Oh et al. [[Dietary Fat Intake and Risk of Coronary Heart Disease in Women: 20 Years of Follow-up of the Nurses' Health Study|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Oh-2005-Dietary_Fat_Intake_and_Risk_of_Coronary_Heart_Disease_in_Women:_20_Years_of_Follow-up_of_the_Nurses'_Health_Study.pdf]] (2005), checking up on those same nurses a few years later. Oh 2005 drops the part about saturated fat associating with heart disease at P=0.1 and just talks about monounsaturated and polyunsaturated fats being good and trans fats being bad (for your heart). It says "Intakes of total fat, saturated fat, and monounsaturated fat had no clear relation to CHD [coronary heart disease] regardless of age group (data not shown).".

Apparently they added the latest questionnaire results, probably refined their multivariate analysis to include more factors, and got a P value of 0.93 for the hypothesis that intake of saturated fat correlated with coronary heart disease (table 2).

One particularly depressing note from Oh 2005 was this: "From 1980 to 1998, the average intake of total fat decreased from 39.0 percent to 29.0 percent, saturated fat intake decreased from 15.6 percent to 9.4 percent, monounsaturated fat intake decreased from 16.0 percent to 11.5 percent, and trans-fat intake decreased from 2.2 percent to 1.6 percent. Polyunsaturated fat intake increased from 5.3 percent to 5.6 percent (figure 1)."

Those nurses were assiduously following the recommendations of the health experts -- American Heart Association, USDA, American Diabetes Association, etc. etc. -- and probably doing damage to their health thereby.

By the way, Skeaff & Miller 2009 casts a lot of doubt on Oh 2005's hypothesis that monounsaturated and polyunsaturated fats are good for your heart. Pretty much the only part of this whole mess that //isn't// thoroughly doubtful after Skeaff & Miller 2009 is that trans-fats are bad for you.

For various other reasons that I have not yet related to John but have to everyone else, I'm of the opinion that saturated fat is good for you.

Regards,

Zooko
<<<

Hello world! This blog is working again!

[[Amazon EC2 pricing per year]]

Updated [[Amazon vs. Rackspace pricing per year]] on request.

I announced on the python mailing list the availability of an open source package that I wrote for ~SimpleGeo: [[announcing jsonutil package |http://mail.python.org/pipermail/python-list/2010-June/1247522.html]]. The code itself is merely a small hack, but I hope that by making it available and by explaining how we used it successfully this will dispel some of the Fear, Uncertainty, and Doubt about the idea of mapping your JSON decimal strings to lossless decimal objects in your programming language instead of mapping them to lossy floats.

~Tahoe-LAFS v1.7.0 is here! [[release announcement|http://tahoe-lafs.org/pipermail/tahoe-dev/2010-June/004484.html]]

* added to [[issue tickets closed]]:
** [[twisted #4468|http://twistedmatrix.com/trac/ticket/4468]]: twisted.python.randbytes

[[how to support Python 2 and Python 3 in one code base]]

Blog, blog, blog it all.
I'm sitting in the chairs on the sidewalk outside //The Cup//.
I'm studying ~JavaScript.
Solid State Depot (the Boulder Hackerspace) has an organizational meeting tonight.
Now I'm showing Dan how I update my blog.

Here's [[a nice article about Noisebridge, the San Francisco hackerspace|http://www.wired.com/gadgetlab/2009/03/hackerspaces]]. It is inspiring! By the way, we're having a big meet-up and demo session for the nascent Boulder hackerspace this August 10.

~Tahoe-LAFS v1.8.0c1 (release candidate 1) is now available! [[Release Announcement|http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004926.html]]
//errata: final release is scheduled for Wednesday, August 11, 2010, not August 15 as stated in the announcement//

I updated [[quick reference guide to JavaScript crypto libraries]] to add a couple of new ones that I learned about and to sort the list in descending order of the most important criterion.

[[Flattr me]]

* added to [[things I have read]]:
** David S. H. Rosenthal, Thomas Lipkis, Thomas S. Robertson, and Seth Morabito (the LOCKSS project) [[Transparent Format Migration of Preserved Web Content|http://www.dlib.org/dlib/january05/rosenthal/01rosenthal.html]]<br>Lightweight article, but it is interesting to see what the LOCKSS folks are noodling on. I'm very interested in bit-precise preservation so that people can use a git-like (or ~Tahoe-LAFS-like) directed acyclic graph of history to audit the history and provenance of the data. This, plus using the secure hash of the bitwise contents as the identifier of the contents also gives us a way to verify the contents without LOCKSS's voting protocol, and to update the contents non-destructively (an update is a new version, both the old and new versions will continue to exist indefinitely, and multiple divergent new versions can exist). If we had all of this, then their strategy of "migration" of formats would not be as scary (to me) due to its potential to destroy information.

* ''on air travel without government identification''<br>Standing in line at security, nervous about whether they would have the new "naked picture maker" scanners and if they did if I could decline to enter one without losing my cool or missing my flight. I get out my boarding pass and my... where's my driver's licence?! Oh crap—I must have left my driver's licence on the dresser at home. What's going to happen? It is my turn at the security kiosk.<br>"I lost my driver's licence." I say apologetically.<br>"And you think you oughta go ahead on in!" he says with aplomb. Before I decide how to respond to that, he says "I need some sort of ID, like a credit card, or a picture..."<br>I show him a credit card and my picture-bearing bus pass. He glances at them at waves me through.<br>Huh. Somehow I always imagined that it would be harder than that.

[[my notes on Strandberg 1991]]

I updated [[quick reference guide to JavaScript crypto libraries]]. The ~JavaScript crypto library named [[SJCL|http://crypto.stanford.edu/sjcl/]] has vaulted into first place in my estimation by adding real tests and a real licence (BSD). It already had the distinctions of implementing the most good crypto primitives and of being written by professional cryptographers. They added a note saying that elliptic curve cryptography is "coming soon". Neat!

I would encourage the developers of web browsers to study SJCL and consider how to improve performance, either by optimizing your ~JavaScript implementation to compute SJCL more efficiently, or by implementing native-code implementations of the same algorithms as SJCL provides and offering the same ~JavaScript API to them.

A next-step would be for someone to write a benchmark script that exercises SJCL in a realistic manner: generate key pairs, symmetric-encrypt and hash large strings (representing files), MAC and encrypt many small strings (representing packets), sign and verify small strings, public-key-encrypt and -decrypt small strings.

Added to [[things to read]]:
* Tom Standage: //The Victorian Internet//

I am showing Moxie Marlinspike how I update my blog. Now I will click "save".

* [[how do people share things on the Web?]]
* added to [[things I have read]]:
** interesting short fiction story: //[[A Troll Story|http://nicolagriffith.com/troll.html]]// by Nicola Griffith

The third meeting of ''Project Merely A Good Hack'' was held 2010-12-10 in San Francisco. In attendance were [[Nathan Wilcox|http://twitter.com/lax_wanton_chi]], [[Jonathan Moore|http://0x0000.org]], [[Brian Warner|http://www.lothar.com]], [[Ali Crockett|http://www.myspace.com/alicrockett]] (for part of the meeting), and me. It was a wide-ranging conversation about many topics within Project Merely A Good Hack, adjacent to it, tangent to it, and even distant from it.
Jonathan was the guest conspirator this time around, and he stubbornly challenged our assumptions about what our users will want, ultimately forcing me to recognize that we have to include the availability story as a first-class part of the story in the initial release.
I drew a first draft of The Map Of Paths Not To Take.
Brian and Nathan and I stood around on the sidewalk of Valencia street long after the last coffeeshop (Muddy's) had closed, detailing particulars of how Firefox plugins can interact with SSL and exploring the possibilities of generalizing PMAGH to the current standard certificate hierarchy and DNS instead of only binding to the leaf cert.

Next step: prototype and/or spec?

//''Mom:'' We hung out and had dinner and coffee and chatted happily. Also we're developing an engineering hack to make the Web uncensorable and unspoofable.//

[img[Map of Paths Not To Take|Map_of_Paths_Not_To_Take-thumb.jpg]]

I've been reading //[[The Magicians|http://levgrossman.com/magicians.html]]//, one chapter per night with a warm bath to help me wind down and get ready for bed. This worked well for several nights, but tonight the book suddenly picked up steam and I kept deciding to read one more chapter, one more chapter. Now it is quarter to two in the morning. I think I can read one more chapter before I really do need to get some sleep…

Finished //[[The Magicians|http://levgrossman.com/magicians.html]]// in one long sitting last night, skipping my normal night-time routine of  ''Project Merely A Good Hack'' and ~Tahoe-LAFS. I kind of feel like I shouldn't say anything about //The Magicians// now that I've finished it because anything I could say would be a spoiler.
I still don't know if I love it or hate it, but it definitely passes the litmus test of Real Literature, which is that it has left a persistent effect on my feelings about everything all day long. I'm kind of hoping the effect wears off soon.

[[three books that aren't “Applied Cryptography”]]

hello world

This journal has experienced an outage because the http://tahoebs1.allmydata.com:8123 public web gateway to the Tahoe test grid is down.  Of course, the journal itself is distributed and accessible even when that gateway is broken, but only if people use a different web gateway (ideally, a local web gateway running on their own computer and access with http://127.0.0.1:8123).

But, it is inconvenient to explain that, so I've just been giving people [[a hyperlink going through the tahoebs1 gateway|http://tahoebs1.allmydata.com:8123/uri/URI:DIR2-RO:hgvn7nhforxhfxbx3nbej53qoi:yhbnnuxl4o2hr4sxuocoi735t6lcosdin72axkrcboulfslwbfwq/wiki.html]].  Therefore, when that single point fails, my blog becomes unavailable to them.

Hm...

Got up this morning and checked the status of all the issues which interfere with the automated installation of libraries that Tahoe depends on:

* [[pyflakes #2709 (Pyflakes svn doesn't install properly due to missing packages)|http://divmod.org/trac/ticket/2709]]
* [[nevow #2798 (setup.py install --home is broken :-()|http://divmod.org/trac/ticket/2798]]
* [[nevow #2699 (build nevow without importing nevow)|http://divmod.org/trac/ticket/2699]]
* [[nevow #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way)|http://divmod.org/trac/ticket/2629]]
* [[nevow #2527 (easy_install compatibility)|http://divmod.org/trac/ticket/2527]]
* [[pyopenssl #238658 (please provide binaries)|https://bugs.launchpad.net/pyopenssl/+bug/238658]]
* [[setuptools #53 (respect the PYTHONPATH)|http://bugs.python.org/setuptools/issue53]]
* [[setuptools #54 (be more like distutils with regard to --prefix=)|http://bugs.python.org/setuptools/issue54]]
* [[setuptools_trial #1 (add all trial args that tahoe and/or Brian want to use)|http://allmydata.org/trac/setuptools_trial/ticket/1]]

and

* [[buildbot #212 (buildbot doesn't respond to darcs tags)|http://buildbot.net/trac/ticket/212]]

Grr...  And today the Hack Tahoe! contest is down.

Hm...

I've been reading a lot about hash functions in order to contribute to the ~SHA-3 project.  Here's a good one: [[Merkle–Damgård revisited: How to construct a hash function|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.60.2072]] by ~Jean-Sébastien Coron, Yevgeniy Dodis, Cécile Malinaud, Prashant Puniya.

small bits of hackery:

darcs: I helped Greg H use darcs.  I deleted spam tickets from http://allmydata.org/trac/darcs-2 and posted [[a message|http://lists.osuosl.org/pipermail/darcs-users/2008-July/012587.html]] to the darcs-users mailing list asking if they could link to it from http://darcs.net.  I updated [[darcs ticket#693|http://bugs.darcs.net/issue693]].

my development tools: I've figured out, thanks to Brian Warner's timely help, that the problem with my Linux workstation locking up is [[this driver issue|https://bugs.launchpad.net/ubuntu/+source/linux/+bug/136836]].  I'm compiling the shiny new Linux 2.6.26 kernel to see if that changes the behavior.  (Update: so far so good, kernel 2.6.26 has yet to lock up.  I'm loading the machine up with buildslaves for [[pycryptopp|http://allmydata.org/buildbot-pycryptopp/waterfall]], [[tahoe|http://allmydata.org/buildbot/waterfall?show_events=false]], and eventually also zfec and other projects in order to stress it.)

open source organization: added https://launchpad.net/foolscap and https://launchpad.net/allmydata.org as being related to the https://launchpad.net/tx superproject.

Today I'm going to be pair-programming with Brian Warner from noon to 15:00 and then from 17:00 to 19:00 (Mountain Time).  I'm pleased that Amber's powerful Linux workstation (yukyuk) seems to be stable now that I've upgraded it to Linux 2.6.26.  You can see that it comes in second in [[the builder races|http://allmydata.org/buildbot/waterfall?show_events=false]].  I'll probably use that for the pair-programming instead of my trusty old Toshiba laptop (~Pentium-M 1.8 ~GHz).

This morning I wrote [[a note|http://lists.osuosl.org/pipermail/darcs-users/2008-July/012607.html]] to the darcs-users mailing list about optimizing darcs-2 performance.

Also of course I'm fixing [[the Tahoe builders|http://allmydata.org/buildbot/waterfall?show_events=false]], some of which are failing due to bugs in 3rd party libraries we use and due to issues in how the Tahoe build system manages which versions of the libraries that it builds.

Whenever I'm not fixing the buildslaves or pair-programming with Brian, I intend to be working on fixing up [[my submission to StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]], which is due Sunday, August 3.

Okay, yesterday went pretty well.  I didn't pair program with Brian as much as planned because I instead spent a long time setting up my development environment and remote access to it for Brian on Amber's newly reinvigorated Linux workstation.  We did get an hour and a half of pair-programming, during which time we made some good unit tests for immutable file checking and repair: [[[2813]|http://allmydata.org/trac/tahoe/changeset/2813]], [[[2814]|http://allmydata.org/trac/tahoe/changeset/2814]].

I also improved our packaging, unit tests, and added a builder.  The [[trac Timeline for yesterday|http://allmydata.org/trac/tahoe/timeline?from=2008-07-30&daysback=1&ticket=on&ticket_details=on&changeset=on&milestone=on&wiki=on&update=Update]] shows all changes that anyone committed to the wiki, the ticket database, or the source code yesterday.

I've also been doing some open source "community building" work.  That is: I've been contributing to the community development process that produces [[darcs|http://darcs.net]] and [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]].  We'll see how that works out.

Today I hope to work on [[my submission to StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]], which is due Sunday, August 3.

Whoops, I allowed days to go by without hlog updates.  It is hard to be motivated to write here since I don't know if anyone other than me is reading it.

Perhaps I should just think of it as a personal hack diary -- that would be a useful thing to have even if nobody else read it.

Okay, self, you really need to work on immutable file checking and repair and download ASAP.  Starting with checking.

Amazon ~EC2 service is priced in per-hour increments. If you are going to run a server for only a few hours then it is very cheap to use an on-demand service like ~EC2 on-demand instances. But what if, like [[SimpleGeo|http://simplegeo.com]], you're going to run your server 24/7 all year round?
Here are the current prices, taken from ”http://aws.amazon.com/ec2/pricing/" on 2010-06-16, along with my calculation of how much it would cost to run such a server for one year:
* on-demand
|!instance type|per-year charge|per-hour charge|total cost for 1 year|
|small|-|$0.085|$745.11|
|large|-|$0.34|$2980.44|
* reserved
|!instance type|per-year charge|per-hour charge|total cost for 1 year|
|small|$227.50|$0.03|$490.48|
|large|$910.00|$0.12|$1961.92|
Note that small instances require 32-bit operating systems, unlike the 64-bit that we've grown accustomed to.

[[@imhavoc|http://twitter.com/imhavoc/status/16363490004]] asked about Rackspace's cloud servers. I earlier posted [[a summary of price/performance|2010-01-11]] of ~EC2 vs. Rackspace, copied from [[this article on thebitsource.com|http://www.thebitsource.com/featured-posts/rackspace-cloud-servers-versus-amazon-ec2-performance-analysis]].
Here's an update assuming, without justification, that the performance of these offerings on kernel-compilation tasks hasn't changed even though their pricing has:
* ~EC2 on-demand
|!instance type|per-year charge|per-hour charge|total cost for 1 year|minutes to compile kernel|
|small|-|$0.085|$745.11|24.83|
|large|-|$0.34|$2980.44|5.18|
* ~EC2 reserved
|!instance type|per-year charge|per-hour charge|total cost for 1 year|minutes to compile kernel|
|small|$227.50|$0.03|$490.48|24.83|
|large|$910.00|$0.12|$1961.92|5.18|
* Rackspace
|!instance type|per-year charge|per-hour charge|total cost for 1 year|minutes to compile kernel|
|256M-10G|-|$0.015|$131.49|5.21|
|2G-80G|-|$0.12|$1051.92|3.38|
|8G-320G|-|$0.48|$4207.68|3.28|
Note that I can't attest to the reliability or customer service of Rackspace—I've used Amazon ~EC2 quite a bit with ~SimpleGeo, but I haven't yet used Rackspace. ~EC2 offers good but not perfect reliability and customer service.
Also, of course, you can't generalize from "minutes to compile a linux kernel" to the performance of what you care about. If you are considering deploying a server in the cloud to do something, and what you want your server to do is compile linux kernels, then this data is a very good guide. If your server is going to do some other task then you'd better rent one of each of these (on-demand, it won't cost too much) and measure the performance of your particular app.

//(My mom, and others, have pointed out that my blog is chock full of jargony goodness and they can get only the vaguest idea of what I'm on about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.)//

Last week I saw from [[Simon Phipp's blog|http://blogs.sun.com/webmink]] this interesting project named [[Bespin|https://bespin.mozilla.com]].  It is a programmer's text editor implemented in ~JavaScript and [[the Canvas element|http://en.wikipedia.org/wiki/Canvas_(HTML_element)]].  That same day Tom Lord mentioned Bespin to me in email.  Bespin is exciting stuff!  I really want to hack it to save the files to a [[tahoe-lafs|http://allmydata.org]] grid.  If you saved your files to a tahoe-lafs grid, then you wouldn't have to rely on the server to make sure your file isn't deleted or corrupted, that its contents aren't [[stolen by someone else|http://www.emergentchaos.com/archives/2009/01/breach_misdirection.html]], nor that permission to write to your file was accidentally given to someone else, nor that permission to write to your file was accidentally //denied// to someone with whom you had intended to share it.  (By the way, the list of claims in that sentence should arouse great skepticism in security and distributed systems experts.  On the other hand, webapp authors and users probably paid little attention, thinking to themselves "Well, why ''shouldn't'' it Just Work?".  I love this chaos.)

I had a lot of fun hacking [[TiddlyWiki|http://tiddlywiki.com]] to store itself to a tahoe-lafs grid, with the help of ~TiddlyWiki hackers [[FND|http://fnd.lewcid.org]], [[Eric Shulman|http://about.unamesa.org/Eric+Shulman]], and [[Jeremy Ruston|http://jermolene.com]].  The result was two small ~JavaScript plugins that anyone can add to their ~TiddlyWiki to make their ~TiddlyWiki into a ''gridapp'': [[HTTPSavingPlugin.js|http://allmydata.org/trac/tiddly_on_tahoe/browser/tahoe_tiddly/HTTPSavingPlugin.js]] and [[TahoePlugin.js|http://allmydata.org/trac/tiddly_on_tahoe/browser/tahoe_tiddly/TahoePlugin.js]].

I would love to do something similar for Bespin, but I can tell that I need to focus on the tahoe core right now.  The [[announcement of tahoe-lafs 1.3.0|http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=20090214000500-92b7f-26d07f519c69a3e086ee4599620f6d2c0c9b2fcd]] has encouraged more people to [[start contributing to Tahoe|http://allmydata.org/pipermail/tahoe-dev/2009-February/date.html]], and I'm busy teaching them how to get their patches accepted into the tahoe core.

That said, if any ~JavaScript/Caja/Tahoe/Bespin/~TiddlyWiki hackers out there want to have a go at {{{Bespin+Tahoe}}}, do let me know and I'll give you a little help.  :-)

//''Mom:'' Hm...  This one is pretty hard to translate into English.  I think that in the future new applications such as word processors, spreadsheets, tax software, photo album browsers, email, etc. will live on a web page and you will use them by pointing your web browser at that page.  I think the most fact about this is that the market for software is likely to get more competitive -- a greater marketplace of useful software, easier to use and with better features, that you are able to try out and then adopt. A large part of the motivation for the Tahoe project is to explore whether we can get all those values and //still// let you retain ownership of your own data.//

//{{{
config.options.txtTheme = "WritableTheme";
//}}}

[[Sébastien Martini|http://twitter.com/seb_martini]] mentioned this new Free Software ~JavaScript crypto library: [[jscrypto|http://crypto.stanford.edu/sjcl]].  Looks good!  I've decided to start a quick-reference guide to all the ~JavaScript crypto libraries that I've looked at: [[quick reference guide to JavaScript crypto libraries]].

I've been spending a little bit of time thinking about how gaming services like //World of Warcraft// and (don't-call-it-a-gaming-service) //Second Life// have succeeded where we at ~DigiCash failed -- convenient, widely-used, programmable digital cash.  A problem is that each of these new currencies are centrally controlled by one entity.  This limits the scope of who will rely on that currency and how much value they will risk on that currency.  There are ideas floating around about how to facilitate transactions between currencies, but this would not solve the problem.  //A plethora of competing centralized services is not the same as a decentralized service.//  Even if it were cheap and convenient to trade some ~LindenBucks for some ~WoW Gold, this would only lead us back to the equivalent of the modern nation-state currencies: mostly centralized (because of //the Network Externality//), heavily taxed/regulated/manipulated, and prone to disastrous failure.  What I want is a currency which everyone can cheaply and conveniently use but which ''no-one'' has the power to manipulate.  No-one has the power to inflate or deflate the currency supply, no-one has the power to monitor, tax, or prevent transactions.  Truly the digital equivalent of gold, during the times and places when gold was the universal currency.  See the [[BitGold|http://unenumerated.blogspot.com/2005/12/bit-gold.html]] idea by Nick Szabo and [[b-money|http://www.weidai.com/bmoney.txt]] idea by Wei Dai, and recent effort to actually implement something along these lines: [[BitCoin|http://www.bitcoin.org]] by Satoshi Nakamoto.

[[PyCon 2011]] [[three books that aren't “Applied Cryptography”]] [[2010-12-18]] [[2010-12-16]] [[2010-12-14]] [[2010-12-11]] [[how do people share things on the Web?]] [[2010-12-10]] [[2010-12-08]] [[2010-12-05]] [[2010-11-30]]

In my [[2010-01-04]] journal entry I mentioned the utter insecurity of USB flash drives with hardware encryption and quoted from the magazine article which said: "The real question, however, remains unanswered – how could USB Flash drives that exhibit such a serious security hole be given one of the highest certificates for crypto devices? Even more importantly, perhaps – what is the value of a certification that fails to detect such holes?".
Jack Lloyd replied in private email, and gave me permission to post this part:
<<<
The incentives for FIPS and Common Criteria evaluations are aligned in such a way as to ensure this happens. Evaluation costs are paid for by the vendor, but risks are borne by the customers. This causes a race to the bottom in the evaluation process. It is much the same incentive scheme as the one that caused the debt rating companies to rate junk ~CDOs as AAA, actually.

I spent a while doing FIPS 140 validations. I once found a set of really catastrophic flaws in a hardware device that had previously been certified FIPS 140 Level 3 and Common Criteria ~EAL4. There were buffer overflows in certain test routines which would allow any user (even one with no access privs at all to the device) to take over the device entirely. It is likely the flaws had existed during the previous evaluations. The client was very angry, and threatened to take their business elsewhere if we did not drop the issue. They did not care about the security problems; they just wanted us to write a clean report so they could be re-certified and continue to sell their product to the government.

Ironically, the publically accessible test routines were there specifically because FIPS 140 required them to be there; there was otherwise no reason for them to exist, much less be accessible to all users.

I long ago came to the conclusion that FIPS 140 does not mean anything at all from a security perspective; in fact, I would wager that much of the time it actually limits/reduces the security of the system. For instance it would be quite difficult to get an object-capability design FIPS certified, because FIPS has specific tests that are phrased in terms of user accounts and ~ACLs. (And of course there are obvious things, like that AES and 3DES are the only ciphers you can use in a FIPS design, meaning if you want to use something that is safer, like Serpent or Noekeon or Salsa20, you're out of luck).
<<<

I received the following letter from an anonymous engineer in response to [[FIPS and Common Criteria: don't rely on them]]:
<<<
Your article didn't note the absolute worst thing (in my opinion) about C2/FIPS certification: they prohibit change.  Especially in today's seriously imperfect security technology ecosystem, the security of a system is effectively measured by how quickly it is able to respond to previously unknown attacks.

The basic assumption underlying the brain-damaged design of the C2/FIPS certification process is that it is possible - and in fact, not only possible, but well-understood by all working professionals in the field - to statically verify that a system is completely secure, now and forever, amen.  The challenge, then, is to manage "risks", and to avoid ever changing that perfectly secure system in order to add functionality or some other non-security-related property.  The reason that neither certification means anything is that this assumption is not only wrong, it is actually the polar opposite of the correct ideas; a complete reversal of the process that, in practice, makes the current generation of software technology "secure".

Effective attacks are the ones that can't be anticipated, pretty much by definition.  The thing that ''really'' makes for some cognitive dissonance (and if you have any friends or family in the US military who talk about their work, you'll know what I mean) this is //exactly// the sort of static reasoning that the new generation of officers is trying to eradicate, and replace with a flexible, dynamic response to changing battlefield conditions on the ground.

(If you want to quote me on this I'd appreciate it if you'd make it anonymous, especially if you want to include the term "brain-damaged" ;)).

Thanks for the article.  We really need more people to speak out against the insanely broken way that the US Gov't deals with computer security.
<<<

Look—there is now a little green–and-orange button on the right-hand side of my blog that says "Flattr this!" on it. If you value reading my blog, please click that button! Clicking it will not cost you any money.

If you are already a subscriber to Flattr.com, then you are already paying at least €2.00 per month so that you can give tips to people on the Internet. Clicking this button on my blog will not increase the amount you spend per month, but it will include me in the set of people who get a share of your €2.00 contribution at the end of this month. Cool, isn't it? This is how Flattr overcomes [[The Mental Accounting Barrier to Micropayments|http://szabo.best.vwh.net/micropayments.html]].

I'm re-reading [[Good Calories, Bad Calories|http://amazon.com/exec/obidos/ASIN/1400040787]] by Gary Taubes.  The first time I read it aloud with my wife, Amber, and this time I'm reading it aloud with my brother Josh, who has recently decided to become a statistician.  (Reading it aloud to him seemed to be necessary in order to persuade him to read it, and so far it has worked well -- he is now very interested in the book.)

I've just finished reading Part One (of three) for the second time, and the words that keep coming to mind are "tour de force".  Part One is a beautiful example of investigative journalism in action.  By the traditional method of digging into the facts, reading the original historical sources, and interviewing the actors, and by the less traditional method of reading the relevant scientific literature, Taubes has uncovered a scientific and public policy failure of historic proportions.  Basically, some influential diet researchers managed to turn their hypothesis -- the low-fat diet hypothesis -- into the national policy of the United States of America starting in the late 70's.  The evidence in support of the hypothesis was tenuous, but confirmation bias -- that all too human weakness -- is richly stimulated by narrative flow combined with noisy data such as epidemiological observations, and so politicians and nutrition researchers alike (but not, as Josh points out, statisticians) came to believe that the hypothesis //must// be true, and that future experiments or observations would confirm it.

This is why everyone I know was taught, starting in the 1980's, that we should reduce the amount of fat -- especially saturated fat -- in our diet.

But it turned out that the subsequent experiments and observations did //not// confirm the hypothesis.  The succession of results over the last 30 years casts increasing doubt on the hypothesis.  The evidence now suggests that reducing your saturated fat intake is bad for your health, or at least not good for your health, or if it is helpful then its benefits are so small that they can't be detected.  Indeed there is good reason to believe that the low-fat diet hypothesis and the official policy promulgating it are actually one of the //causes// of the current epidemics of obesity, diabetes, and other problems.  If that is true, then the costs -- in dollars and in lives -- of this mistake are enormous.

In a thread on the "overcoming bias" blog, Hal Finney [[wrote|http://overcomingbias.com/2008/07/gary-taubes-goo.html#comment-124240968]]:

> My problem is, why can't I trust the people I pay to be experts on this topic?
> ...
> Why is it that I, or Gary Taubes, can read the literature and learn the truth, while a professional with far more experience and knowledge than either of us is unable to do so? This is the fundamental paradox I run into again and again on controversial topics.
> ...
> Two possible answers are, A, you can't, so trust the experts; or B, the experts are committed to the conventional wisdom and institutional forces prevent them from acknowledging the truth, while you are free from such prejudices. All I can really say is, I'll be really angry if B turns out to be true. Why should I have to do every damn thing for myself? Why can't I live in a world where people have a reasonable level of competence, where experts actually have expertise? For now, I just hope that A is correct

I think this is one of the most interesting aspects of this book, to me -- a compelling, factual investigation of Hal's question: Why can't we trust the experts on this topic?  I'm afraid Hal is going to be angry.  And rightly so.

It is not a diet book.  (If you need to lose weight, don't read this book.  It makes no specific dietary recommendations and you'll probably find it boring unless you are fascinated by the topic of "Science As She Is Practiced", in which case it is exhilarating.)

One lesson that I draw from this story is the importance of //experiment// in science.  Before //Good Calories, Bad Calories// was written, I had already become interested in this topic.  It all started in the year 1999, when I learned that my then girlfriend, Amber, was following some weird New Age fad diet -- a "low-carbohydrate" diet.  Concerned, I started paying attention whenever I learned about a new clinical trial in which this much-hyped fad diet was tested by experiment.  To my surprise, every experiment yielded results which were inconsistent with the established theory.  I read the abstracts and press releases and news coverage on probably half a dozen clinical trials over the next few years, and usually the scientists who had conducted the experiment were surprised too.  They, or other scientists in the field who were interviewed by journalists, would say something like "This is really weird -- we can't explain why the results are like that.".  (By the way, I still have notes naming those studies if you want them.)  I began to suspect that there was something amiss in the reigning theory.

Further reading, discussion with Amber, and personal experimentation persuaded me that carbohydrates are bad for you, and so when I read //Good Calories, Bad Calories//, the content about nutrition experiments and epidemiology wasn't too surprising.  The lesson that I drew from this is that it is easy to have too much confidence in a hypothesis which is based on ex post facto observation and theory.  For many years now I have found myself in the strange position of knowing almost nothing about biochemistry, human metabolism, or epidemiology, and yet being pretty sure that the experts who did know about these things were incorrect in their conclusions, for the simple reason that I had seen the results of several experiments designed to test their hypothesis, and the results had come out inconsistent with their hypothesis.  (Okay, nowadays I know a tad more about biochemistry and metabolism and epidemiology, almost all thanks to Amber, Josh, and Gary Taubes.)

//Aside: This reminds me of Robin Hanson's [[contribution|http://www.overcomingbias.com/2008/07/gary-taubes-goo.html#comment-124037158]] to the overcoming bias blog thread: "Deviant prediction market estimates here would of course influence me much more [than scientific consensus]".

I think prediction markets might be a way to head off this kind of scientific fiasco.  I can't really justify this idea -- why should prediction markets work any better than the "market" for scientific reputation already does?  Nonetheless, I have a strong feeling that prediction markets would somehow focus more attention on experiment, or on other objective results which are robust against bias or interference, and less on the arts of persuasion and politics which can be used to build scientific consensus.

(Or at least, if the prediction market doesn't have this effect, then maybe I'll get lucky again -- correctly detect a major error long before the established scientists and officials detect it or admit to it -- and make myself rich by speculating.  ;-))//


Okay, so I strongly recommend to all intelligent people who are interested in science, public policy, and/or human nutrition that they read this book.  By the way, I haven't even mentioned Parts Two and Three yet, which contain a wealth of intriguing ideas about such topics as aging (yes, I know it sounds crazy, but there is good reason to believe that low-carb dieting makes certain kinds of aging go slower).  The literary quality of the book drops in Part Three (perhaps there was a time crunch in the process of writing and editing it?) and Taubes repeats himself far too many times in Part Three, saying that the reigning hypothesis of human weight regulation is ''wrong, wrong wrong'' and ''utterly inconsistent with plenty of observed facts''.  This is a shame, because the writing in the first two Parts is skillful and engaging.  I will forgive you for skimming a few pages here and there of Part Three, but I will not forgive you for skipping this book and continuing to have a confident opinion about human nutrition.

Added to [[things I have read]]: Kevin D. Bowers, Ari Juels, Alina Oprea: [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]].
This is a good, thorough design of a ~Tahoe-LAFS-like cryptosystem from some accomplished cryptographers. I personally think that ~Tahoe-LAFS's cryptosystem is better than HAIL for all practical purposes.  (Also, ~Tahoe-LAFS has the major advantage of being an implemented, battle-tested system that has been running in production for years, storing massive amounts of data for real users.) I'll briefly sketch out why I think ~Tahoe-LAFS design is better.
* The important difference is in access control. Symmetric encryption primitives such as the ones HAIL is built out of have the property that when you authorize someone to check the integrity of a file (by giving them the secret key), you also thereby authorize them to change the file.  That's not the behavior we want.  Instead, ~Tahoe-LAFS provides a pair of mechanisms: for immutable files I can create a file and I can authorize you to check its integrity, but nobody (not even me!) can change the file, and for mutable files I can create a file and authorize you to check its integrity while withholding from you the authority to change it.  We implement immutable file integrity using secure hash functions and mutable file integrity using a combination of secure hash functions and digital signatures.
* ~Tahoe-LAFS unifies the download protocol and the "checking up on the storage server" (or "Proof of Retrievability") protocol.  In contrast HAIL, provides a separate "Proof of Retrievability" protocol which is very efficient at checking up on a set of cooperating servers, but impractically slow to use for downloading your files.  The disadvantages of this approach are:
** We're never going to get around to implementing and deploying a secondary protocol which is substantially more complicated than the primary one and which isn't strictly necessary to get our work done.  (I would be surprised if anyone else got around to implementing and deploying such a protocol either.)
** Using a different protocol for download and for ~PoR gives storage servers extra information which malicious ones can use to discriminate among requests.  That is: suppose you want to trick someone into entrusting their data to your storage server but then prevent them from getting it back.  A pretty good attack would be to simply hack your HAIL server so that it continues to participate in the ~PoR protocol, thus convincing the client that it is still holding onto the file correctly, but then it refuses to serve up the file when the client attempts to actually download it.
** In theory the ~PoR protocol has the property that the client can use it to retrieve the actual file but practically speaking I don't expect that to work because (a) there's another layer of complexity that nobody is going to get around to implementing and deploying, and (b) it seems like it would take many years to completely download your file that way -- the ~PoR protocol is a very slow way to download a file.  In contrast the ~Tahoe-LAFS download protocol appears to be a reasonably efficient way to do the ~PoR job while being a near-optimal way to download entire files.
** The HAIL ~PoR protocol seems to require the cooperation of all servers in order to test any of the servers. This means that if a server is unreachable (or if it is maliciously pretending to be unreachable), then you'll just have to wait until it comes back before you can tell whether any servers are correctly holding the files you entrusted to them.  I suppose you could work-around this problem by triggering a full file repair using your own local storage as a stand-in for the missing server, but this would be yet another layer of complexity and performance problems.
* ~Tahoe-LAFS integrates confidentiality (i.e. encryption) along with integrity and availability, which implements all the security properties that we want and presents it all to the user in the form of a simple opaque file handle (a "capability").
In sum, I'm glad that professional cryptographers are thinking carefully about how to gain assurance that data which you've stored on remote servers is still intact and available.  Hopefully the HAIL protocol and the careful analysis that it comes with will inspire further research along these lines.
It is frustrating to me that the authors of HAIL are apparently unaware of ~Tahoe-LAFS, which elegantly solves most of the problems that they set out to solve, which is open source, and which was deployed to the public, storing millions of files, and summarized in a [[peer-reviewed workshop paper|http://allmydata.org/~zooko/lafs.pdf]] before the HAIL paper was published.  I don't know how to get more "exposure" for ~Tahoe-LAFS within academia so that researchers will be able to learn from our successes and mistakes.  I guess the next step is to write a longer, more detailed paper and submit it for inclusion in a more prestigious conference.

/***
|''Name''|HTTPSavingPlugin|
|''Description''|<...>|
|''Author''|Zooko|
|''Contributors''|FND|
|''Version''|0.2.1|
|''Status''|@@experimental@@|
|''Source''|http://allmydata.org/trac/tiddly_on_tahoe|
|''CodeRepository''|http://allmydata.org/source/tiddly_on_tahoe/trunk/|
|''License''|GPLv2+ or TGPPLv1.0+|
|''Keywords''|<...>|
!Description
<...>
!Notes
This plugin is being developed for [[Tiddly on Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]].
***/
/* The following comment is to let jslint know which variables are supposed to be global. */
/*global clearMessage, config, getPath, readOnly, saveChanges, saveTest, showBackstage, store, story, version, convertUriToUTF8, convertUnicodeToFileFormat, getLocalPath, loadRemoteFile, locateStoreArea, saveBackup, saveEmpty, saveFile, saveMain, saveRss, unescape, displayMessage, httpReq */
//{{{
if (!version.extensions.HTTPSavingPlugin) { //# ensure that the plugin is only installed once
	version.extensions.HTTPSavingPlugin = { installed: true };

	(function () { //# wrapper
		readOnly = false;
		config.options.chkHttpReadOnly = false;
		showBackstage = true;

		saveTest = function () {
			var s = document.getElementById("saveTest");
			/*if (s.hasChildNodes()) {
			  alert(config.messages.savedSnapshotError);
			  }*/
			s.appendChild(document.createTextNode("savetest"));
		};

		// Save this TiddlyWiki with the pending changes
		saveChanges = function (onlyIfDirty, tiddlers) {
			var originalPath, localCallback, result;
			if (onlyIfDirty && !store.isDirty()) {
				return;
			}
			clearMessage();
			// Get the URL of the document
			originalPath = getPath(document.location.toString());
			// Load the original file
			localCallback = function (status, context, original, url, xhr) {
				//log("loaded remote file from ", originalPath);
				/*log("got callback status ", status, "\n", context: ", context, "\n",
				  URL: ", url, "\n", XHR: ", xhr);*/
				if (original === null) {
					alert(config.messages.cantSaveError);
					if (store.tiddlerExists(config.messages.saveInstructions)) {
						story.displayTiddler(null, config.messages.saveInstructions);
					}
					return;
				}
				// Locate the storeArea div's
				var posDiv = locateStoreArea(original);
				if (!posDiv) {
					alert(config.messages.invalidFileError.format([originalPath]));
					return;
				}
				saveRss(originalPath);
				saveEmpty(originalPath, original, posDiv);
				saveMain(originalPath, original, posDiv);
			};
			result = loadRemoteFile(originalPath, localCallback);
			//log("result from loadRemoteFile: ", result);
			return true;
		};

		// override and disable saveBackup()
		saveBackup = function (localPath, original) {};

		// override and disable getLocalPath()
		getLocalPath = function (origPath) {};

		// override getPath()
		getPath = function (origPath) {
			var originalPath, argPos, hashPos, resultPath;
			originalPath = convertUriToUTF8(origPath, config.options.txtFileSystemCharSet);
			// Remove any location or query part of the URL
			argPos = originalPath.indexOf("?");
			if (argPos !== -1) {
				originalPath = originalPath.substr(0, argPos);
			}
			hashPos = originalPath.indexOf("#");
			if (hashPos !== -1) {
				originalPath = originalPath.substr(0, hashPos);
			}
			// Convert file://localhost/ to file:///
			if (originalPath.indexOf("file://localhost/") === 0) {
				originalPath = "file://" + originalPath.substr(16);
			}
			// Convert to a native file format
			if (originalPath.indexOf("http://") === 0) { // HTTP file
				resultPath = originalPath;
			} else if (originalPath.charAt(9) === ":") { // PC local file
				resultPath = unescape(originalPath.substr(8)).replace(new RegExp("/", "g"), "\\");
			} else if (originalPath.indexOf("file://///") === 0) { // Firefox PC network file
				resultPath = "\\\\" + unescape(originalPath.substr(10)).replace(new RegExp("/", "g"), "\\");
			} else if (originalPath.indexOf("file:///") === 0) { // *nix local file
				resultPath = unescape(originalPath.substr(7));
			} else if (originalPath.indexOf("file:/") === 0) { // *nix local file
				resultPath = unescape(originalPath.substr(5));
			} else { // PC local file
				resultPath = "\\\\" + unescape(originalPath.substr(7)).replace(new RegExp("/", "g"), "\\");
			}
			return resultPath;
		};

		// override saveFile()
		saveFile = function (fileUrl, content, callb) {
			displayMessage("saving... please wait"); // XXX: belongs into command handler -- TODO: i18n
			//alert("whee! about to save to " + fileUrl);
			var localCallback = function (status, params, responseText, url, xhr) {
				if (!status) {
					displayMessage("saving failed: " + responseText);
				}
			};
			return httpReq("PUT", fileUrl, localCallback, null, null, content, "text/html;charset=utf-8");
		};

		// override convertUnicodeToFileFormat()
		convertUnicodeToFileFormat = function (s)
		{
			return s;
		};

	})(); //# end of wrapper
} //# end of "install only once"
//}}}

My mom gave me a gift certificate for amazon.com for Christmas.  Yay!

What shall I get with it?  I'm considering:
* [[Programming Erlang: Software For a Concurrent World|http://amazon.com/exec/obidos/ASIN/193435600X]] by Joe Armstrong (suggested by [[Myers|http://icepick.info]] -- he's reading it right now)
* [[Security Engineering: A Guide to Building Dependable Distributed Systems|http://amazon.com/exec/obidos/ASIN/0470068523]] by Ross Anderson
* [[Concepts, Techniques, and Models of Computer Programming|http://amazon.com/exec/obidos/ASIN/0262220695]] by Peter Van Roy and Seif Haridi
* [[Hacker's Delight|http://amazon.com/exec/obidos/ASIN/0201914654]] by Henry S. Warren
* [[Real World Haskell|http://amazon.com/exec/obidos/ASIN/0596514980]] by Bryan O'Sullivan, John Goerzen, and Don Stewart
* [[The New School of Information Security|http://amazon.com/exec/obidos/ASIN/0321502787]] by Adam Shostack and Andrew Stewart

Allmydata.com has been sued for patent infringement! [[News Article|http://cloudstorage.wordpress.com]] 

I had a look at the patent in question, [[US Patent 5133065|http://www.wikipatents.com/5133065.html-1]] last night at bed-time.  I didn't really understand what part of the patent was supposed to involve something novel.  Is it just claiming a patent on the idea of backing up your data over the network?  Surely that was done before 1992, wasn't it?

I've faced down patent threats before.  I helped a loose collaboration of peer-to-peer hackers discover prior art on [[Microsoft's patent on convergent encryption|http://www.wikipatents.com/6983365.html-1]]  (although I see that Microsoft is [[still selling licences|http://www.microsoft.com/iplicensing/productDetail.aspx?productTitle=Convergent%20Encryption for convergent encryption]]), and I actually implemented the prior art that destroyed [[Amazon's infamous 1-click patent|http://www.wikipatents.com/5960411.html-1]]. (Actually Branko Lankester implemented the first version of the ~DigiCash shopping cart, and I think his version already had the 1-click feature before he handed it off to me, but it was my version that was used as prior art in the case to invalidate the Amazon patent.)

Does anyone out there know of publications describing backup over a network before 1992, or of any deployed implementations of such backup over a network before 1992?  Please write to me!  Actually, I'm very busy with my new day job and I might not see your mail in time, so please write to [[tahoe-dev@allmydata.org|mailto:tahoe-dev@allmydata.org]] .  Thank you!

''Important Disclaimer: I am not an employee of allmydata.com.  I have not been employed by allmydata.com in a long time.  Nothing that I say about this patent lawsuit should be interpreted as being the opinion or stance of allmydata.com, because I do not work for allmydata.com.  My interest in this case largely stems from my participation in the Free Software / Open Source project [[Tahoe-LAFS|http://allmydata.org]], which software is used by allmydata.com as the basis of their consumer backup product.

This is important, because if I were able to speak for allmydata.com then I would have to be very careful about what I say in public in order to avoid saying something wrong and giving the lawyers which are suing allmydata.com some sort of advantage in court.''

Update: Hal Finney pointed out that patent 5133065 should already be expired!  The transaction history on the USPTO site shows something that seems a bit unusual, that the "case was reported lost" many times starting in December of 1995 and it wasn't found again until August of 1997.  I've seen "case reported lost" transactions on other patent histories, but in the ones that I've seen they were then reported found again on the same day.  I'll bet it is rare for it to take 20 months for the USPTO to "find" the "case" again.  (Maybe the manilla folder containing the patent was misfiled into the wrong metal filing cabinet in the USPTO office?  Is that what things were like there in mid-90's?)  I'm guessing that the USPTO gave the patent holders a term extension to make up for the fact that the patent holders weren't able to exercise it while it was lost.  Is that possible?  How would we check?  I find no mention of a term extension on the USPTO  search engine for patent 5133065.

We have a boy! Anonymous O'Whielacronx, born 2009-10-17 at 17:24, weighing 7 beautiful pounds and 0 ounces. Mother and baby are resting at home and doing fine.

I presented [[Tahoe-LAFS|http://allmydata.org]] at ~CodeCon last weekend.  [[CodeCon|http://www.codecon.org/2009/schedule.html]]'s prime directive is that every presentation has to have a live demo of working code, and that the presenter has to be an author of that code.

For my demo, I leaned an axe against the speaker's podium, strapped safety goggles around my neck, and then I showed three laptops on stage, each running a ~Tahoe-LAFS node, and then uploaded a movie file to the ~Tahoe-LAFS grid made up of those three nodes.  (This means the file gets automatically encrypted, digitally signed, and erasure-coded.)

Then I explained that after uploading your movie to the ~Tahoe-LAFS grid, you might turn off your ~Tahoe-LAFS node and go away.  And while you are gone, something BAD might happen...

<html><object width="425" height="344"><param name="movie" value="http://www.youtube.com/v/ztbIwH7gz7o&hl=de&fs=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/ztbIwH7gz7o&hl=de&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object></html>

KRUSH! KRUSH! I KRUSH YOUR FILE!

(Note that the axe breaks into pieces on the first blow.)

The laptop was donated by Adam Langley. The axe was provided by Brian Warner.  The video recording was provided by Jake Appelbaum.

[[the same video in the freedom-compatible Theora format|http://testgrid.allmydata.org:3567/file/URI%3ACHK%3Auduug6b7m2p4wlvtosozisg2su%3Azy7xex4ocpico53uxex4pb6cpohbuqzpxoym4ioucvlrccgii6ia%3A3%3A10%3A35133367/@@named=/Axe_Versus_Laptop-ffmpeg2theora-audioquality7.ogv]]

<!--{{{-->
<link rel='alternate' type='application/rss+xml' title='RSS' href='wiki.xml' />
<!--}}}-->

<!--{{{-->
<div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'>
<div class='headerShadow'>
<div id='accessControlExplanationDivId' macro='accessControlExplanation'></div>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div class='headerForeground'>
<div id='accessControlExplanationDivId' macro='accessControlExplanation'></div>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
</div>
<div id='sidebar'>
<div id='sideStuff' refresh='content' tiddler='side stuff'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->

Wow, these look like some good talks!
* [[Handling ridiculous amounts of data with probabilistic data structures|http://us.pycon.org/2011/schedule/sessions/222/]] by C. Titus Brown
* [[Panel: Python VMs|http://us.pycon.org/2011/schedule/sessions/14/]] by Brett Cannon, Jacob ~Kaplan-Moss, Maciej Fijalkowski, Dino Viehland
* [["Dude, Where's My RAM?" - A deep dive into how Python uses memory|http://us.pycon.org/2011/schedule/sessions/25/]] by Dave Malcolm
* [[Python-Aware Python|http://us.pycon.org/2011/schedule/sessions/58/]] by Ned Batchelder
* [[Supporting All Versions of Python All The Time With Tox|http://us.pycon.org/2011/schedule/sessions/153/]] by Kumar ~McMillan
* [[Backup Is Hard; Let's Go Shopping|http://us.pycon.org/2011/schedule/sessions/170/]] by Gary Bernhardt (success reports from commercial projects are inevitably bullshit sales presentations and should be avoided. But ''failure'' reports! Now those are treasure troves of useful information.)
* [[Ten Years of Twisted|http://us.pycon.org/2011/schedule/sessions/208/]] by Glyph Lefkowitz
* [[Packaging, from Distutils to Distutils2|http://us.pycon.org/2011/schedule/sessions/81/]] by Tarek Ziadé
* [[Reverse-engineering Ian Bicking's brain: inside pip and virtualenv|http://us.pycon.org/2011/schedule/sessions/198/]] by Carl Meyer
* [[Javascript for people who know Python|http://us.pycon.org/2011/schedule/sessions/41/]] by Ian Bicking
* [[Why is Python slow and how PyPy can help?|http://us.pycon.org/2011/schedule/sessions/160/]] by Maciej Fijalkowski, Alex Gaynor
* [[Python - The Secret Sauce in the Open Cloud|http://us.pycon.org/2011/schedule/sessions/248/]] by Jason Huggins

{{{
From: zooko 
Date: February 5, 2008 12:15:53 PM MST
To: Multiple recipients of list 
Subject: bulk data use cases -- SHA-256 is too slow
}}}

Folks:

Cryptographic hash functions were invented for hashing small variable-length strings, such as human-readable text documents, public keys, or certificates, into tiny fixed-length strings in order to sign them. When considering such usage, the inputs to the hash function are short -- often only hundreds or thousands of bytes, rarely as much as a million bytes. Also, the computational cost of the hash function is likely to be swamped by the computational cost of the public key operation.

Later, hash functions were pressed into service in ~MACs as exemplified by HMAC. In that usage, the inputs to the hash function tend to be small -- typically hundreds of bytes in a network packet. Also, the network is often the limiting factor on performance, in which case the time to compute the MAC is not the performance bottleneck.

I would like to draw your attention to another way that cryptographic hash functions have been pressed into service -- as core security mechanisms in a myriad of bulk data systems. Examples include local filesystems (e.g. {{{ZFS}}} [1]), decentralized filesystems (e.g. a project that I hack on: {{{tahoe-lafs}}} [2]), p2p file-sharing tools (e.g. {{{BitTorrent}}} [3], {{{Bitzi}}} [4]), decentralized revision control tools (e.g. {{{monotone}}} [5], {{{git}}} [6], {{{mercurial}}} [7], {{{darcs}}} [8]), intrusion detection systems (e.g. {{{Samhain}}} [9]), and software package tools (e.g. {{{Microsoft CLR strong names}}} [10], {{{Python setuptools}}} [11], {{{Debian control files}}} [12], {{{Ubuntu system-integrity-check}}} [13]).

Commonly in this third category of uses the size of the data being hashed can be large -- millions, billions or even trillions of bytes at once -- and there is no public key operation or network delay to hide the cost of the hash function. The hash function typically sits squarely on the critical path of certain operations, and the speed of the hash function is the limiting factor for the speed of those operations.

Something else common about these applications are that the designers are cryptographically unsophisticated, compared to designers in the earlier two use cases. It is not uncommon within those communities for the designers to believe that hash collisions are not a problem as long as second pre-image attacks are impossible, or to believe that the natural redundancy and structure of their formats protect them ("only meaningless files can have hash collisions", they say).

A consequence of these conditions is that raw speed of a hash function is very important for adoption in these systems. If you browse the references I've given above, you'll find that ~SHA-1, Tiger, and ~MD5 (!!) are commonly used, and ~SHA-256 is rare. In fact, of all the examples listed above, ~SHA-256 is used only in my own project -- tahoe-lafs. It is available in ZFS, but it is never turned on because it is too slow compared to the alternative non-cryptographic checksum.

I should emphasize that this is not just a matter of legacy -- it is not just that these older hash functions have been "grandfathered in". Legacy is certainly a very important part of it, but newly designed and deployed systems often use ~SHA-1. Linus Torvalds chose to use ~SHA-1 in his newly designed {{{git}}} decentralized revision control tool, //after// the original 2005-02-15 Wang et al. attack was announced, and roundly mocked people who suggested that he choose a more secure alternative [6]. I recently plead with the developers of the {{{darcs}}} revision control tool that they should not use ~SHA-1 for their new, backwards-incompatible design. (The issue currently hangs on whether I can find a sufficiently fast implementation of ~SHA-256 or Tiger with Haskell bindings.)

Because of my exposure to these systems, I was surprised to see a few comments recently on this mailing list that ~SHA-256 is fast enough. My surprise abated when I decided that the commentors are coming from a background where the first two use cases -- public key signatures and ~MACs -- are common, and they may not be aware that ~SHA-256 is potentially too slow for some other use cases.

Regards,

Zooko O'Whielacronx

[1] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Tuning_ZFS_Checksums
[2] http://allmydata.org
[3] http://en.wikipedia.org/wiki/BitTorrent_%28protocol%29
[4] http://bitzi.com/developer/bitprint
[5] http://www.venge.net/mtn-wiki/FutureCryptography
[6] http://www.gelato.unsw.edu.au/archives/git/0506/5299.html
[7] http://www.selenic.com/pipermail/mercurial/2005-August/003832.html
[8] http://www.nabble.com/announcing-darcs-2.0.0pre3-tt15027931.html#a15048993
[9] http://la-samhna.de/samhain/manual/hash-function.html
[10] http://blogs.msdn.com/shawnfa/archive/2005/02/28/382027.aspx
[11] http://peak.telecommunity.com/DevCenter/setuptools
[12] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Files
[13] https://wiki.ubuntu.com/IntegrityCheck

This morning I perused [[the current benchmark results|http://bench.cr.yp.to/results-hash.html]] for the ~SHA-3 contest; (Note that only a few of the ~SHA-3 candidates have yet been benchmarked -- the vast majority of them have yet to appear on these graphs.) Here are [[the graphs on a powerful modern 64-bit Intel chip|http://bench.cr.yp.to/graph-hash/amd64-nmiv003.png]]: //edonr512// and //bmw512// are actually about as efficient as ~MD5 -- hooray!  But here are [[the graphs on a 32-bit PowerPC|http://bench.cr.yp.to/graph-hash/ppc32-nmi0042.png]]: not so encouraging.  Here are [[the graphs on a 64-bit PowerPC|http://bench.cr.yp.to/graph-hash/ppc64-nmi0055.png]]: also not so encouraging.  64-bit PPC is the architecture used in the Sony Playstation 3 and the Microsoft Xbox 360.  32-bit PPC is the architecture used in the Nintendo Wii, which is currently out-selling the other two consoles //combined//.

There is a tendency among programmers, who mainly work on ~PCs, to think of 32-bit architectures as the past and 64-bit as the future.  However, if you look at the numbers, it appears that 32-bit chips are becoming more and more common, not less and less.  In 2008, about 300 million ~PCs were shipped [[[1]|http://news.cnet.com/8301-10784_3-9902716-7.html]].  I guesstimate that about half of those, including the new, fast-growing segment of "netbooks" such as OLPC XO and Asus EEE PC, were 32-bit.  In the same year, //billions// of 32-bit embedded devices were shipped.  The ARM architecture alone shipped 3 billion units in 2008, and other architectures such as MIPS and 32-bit x86 shipped uncounted billions of embedded units.  Many of the newest and fastest-growing products such as the iPhone are 32-bit.

Rather than dying out, 32-bit ~CPUs are getting smaller, cheaper, lower-power, and becoming more and more common and important.

Back in 1997, during the AES contest, a smart fellow accurately predicted "The lesson of the past 20 years of computing seems to be that while the high end always gets better, the low end never goes away. We're still programming tiny 8-bit microprocessors; instead of being used in desktop computers, they're in cellular phones, automobiles, electrical meters, and smart cards. These processors will be around for a long time to come, in ~Dick-Tracy-like wristwatch communicators, small affixable processors, [...] and who knows what else (nanotechnology?)."

The same appears to be true of the 32-bit architectures.

(Other references: that were used in the writing of this article: [[[2]|http://news.cnet.com/8301-13924_3-10076795-64.html]], [[[3]|http://news.cnet.com/8301-10784_3-9902716-7.html]], [[[4]|http://landley.net/ols/ols2007/platforms.txt]], [[[4]|http://arstechnica.com/news.ars/post/20081017-september-npd-numbers-star-wars-trumps-flagging-us-economy.html]], [[[5]|http://www.shacknews.com/onearticle.x/55644]].)

<<search>><<closeAll>><<permaview>><<newTiddler>><<newJournal "YYYY-0MM-0DD">><<saveChanges>><<slider chkSliderOptionsPanel OptionsPanel "options »" "Change TiddlyWiki advanced options">>

a chronological arrangement of Zooko's work/play ; also known as a "klog"

Zooko's Hack Log

http://pubgrid.tahoe-lafs.org/uri/URI:DIR2-RO:ixqhc4kdbjxc7o65xjnveoewym:5x6lwoxghrd5rxhwunzavft2qygfkt27oj3fbxlq4c6p45z5uneq/blog.html

#sideStuff { background-color: #ccc; margin-bottom: 1em; padding: 5px;}
#accessControlExplanationDivId { margin-bottom: 3em;}

After I posted [[Tahoe: the Cloud Storage System that comes with Privacy-Compatibility]], I was gratified to hear "buzz" indicating that people were interested.

I also heard that some people (paranoid security researchers) misunderstood the diagram:

[img[network-and-reliance-topology.png|/file/URI:CHK:clctgzgff4bsyai7ddl2gchuze:qvvg2fcwmqgurevwdpzopfn3bq2yulesbvbmg73uhzentzn4qfaa:3:10:89489/@@named=/network-and-reliance-topology.png]]

These security researchers, reasonably enough, thought of the right-most component, the client, as being what runs locally on your computer and the middle component, the gateway, as being a remote service.  Then they said something like "What's the point!?  Any hacker who takes control of your remote server gets to take over all your files!".

Accordingly I have prepared two new diagrams.

[img[network-and-reliance-topology-paranoid.png|/file/URI:CHK:4rd7ous7b5xgbmpan6mmdbx3za:2jywqfnobreondkanwnekugmxv3cyuzdv34fpyazkb5htjmokdta:3:10:102761/@@named=/network-and-reliance-topology-paranoid.png]]

[img[network-and-reliance-topology-corporate.png|/file/URI:CHK:mpa737uu7suao7lva2axhbtgw4:5rpemho4d3cqsgvgsqmg3hbn2mzeibsbdpthmpyo5jwnj7f2fqfa:3:10:114022
/@@named=/network-and-reliance-topology-corporate.png]]

Both of these deployment scenarios are currently in use.

//''Mom:'' Don't worry, security researchers like it when you call them "paranoid".//

~WHOO-HOO!  Finally, after a seemingly infinite number of nitpicks from Ubuntu developers about packaging and licensing details, [[Tahoe-LAFS|http://allmydata.org]] has been accepted into the Ubuntu Karmic Koala archive!  Here is its web page within the Ubuntu Packaging System: https://launchpad.net/ubuntu/+source/tahoe-lafs .  By the time you read this, you should be able to run "sudo apt-get install tahoe-lafs" and get the latest stable release of ~Tahoe-LAFS installed in your Ubuntu Karmic system!

(You ''are'' running Ubuntu Karmic, right?  Start here: [[Karmic Alpha Testing|http://www.ubuntu.com/testing/karmic/alpha6]] .  Pay no attention to the prominent warnings about Karmic being unreleased development software that might make your computer explode.  What are you, a wimp?)

Many thanks to John Dong, James Westby, and especially Scott Kitterman in addition to [[the numerous people who helped|2009-09-02]] package ~Tahoe-LAFS for Ubuntu in the first place.

[img[network-and-reliance-topology.png|/file/URI:CHK:clctgzgff4bsyai7ddl2gchuze:qvvg2fcwmqgurevwdpzopfn3bq2yulesbvbmg73uhzentzn4qfaa:3:10:89489/@@named=/network-and-reliance-topology.png]]

I updated [[the Tahoe-LAFS introduction|http://allmydata.org/source/tahoe/trunk/docs/about.html]] to include the diagram that shows how ~Tahoe-LAFS is privacy-compatible.  The diagram uses red/black coloration to show which components have the ability to read the contents or change the contents of files.

The point of this diagram is that the storage servers are black, not red.  The user does not rely on the storage servers for confidentiality or integrity.  That's why Tahoe is the only "Cloud storage" system for people who want to delegate the //storage// of their data to remote servers (possibly owned and operated by someone else) without thereby delegating responsibility for the //confidentiality// or //integrity// of their data.

I think this is a big issue and is likely to become bigger over time as "Cloud" computing becomes more widespread.  It is a bit too much to hope that Tahoe will become the universal solution -- after all we're a tiny, volunteer-driven open source project and Tahoe has only a few features -- just barely enough to do what we currently need it to do.  The other Cloud computing systems out there are being developed by billion-dollar corporations and major governments.

However, I do hope that the existence of Tahoe and the fact that it really is deployed and used will convince people that secure, privacy-compatible Cloud storage //is// possible.  It is an option.

(from [[a message I wrote to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-May/001828.html]])

//''Mom:'' I'm trying to get a wider audience of people to appreciate what we've done in the Tahoe project.  Does the text above makes sense to you or does it need translation from programmer-speak into English?//

/***
|''Name''|TahoePlugin|
|''Description''|<...>|
|''Author''|Zooko|
|''Contributors''|FND, EricShulman|
|''Version''|0.2.3.1|
|''Requires''|HTTPSavingPlugin| 
|''Status''|@@experimental@@|
|''Source''|http://allmydata.org/trac/tiddly_on_tahoe|
|''CodeRepository''|http://allmydata.org/source/tiddly_on_tahoe/trunk/|
|''License''|GPLv2+ or TGPPLv1.0+|
|''Keywords''|<...>|
!Description
<...>
!Notes
This plugin is being developed for [[Tiddly on Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]].
***/
//{{{
/* The following comment is to let jslint know which variables are supposed to be global. */
/*global version, readOnly, showBackstage, config, loadRemoteFile, wikify */
if (!version.extensions.TahoePlugin) { //# ensure that the plugin is only installed once
	version.extensions.TahoePlugin = { installed: true };

	(function () { //# wrapper
		var BASE32CHAR, BASE32CHAR_3bits, BASE32CHAR_1bits, SEP, NUMBER, HTTPLEAD, BASE32STR_128bits, BASE32STR_256bits, ALPHANUMERIC_STRING, TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR, TAHOE_FUTURE_READONLY_CAP_RE_STR, TAHOE_FUTURE_WRITABLE_CAP_RE_STR, TAHOE_IMMUTABLE_CAP_RE_STR, TAHOE_READONLY_FILE_CAP_RE_STR, TAHOE_READONLY_DIR_CAP_RE_STR, TAHOE_WRITABLE_FILE_CAP_RE_STR, TAHOE_WRITABLE_DIR_CAP_RE_STR, TAHOE_NONWRITABLE_THING_CAP_RE_STR, TAHOE_WRITABLE_THING_CAP_RE_STR, TAHOE_ANY_CAP_RE_STR, splitTahoeURL, scrapeOutReadonlyCap, diminishToReadonlyCap, getReadonlyURLToThisPage;

		BASE32CHAR = '[abcdefghijklmnopqrstuvwxyz234567]';
		BASE32CHAR_3bits = '[aqiyemu4]';
		BASE32CHAR_1bits = '[aq]';
		SEP = '(?::|%3A)';
		NUMBER = '[0-9]+';
		HTTPLEAD = 'https?://(?:[^:/]+)(?::' + NUMBER + ')?/(uri|file|cap)/?';

		BASE32STR_128bits = '(' + BASE32CHAR + '{25}' + BASE32CHAR_3bits + ')';
		BASE32STR_256bits = '(' + BASE32CHAR + '{51}' + BASE32CHAR_1bits + ')';

		ALPHANUMERIC_STRING = '[A-Za-z0-9]+';

		// This is speculative: maybe in the future there will be a version of Tahoe where caps 
		// start with these symbols, and if so then this JavaScript code will magically work with 
		// that version of Tahoe.
		TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR = "i_" + ALPHANUMERIC_STRING;
		TAHOE_FUTURE_READONLY_CAP_RE_STR = "r_" + ALPHANUMERIC_STRING;
		TAHOE_FUTURE_WRITABLE_CAP_RE_STR = "W_" + ALPHANUMERIC_STRING;

		TAHOE_IMMUTABLE_CAP_RE_STR = "(?:URI" + SEP + "CHK" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits + SEP + NUMBER + SEP + NUMBER + SEP + NUMBER + '|' + TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR + ')';
		TAHOE_READONLY_FILE_CAP_RE_STR = "URI" + SEP + "SSK-RO" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_READONLY_DIR_CAP_RE_STR = "URI" + SEP + "DIR2-RO" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_WRITABLE_FILE_CAP_RE_STR = "URI" + SEP + "SSK" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_WRITABLE_DIR_CAP_RE_STR = "URI" + SEP + "DIR2" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;

		TAHOE_NONWRITABLE_THING_CAP_RE_STR = '(' + TAHOE_READONLY_FILE_CAP_RE_STR + '|' + TAHOE_READONLY_DIR_CAP_RE_STR + '|' + TAHOE_IMMUTABLE_CAP_RE_STR + '|' + TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR + '|' + TAHOE_FUTURE_READONLY_CAP_RE_STR + ')';
		TAHOE_WRITABLE_THING_CAP_RE_STR = '(' + TAHOE_WRITABLE_DIR_CAP_RE_STR + '|' + TAHOE_WRITABLE_FILE_CAP_RE_STR + '|' + TAHOE_FUTURE_WRITABLE_CAP_RE_STR + ')';

		TAHOE_ANY_CAP_RE_STR = '(' + TAHOE_NONWRITABLE_THING_CAP_RE_STR + '|' + TAHOE_WRITABLE_THING_CAP_RE_STR + ')';

		readOnly = document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_NONWRITABLE_THING_CAP_RE_STR));
		showBackstage = !readOnly;
		config.options.chkHttpReadOnly = false;
		
		/* Returns server (which is "http://$HOST:$PORT/uri"), cap, and suffix, which can be a 
		   path from the cap through the tahoe filesystem and/or trailing extra arguments. */
		splitTahoeURL = function (someURL) {
			var u, urlSuffix, candidate_cap, urlPrefix;

			u = someURL.split('/');
			urlSuffix = [];
			candidate_cap = u.pop();
			urlPrefix = u.join('/');
			while ((u.length > 0) && (!urlPrefix.match(new RegExp("^" + HTTPLEAD + "$")))) {
				urlSuffix.unshift(candidate_cap);
				candidate_cap = u.pop();
				urlPrefix = u.join('/');
			}
			// Okay we've found the HTTPLEAD.  Is the following thing shaped like a Tahoe capability?
			if (candidate_cap.match(new RegExp(TAHOE_ANY_CAP_RE_STR))) {
				// Yes!
				return {'urlPrefix': urlPrefix, 'cap': candidate_cap, 'urlSuffix': urlSuffix};
			} else {
				// No!
				return;
			}
		};

		scrapeOutReadonlyCap = function (metadata) {
			// example of tahoe-lafs json-encoded metadata:
			// [
			// "dirnode", 
			// {
			//  "rw_uri": "URI:DIR2:ouojn4oj2fa7fphdf54hz5bfaq:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//  "ro_uri": "URI:DIR2-RO:sznrgoyz7lbjorhe4ipzcnmluy:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//  "children": {
			//   "tw_empty.html": [
			//    "filenode", 
			//    {
			//     "mutable": false, 
			//     "metadata": {
			//      "ctime": 1229263396.69, 
			//      "mtime": 1229263396.69
			//     }, 
			//     "ro_uri": "URI:CHK:cofm2lm3ywu4r4efeqwjzuzyeq:dfw7oi65smf7dhtcx6wvr4ouazswprhwkvc3uopqtmvn3e7cactq:3:10:295520", 
			//     "size": 295520
			//    }
			//   ]
			//  }, 
			//  "mutable": true
			// }
			//]

			// another example:
			// [
			//  "filenode", 
			//  {
			//   "rw_uri": "URI:SSK:ouojn4oj2fa7fphdf54hz5bfaq:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//   "mutable": true, 
			//   "ro_uri": "URI:SSK-RO:sznrgoyz7lbjorhe4ipzcnmluy:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//   "size": "?"
			//  }
			// ]
			var matchobj = metadata.match(new RegExp("^\\s*\\[[^\\[]*\"ro_uri\"\\s*:\\s*\"([^\"]*)\""));
			if (matchobj) {
				return matchobj[1];
			}
		};

		diminishToReadonlyCap = function (urlPrefix, writableCap, callback) {
			var queryURL = [urlPrefix, writableCap, "?t=json"].join("/");

			loadRemoteFile(queryURL, function (success, param, txt, src, xhr) {
				if (success) {
					callback(scrapeOutReadonlyCap(txt));
				}
			}); 
		};
    
		getReadonlyURLToThisPage = function (callback) {
			if (document.location.tahoeDiminishedCapabilityURL) {
				return callback(document.location.tahoeDiminishedCapabilityURL);
			} else {
                                var base = document.location.toString().split('#')[0];
				var pieces = splitTahoeURL(base);
				diminishToReadonlyCap(pieces.urlPrefix, pieces.cap, function (diminishedCap) {
					var diminishedURL = pieces.urlPrefix + "/" + diminishedCap + "/" + pieces.urlSuffix;
					document.location.tahoeDiminishedCapabilityURL = diminishedURL;
					callback(diminishedURL);
				});
			}
		};

		config.macros.accessControlExplanation = {
	
			handler: function (place, macroName, params, wikifier, paramString, tiddler) {
				if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_IMMUTABLE_CAP_RE_STR))) {
					wikify("This is an immutable view of this page.  Using this URL will always show this version of this page, even if a newer version has been uploaded.", place);
				} else if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_NONWRITABLE_THING_CAP_RE_STR))) {
					getReadonlyURLToThisPage(function (readonlyCap) {
						wikify("This is a read-only view of this page.  If you share this URL with someone, they will be able to see the most recent version of this page, but not to change the page. [[Home|" + readonlyCap + "]] -- see the latest posts.", place);
					});
				} else if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_WRITABLE_THING_CAP_RE_STR))) {
					getReadonlyURLToThisPage(function (readonlyCap) {
						wikify("This is a writable view of this page.  If you share this URL with someone, they will be able to change this page.  Click here for a [[read-only view of this page|" + readonlyCap + "]].", place);
					});
				} else {
					wikify("You are accessing this page not through the Tahoe-LAFS secure, distributed filesystem.", place);
				}
			}
		};
	})(); //# end of wrapper
} //# end of "install only once"
//}}}

Several people have publicly and privately voiced their discontent with the way that the Open Source Initiative is handling [[my Request For Approval|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:sss:478:200901:ofpndmgcgmbhbmimpkpe#b]] for [[The Transitive Grace Period Public Licence|http://zooko.com/tgppl.pdf]] and [[my accusation of wrongdoing|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:491:200901:hnghfggnikagbelnanba]].  I'm glad that other people are paying attention because there is an issue at stake which is wider than the TGPPL itself -- it has to do with the OSI's role in the Open Source/Free Software community.

The OSI is trying to exercise a general authority to encourage or suppress the use of open source licences.  That's fine -- perhaps they //should// have that authority.  But if so, it needs to be explicitly stated and transparently executed so that the wider community understands what they are doing and what is meant by the phrase "~OSI-approved".  Currently they are trying to accomplish their goal of discriminating //among// open source licences by overusing their community-sanctioned authority to judge //whether// a licence is open source.  That is, they are refusing to formally recognize a licence (the TGPPL) as ~Open-Source-Definition-conformant even though they admit that it //is// ~OSD-conformant, because they wish to reduce the number of open source licences in use (the "licence proliferation problem").  Since this is inconsistent with the community's expectations and with the OSI's own documentation, it is wrong.

One particularly frustrating thing about this is that OSI [[has already initiated|https://ideas.opensource.org/attachment/ticket/157/email]] and is [[currently working on|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:msn:491:hnghfggnikagbelnanba]] a plan to separate out the question of whether a licence is open source from the question of whether they recommend it.  They could go ahead and formally acknowledge what they have already publicly admitted -- that the TGPPL is an open source licence -- and proceed with their plan to set up an explicit, transparent, and community-sanctioned process for formalizing their broader judgments.  This would protect such trust and confidence as the people vest in OSI.  I have already had several well-known members of the community tell me privately that they recommend simply ignoring the Open Source Initiative.  That's too bad, and it is unnecessary.

This klog is insecure.

Last night while saying "Good-bye" to Trevor Stone as he left Game Night At The O'Whielacronxes House, I mentioned to him that I was now I was now linking to his blog from mine.  (He replied "Oh.  I didn't know you had a blog.")

A moment later I realized that my klog is insecure.  I realized that there are a few people that have the ability to change its contents, to whom I hadn't intended to grant that ability.  Sigh.  Well, learning about things like this is why I use {{{tiddly_on_tahoe}}} for this klog (see [[about this klog]]).  Computer security is often not about math, but about [[human-computer interaction|http://en.wikipedia.org/wiki/Human-computer_interaction]], and therefore many important questions about not theoretical questions best answered with proofs but empirical questions best answered by observation and experiment.  This klog is an experiment (a barely controlled one), and it has now delivered its second useful result to me.  Hopefully I'll write up my conclusions for [[the tahoe-dev list|http://allmydata.org/pipermail/tahoe-dev]], but for now it is much more urgent that I finish release Tahoe-1.3.0.
added to [[things to read]]:
* [[Usable Security Blog|http://usablesecurity.com]]
P.S.  If any of those people to whom I've accidentally given write-access to my klog read this, figure out what I meant, //and// actually exercise that access to edit this page without my help, then I'll praise them and buy them a congratulatory drink.  If you're not one of those people, you'll require the cooperation of one of those people in order to get write access to my klog.  I'll say no more, for now.

My cousin Paul sent me this link: [[Insurgents Hack U.S. Drones|http://online.wsj.com/article/SB126102247889095011.html?mod=yhoofront]].  It turns out the Predator drone  -- a remote-control flying weapon which is apparently a linchpin of the U.S. strategy in Iraq and Afghanistan -- sends its video feed down in cleartext, and anti-Occupation forces in Iraq and Afghanistan have been watching the video feed from drones in real-time.  Various U.S. officials had the following excuses to make:
 * //"There's been no harm done to troops or missions compromised as a result of it."// Baloney. How could he know that?  I'll bet there has been plenty of harm done.
 * //"There are inherent risks to using drones since they are remotely controlled and need to send and receive video and other data over great distances... Those kinds of things are subject to listening and exploitation"// No kidding! That's why militaries have been using this thing called "encryption" for the last two thousand years or so.
 * //"Some of its communications technology is proprietary, so widely used encryption systems aren't readily compatible."// Sounds like someone is planning to charge the ~DoD a ton of money to fix it. Yes, if the video downlink isn't streamed over some standard like RTP, then indeed, you can't simply secure it by adding in some other standard such as DTLS or ZRTP (plug: I helped invent ZRTP), but on the other hand the craft of adding strong encryption to a protocol is well-understood these days and I doubt that there is anything special about this protocol that makes it harder to encrypt than normal.  This sounds like a bog-standard crypto problem that is covered in modern university courses on cryptography.
 * //"Fixing the security gap would have caused delays, according to current and former military officials. It would have added to the Predator's price."// Okay I can't argue with that.  Except hey wait don't Predator drones cost $4.5M apiece?  We Internet hackers fix security gaps for a much smaller price tag.
 * //"Some officials worried that adding encryption would make it harder to quickly share time-sensitive data within the U.S. military, and with allies."// Ah now this one is interesting. Some Internet hackers have the motto that "every protocol and every transmission should be encrypted". This makes sense, ''but only if sharing the decryption keys is as easy and convenient as sharing the cleartext itself would have been''.  The current state of the art of key-management is complex and inconvenient, undermining this principle and making it wiser in many cases to skip encryption when possible in order to avoid its attendant key-management headaches. And then Iraqi insurgents sitting in a hut watch through your Predator's eyes.  (Ian Grigg points out that ease-of-use is "Kerckhoff's Other Principle" -- Kerckhoff's sixth rule of good crypto design.)  So convenient key-management is the crux of the issue (plug: my current project [[Tahoe-LAFS|http://allmydata.org]] is attempting to push forward the boundaries of that problem).
 * //"The U.S. government has known about the flaw since the U.S. campaign in Bosnia in the 1990s, current and former officials said. But the Pentagon assumed local adversaries wouldn't know how to exploit it."// Ah, there's the real story!  This is a familiar story in the history of cryptography and security.  They assumed that "local adversaries" wouldn't know how to exploit it.  This goes along with the "blame Iran" theme that you can find in the article.  They seem to think of the nearby nation-state, Iran, as being the real enemy; sophisticated enough to think of tuning a $99 satellite TV receiver and $25.95 software to the Predator's downlink frequency.  These "local adversaries" sound like rubes -- tribesmen or back-alley thugs who are easily outwitted and require help from a nation-state.  I'm not so sure though -- those people are literate and have the Internet, so you can't draw a perimeter around their knowledge and skills.  (See [[John Robb's|http://globalguerrillas.typepad.com/]] notion of "Open Source Warfare".)
Of course there are overarching political reasons to insinuate, as this article does without any evidence, that Iran must be behind it.  For one thing, the U.S. military can't really "win" against an open source insurgency.  How do you "win" against a something that is so decentralized it isn't even an "organization", it isn't even a "network", it's almost just a "culture" of hating the occupiers and of knowing how to learn the skills you need to put that hate into practice.  On the other hand it would be very easy for the U.S. military to "win" against the government of Iran. They must really wish that they could turn their attention from these ineradicable, decentralized insurgents to a nice fat centralized power structure against which they can "win".
The final paragraph from the article: //"Today, the Air Force is buying hundreds of Reaper drones, a newer model, whose video feeds could be intercepted in much the same way as with the Predators, according to people familiar with the matter. A Reaper costs between $10 million and $12 million each and is faster and better armed than the Predator. General Atomics expects the Air Force to buy as many as 375 Reapers."//

|StyleSheet|##AuthorStyles|
|StyleSheetReadOnly|##ReaderStyles|

!AuthorStyles
/*{{{*/
[[StyleSheet]]
body {
	background: #eee;
}
/*}}}*/

!ReaderStyles
/*{{{*/
[[StyleSheet]]
body {
}
/*}}}*/

http://xkcd.com/428

This klog is a [[TiddlyWiki|http://tiddlywiki.com]] which is stored on a ~Tahoe-LAFS decentralized filesystem.  That means it is a //decentralized web app// or an //Unhosted App//.  Is that not cool?  I think that's awesome.  This also means, as Zandr informed me, that this decentralized version of ~TiddlyWiki Just Works on the iPhone, unlike the normal ~TiddlyWiki.

If this klog doesn't satisfy your thirst to know what Zooko is up to, you can also peruse [[the SimpleGeo Engineering Blog|http://developers.simplegeo.com/]], [[the tahoe-lafs.org Timeline|http://tahoe-lafs.org/trac/tahoe/timeline]] and the [[tahoe-dev|http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev]] mailing list. Also [[my account on twitter|http://twitter.com/zooko]].

You can have a ~TiddlyWiki-on-Tahoe-LAFS of your very own -- just follow [[these instructions|http://tahoe-lafs.org/trac/tiddly_on_tahoe]].

I finished //[[The Steel Remains|http://books.google.com/books?id=AsRF3s1MK08C&dq=the+steel+remains&printsec=frontcover&source=bl&ots=keAI_8SMcl&sig=DKVlGa63-Q0r9pI5PLsQKGNQnLI&hl=en&ei=mrOSSs2xAonuswOUo4wM&sa=X&oi=book_result&ct=result&resnum=8#v=onepage&q=&f=false]]// by Richard K. Morgan in two sittings plus on the bus to work plus lunch break and coffee break at work yesterday.

It was very exciting -- a fast-paced, skillfully woven fantasy novel in the good old classic style -- brave warriors, exotic cultures, decadent emperors, clashing armies, villainous slavers, cunning intrigues, and mysterious magical fairies/demons/gods. These well-worn elements spring to fresh life in expert hands, and there is a twist to it -- one immediately gets the strong suspicion that the vanished Elder Race whose powerful artifacts are left lying around were actually technologically advanced travellers who crash-landed on this planet. ("Sorcery or science," the Emperor lazily declares early on, "I never can remember which is which.") This weaves into the plot and, along with the modern vernacular and gritty "hard-boiled" style, make it seem almost as realistic as a science fiction story.

Oh, and the brave warrior protagonist is gay and spends at least as much time thinking about and having sex with men as most fantasy novel protagonists spend thinking about and having sex with women.  In fact he spends quite a bit of his time on that activity and this too becomes a core strand of the plot.  (The author also throws in one heterosexual sex scene, one near-miss on a lesbian sex scene, and one horrific demonic rape/torture/snuff scene, just to make sure no-one feels left out.)

The gay sex is quite explicit, but I don't recommend that you avoid this book because you're averse to reading about gay sex. I recommend that you avoid this book because you're averse to reading many long episodes of brutal, bloody, gory violence.  If detailed descriptions of which body parts are getting viciously hacked off of which characters isn't your cup of tea, then this is not the book for you. I imagine the author imagining a reader who is a bit uptight, perhaps a little bit disgusted by the notion of homosexual intimacy.  "Phewf," the reader says, putting down the book, "After the hellish rape and murder and horrors of war that I've just been through, I must say that the anal sex between consenting adults really wasn't that bad.".

That said, if you don't mind sex and violence in your fantasy novels, and you value intelligence, creativity, and skillful writing, then I recommend this book.

A well-engineered cryptosystem should have a "security parameter", //K//, such that cracking any of its security guarantees by brute force should take at least //2^^K^^// expense on the part of the attacker.

//Aside: Cryptographers differ on the definition of "expense".  The most common definition is "computation required" in the sense of computational complexity theory, but a better definition, championed by Daniel J. Bernstein and Sean O'Neill among others, includes memory and communication as well.  In my opinion the proper definition of "expense" is "expense"!  You know: money, or the equivalent of money that they use in the future.  Basically -- the sum total of all things that you have to forego in order to perform this brute force attack (which economists call "opportunity cost").  The only reason anyone actually cares about the traditional computational-complexity-theory definition in practice is because it happens to be a good proxy for real monetary cost.//

A well-designed cryptosystem will certainly not have any points of weakness which can be brute-forced for less than //2^^K^^// expense.  In addition, a well-designed cryptosystem ought not pay an engineering cost for "extra strong" points which require much more than //2^^K^^// to brute force.  By "an engineering cost" I mean either a cost in runtime resources such as computation or storage or communications or a cost in complexity of the design or difficulty of implementation.  That is: you sometimes "accidentally" end up with crypto values which are stronger than necessary.  For example you might have two secrets, each of which can be used independently and thus needs to be at least //K// bits long, but then at one point in the design you combine both of them which means an attacker would have to guess all //2*K// bits in order to accomplish a brute force attack.  Whether you keep these //2*K// bits of secret or whether you boil them down into a single //K// bits of secret should be determined solely by engineering considerations -- letting this particular point in the design have a crypto strength of //2*K// should not be considered any more valuable than letting it having a crypto strength of //K//.  (Because other points in the design have a strength of only //K//, and because //K// should be chosen to be large enough that it is more than enough security -- it provides an adequate "security margin".)

The current crypto system in ~Tahoe-LAFS v1 doesn't quite live up to these noble goals of good crypto engineering.  It uses 256-bit secure hash values and 128-bit symmetric encryption keys (both of which equate to a security parameter of //128//, but the RSA public key signatures use 2048-bit RSA keys which cryptographers nowadays estimate to have a security parameter somewhere between //88// and //103//. (A collection of cryptographers' estimates and recommendations about key lengths is visible on [[this handy interactive web page|http://www.keylength.com]]. ) (At the same time 2048-bit RSA key generation is a significant performance problem.)

So what security parameter should we use when designing the next generation of ~Tahoe-LAFS crypto system?  We want it to be secure against even the most powerful attacker that exists today, as well as to remain secure for a very long time into the future -- for decades -- maybe for a hundred years!  That goal of long-term security makes ~Tahoe-LAFS's job much harder than the job of communications-protecting tools such as SSL, for which people assume (perhaps wrongly) that if the algorithms used today are cracked in a decade or two that the ciphertext will long since have been lost or the message content will long since have ceased to be important.  The timespan for people to cease caring about the confidentiality or integrity of their stored documents appears to be measured in decades, at least.

In addition to long-term security, we want people to be able to predict how likely it is that the crypto system will turn out to be breakable as time passes!  I.e., if the crypto system turns out to be breakable in 2040 due to a combination of dropping cost of computation (and memory and communication) and improvements in cryptanalysis then we would much prefer it if it seemed ''likely'' to people in 2030 that the system was going to turn out to be breakable. We don't want it to seem to people in 2010, in 2030, and in 2039 that the system was strong, but then have them be surprised when it turned out to be weak in 2040.

That means using algorithms that are widely studied (or will be widely studied in the coming decades), because algorithms that are widely studied reveal their weaknesses in many small revelations over a long period of time, where those that are little-studied may reveal a catastrophic weakness in a single revelation.  It also means using combinations (called cascades) of two algorithms, because then the news that one of the algorithms has proven to be weak puts you on alert that the whole cascade may turn out to be weak.

//Aside: it ''would'' also mean using algorithms that will still be secure even if it turns out that someone has built a large quantum computer.  Unfortunately cryptographers have not yet worked out the details of [[post-quantum cryptography|http://pqcrypto.org]] well enough that we could actually implement and rely on it.  Please, oh great Cryptographers, hurry up and develop a good post-quantum public key cryptosystem and study it widely and define it in an implementable level of detail and produce lots of high quality implementations and standardize it! Here's a neat new public-key crypto idea which is post-quantum: [[Public-Key Cryptographic Primitives Provably as Secure as Subset Sum|http://eprint.iacr.org/2009/576]].  Anybody that breaks this scheme also proves that ''P=NP''!  That's a very nice reason to believe that nobody will break it anytime soon.  It is not the only crypto system that can claim that, but it is a neat one.//

Another way to give the users more warning that the system may turn out to be weak is to have a high security parameter.  As long as there are no major cryptanalytic breakthroughs, a security parameter of //256// is not really any more secure than a security parameter of //128//.  (See [[this|http://www.mail-archive.com/cryptography@metzdowd.com/msg10764.html]], [[this|http://www.mail-archive.com/cryptography@metzdowd.com/msg10776.html]], and [[this|http://www.mail-archive.com/cryptography@metzdowd.com/msg10749.html]] for examples of why I say that.)  But if there ''are'' major cryptanalytic breakthroughs against the algorithms we rely on, then a security parameter circa 2010 of //128// could turn out to be insecure, or become ''likely'' to turn out to be insecure in coming decades, while a security parameter circa 2010 of //256//, or even of //148//, may remain likely to be secure. 

Okay so all these factors argue for choosing a high security parameter.  Why not just choose a huge one?  Well, computational cost is certainly a factor, especially on the 32-bit ARM architecture that is increasingly important for embedded devices, ~Network-Attached-Storage devices, and low-power-and-low-cost servers.  See benchmarks of [[hash functions|http://bench.cr.yp.to/results-hash.html#arm-apollo]] and [[ciphers|http://bench.cr.yp.to/results-stream.html#arm-apollo]] on a 32-bit ARM CPU, and think that the cost of spending a lot of CPU cycles is that it drains power even if it isn't a speed bottleneck.  This shows why we can't use ~SHA-512.

But the more important consideration for me is the size of the caps.  Some uses of caps, especially embedding them into ~URLs, impose a usability cost on the user if they are too large.  See [[Tahoe-LAFS ticket #882|http://allmydata.org/trac/tahoe/ticket/882]] for a litany of how the current ~Tahoe-LAFS caps-in-~URLs, which look like this: http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html are unusable, even by computer programmers.

If we're going to keep caps small, this imposes an upper bound on the security parameter.  For example, if a cap to an immutable file were to be as small as this: {{{lafs:fi-072Og6e75IOP9ZZsbR1Twjs6X5xXJnBAF}}} then we couldn't have a security parameter higher than //96//.  These two goals: performance and usability, make me want to choose the ''smallest'' security parameter which will also satisfy the goals described above for long-term, predictable security.

One possibly future solution to the usability issue would be to use a larger character set than ascii and the base-62 encoding that these examples above use.  I've been playing with this tiny-url service named http://tinyarro.ws a.k.a. http://➡.ws .  It produces tiny-urls like this one with random unicode chars: [[http://➡.ws/૜釟|http://➡.ws/૜釟]] .  So far it seems like such ~URLs are usable!  I've been handing them out to lots of people in twitter, email, IRC, and IM for weeks now, and so far nobody has told me that they couldn't follow them or that they were stuck because they needed to type the ~URLs in or read it to someone over the phone.  People apparently use ~URLs primarily by clicking or cut-and-pasting, and these ~URLs apparently work for them.  (Shawn Willden also suggested this idea a few months back.)

If such an encoding would work for ~Tahoe-LAFS ~URLs, then we could have ~URLs that look something like this: {{{lafs:fi-蜔쳨欝遃䝦舜琇襇邤䍏㵦☚✸킾궑蒴}}} and have a security parameter of //128// or {{{lafs:fi-蜔쳨欝遃䝦舜琇襇邤䍏㵦☚✸킾궑蒴犏띎냔㳆㼿졨浴䒉ΐ屝稜퍙鉧迴}}} and have a security parameter of //256//.  My "in the wild" usability experimentation (above) so far suggests that those would be much more usable than the ascii base-62 encoded equivalents.  (Those examples are assuming that we can fit 16 bits into each character, which seems reasonable.  Perhaps we could even squeeze in more, depending in part on whether we care whether two different caps are visually distinguishable from one another.)

Okay, so the quesion is: what security parameter should we use in the next generation of ~Tahoe-LAFS?  I've been waffling back and forth for months between //128// and //96//, but ~David-Sarah Hopwood [[posted|http://allmydata.org/trac/tahoe/wiki/NewMutableEncodingDesign]] that they think //96// is too small.  ~David-Sarah, please tell why you think so.  If the unicode-~URLs hack works then I could go for //128// or even //256//.  We don't have a secure hash function with a //256//-bit security parameter (//512//-bit digest size) that can run reasonably efficiently on 32-bit ARM ~CPUs and is also old and widely studied, but we could use a cascade of, for example, ~SHA-256 and Blake-64 with Blake-64 as the outermost function.  That would produce //512//-bit outputs and would ''probably'' require both a dramatic breakthrough in the cryptanalysis of ~SHA-256 ''and'' a dramatic breakthrough in the cryptanalysis of Blake-64 to crack.

Okay, after writing all this down and thinking it through I now think that //128// is the right security parameter for the next generation of ~Tahoe-LAFS.  It is large enough that we have a good chance it won't turn out to be weak for decades or even longer, without being so large that it gets us into usability and performance problems that require creative new solutions.  //128//-bit is considered the minimum standard security level by most people nowadays, so going below that would invite skepticism and incur risk.  Now that I have the promise of someday solving the usability problems with improved encoding or other hacks, I think I can finally let go of my idea of deploying //96//-bit security.

Oh, one final comment: a lot of the confidence that I have in the long-term and predictable security of this envisioned crypto system is because of the cascades.  We know how to build good cascades [[of hash functions|http://www.mail-archive.com/cryptography@metzdowd.com/msg11041.html]] and [[of stream ciphers|http://www.mail-archive.com/tahoe-dev@allmydata.org/msg02611.html]].  What about the third and last cryptographic primitive that we currently need: digital signatures?  :-)

Here's a good candidate for a companion digital signature scheme: [[Efficient Signature Schemes with Tight Reductions to the Diffie-Hellman Problems|http://www.cs.umd.edu/~jkatz/papers/dh-sigs-full.pdf]] (scheme 1 in this paper, which is reasonably efficient, is proven to be as hard as the Computational ~Diffie-Hellman problem and is based on a scheme proposed by David Chaum and H. van Antwerpen twenty years ago in 1990).  This paper was recommended to us by Paul Crowley.

On the tahoe-lafs wiki we have an overview of possible [[new mutable cap designs|http://allmydata.org/trac/tahoe/wiki/NewMutableEncodingDesign]].  There are currently three general types of candidate designs: "Simple", "~Semi-Private Keys", and "~ElkPoint".  (Also ~David-Sarah Hopwood has leaked the fact that they are inventing a new improved design codenamed "Rainhill" but I believe it will be similar to ~ElkPoint for the purposes of this paragraph.)  If we use the semi-private keys idea for mutable files then we will not be able to cascade two digital signature schemes, as far as I can see, without doubling the size of the read-caps.  If we use either Simple or ~ElkPoint/Rainhill then there is a simple and natural way to cascade two signature schemes together -- simply take the hash of both public keys instead of the hash of one public key to generate the read-cap, and include both signatures in the ciphertext.

Here are some assorted documents about technical failures.  I'm mainly using this as a permanent repository of this information for my own future reference, since the original sources themselves, as well as their context, will probably be lost in a couple of years.  Obviously there is a lot more that I could do for this collection: better and more thorough acquisition of failure "war stories" from the wild (which reminds me that I ought to subscribe to RISKS digest again), better curating of this collection (i.e. explaining the context of each one, and analyzing them), etc..
So far the first two have to do with the way that tiny little details about clocks can lead to bigger failures.  I think one of the reasons that I am paying attention to these stories is that Brian doesn't entirely share my intuitions about safety engineering and clocks.  :-)
* The cron job for the Debian/Ubuntu update tool "apt" can get stuck, depending on when Daylight Savings Time happens in your timezone and time of day that the cron job starts and completes: [[apt lost progress due to Daylight Savings Time|../../file/URI%3ACHK%3Axuwyk7jji7nirgwmef3mhdpr74%3A6ba7od2ydryediodgzcstfikzb7caty6drapuozmcj7yxwliztxa%3A3%3A10%3A15331/@@named=/bugreport.cgi%3Fbug%3D523213.html]]
* Every Zune in the world locked up on December 31, 2008: [[Zune lock up on December 31, 2008|../../file/URI%3ACHK%3A27dycwcdsncqcdeiewmarhyqf4%3Aswt2xah4mfzd35djubfdoq7kgxa5g2izehnxnmsuw2zhnbflapca%3A3%3A10%3A278416/@@named=/38143-cause-zune-30-leapyear-problem-isolated.html]]
Here is [[the tahoe dir|http://testgrid.allmydata.com:3567/uri/URI:DIR2-RO:h7pjn6prggf37p4of47b5ustb4:h7vf55ularzdwnoyp6hi5bqaka4wixstnhlz43xbaowohz3upnwa]] where I am keeping these.

//My mom and others have pointed out that my blog is chock full of acronymy goodness and they can get only the vaguest idea of what I'm talking about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.//

My name is "Zooko ~Wilcox-O'Hearn". Please address correspondence to:
<<<
Zooko ~Wilcox-O'Hearn
3450 Emerson Ave.
Boulder, CO, USA
80305-6452
<<<

(I wrote this as a response to Joseph Miklojcik's blog entry [[Darcs' Theory of Patches Isn't Useful|http://jfm3-repl.blogspot.com/2009/01/darcs-theory-of-patches-isnt-useful.html]].)

Hi!  I'm a darcs-lover.  Thanks for writing this blog entry.

Your criticism of darcs is a common one.  It is nice that you write a concrete example -- that helps focus the mind.

The only thing that I would disagree with you about is your suggestion that darcs is //intended// to somehow detect or solve the semantic issues that you illustrate, or that darcs's failure to do so means that darcs isn't useful.

I've been struggling for a while now to articulate how it is that darcs can use more information than git does in computing its merges, and compute correct answers to more interesting questions about the source code, without actually attempting to reach all the way up and answer questions about the semantics of the program (or other text or data) which is being merged.

To someone who is used to cvs/svn/git style merge, darcs's merge seems to be doing something magical, and suspiciously close to "guessing what the user meant". Of course it isn't actually doing either of these things. (To place blame where due, darcs's merge algorithm is woefully under-documented and is mixed in with a whole bunch of other ideas, so if the git users fail to understand what darcs is doing, this is mostly darcs's fault for not explaining it properly.  I expect that eventually someone will either steal or rediscover the simple idea of darcs merge and integrate it into a different revision control tool -- possibly git.)

So what is this "simple idea"?  It's like this: suppose you've written a patch on your branch which changes a line of code in a file.  Now suppose you give your patch to someone else, but over in their branch that file has moved.  The simple idea is that //your patch should be applied to the right file//.  It should not be applied to "whatever file is currently located at that pathname in the other branch", and it should not be applied to "whatever file has contents like the contents of the original file".  It should be applied to ''that file'', even if the file has moved locations or changed its contents.  This is what users naively expect the revision control tool to do, and the revision control tool is able to do it without any magic or "guessing what the user meant", and it is more useful than the alternatives, so that's what should be done.

Tthis same simple idea can be used to determine "which lines of code within a file should this hunk apply to", just as "which file within the tree should this patch apply to".  Here is my concrete example: [[badmerge/simple.html|https://zooko.com/badmerge/simple.html]].

I hope that you can see how an algorithm which computes an answer to this question: "where did the file move to in this tree" or "where did the line move to in this file" can be reliable and useful without attempting to interpret the meaning of C source code (nor of HTML or English text if that's what you use your revision control tool for).

Note for nitpickers: yes, in order to correctly compute "where did the file move to in this tree", darcs needs to be explicitly told which actions are file mv's and which are deletion of the old file followed by creation of a new file (with suspiciously similar contents) in a different location.  Some nitpickers have argued that explicitly writing {{{darcs mv file new/path/to/file}}} is too much to ask of users and that instead the source control tool is going to have to figure out what happened after the fact once the user has run {{{mv file new/path/to/file}}}.  When they were arguing this in the year 2000, I agreed with them that this might turn out to be a serious problem, but in the ensuing years it has turned out that the vast majority of darcs users are happy to write {{{darcs mv file new/path/to/file}}}, and that on the occasions that they forget to do so the consequences are not too bad, so in 2009, this argument is wrong and should not be repeated.  There is a similar but subtler nitpick concerning which-lines-in-a-file but I'm going to stop here.

("OSD" stands for the [["Open Source Definition"|http://www.opensource.org/docs/osd]].)

From: Zooko
To: the Open Source Initiative
Date: Wed, 21 Jan 2009 17:19:29 UTC
Subject: [[TGPPL, deterring licence proliferation by withholding the stamp of OSD-conformance|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:491:200901:hnghfggnikagbelnanba]]

"It appears that in your concern about licence proliferation you have started to use your power of withholding certification of ~OSD-conformance as a general purpose tool to deter the use of [open source] licences that you don't like.  This is wrong." 

* Amir Herzberg: [[Folklore, Practice and Theory of Robust Combiners|http://amir.herzberg.googlepages.com/tolerant.pdf]] says that if {{{H}}} and {{{G}}} are hash functions, {{{H(G(x))}}} isn't a good way to combine them (no kidding).  And that {{{H(x)||G(x)}}} is (no kidding).
* Marc Fischlin, Anja Lehmann, and Krzysztof Pietrzak: [[Robust Multi-Property Combiners for Hash Functions Revisited|http://homepages.cwi.nl/~pietrzak/publications/FLP08.pdf]] says that Fischlin and Lehmann can build a super-duper-robust combiner which produces outputs five times as big as the outputs of the underlying hash functions and that they can build a super-robust combiner with outputs only two times as big.  Sheesh!  I'm looking for an output no bigger than the underlying hash functions, and on top of that I'm probably going to be truncating!  And yes, I know that Krzysztof Pietrzak says [[Non-Trivial Black-Box Combiners for Collision-Resistant Hash-Functions don't Exist|http://eprint.iacr.org/2006/348.pdf]] for outputs no bigger than the outputs of the underlying hash functions, but that doesn't matter because I don't need something that works for black boxes, I need only something that works for (let's say) ~SHA-256 and ~CubeHash.

Mozilla Lab's [[Test Pilot|https://testpilot.mozillalabs.com]] is a neat project. The goal is to learn how users use the Web. Mozilla asks users to opt-in and then collects statistics about how they use their web browser (but not personal data about what sites they visit or what queries they make).

The Test Pilot folks really want the results of their work to be integrated into humanity's store of knowledge, so they are publishing all of their data and encouraging others to collaborate with them on visualizing and analyzing it. They are also soliciting ideas from others about what experiments to run in the future. What do we (and by "we" I mean all of humanity) need to learn about how people use the Web?

Here are my notes in answer to that question, which are motivated by a top-secret project called ''Project Merely A Good Hack'' which aims to make a strong, censorship-resistant web.

As I make these questions more specific (perhaps with your help), I'll open issue tickets in the Mozilla Labs issue tracker, which is how you officially request future Test Pilot experiments.

Things we want to quantify:
* the differences between the two use cases: typing ~URLs in by hand vs. following links; Who performs these behaviors? In what contexts? Does the recent prevalence of ~URL-shorteners effect these metrics?
* the effect of automation-support on the "typing ~URLs in by hand", for example auto-completion in the URL widget, "~AwesomeBar" behavior which performs auto-complete by searching in your history and bookmarks
* the degree to which people use DNS versus using a Search Engine when their goal is to get to the home page for a well-known name or brand (consider the hilarious event in which there was a story on ~CNet which became the first google hit for the query "Facebook Login" and then the comments section on that story filling up with people saying that they hate the new Facebook design and they can't figure out how to log in! :-))
* Do many users notice the difference between IP address and DNS name in the "host" section of a URL? Do they prefer one kind over the other kind?
* When following links, do users parse out the domain name part of the URL and notice it, think about it, or judge it? (This is the phishing question, and there is probably good quantitative research or even controlled experiments from security-usability researchers on this one.)
* Are users turned-off by, or even suspicious of, longer links or links with more opaque data in them? I have [[a collection of many people spontaneously complaining|http://tahoe-lafs.org/trac/tahoe-lafs/ticket/882]] about ~Tahoe-LAFS ~URLs (such as the one for the page you are looking at right now), so I know that a lot of people really dislike these long, ugly ~URLs, but is this feeling limited to only über-geek Web engineers or do other groups of users also have a reaction to the length and appearance of ~URLs?
* How often does ~JavaScript running in a page inspect the URL for that page? How often does it modify it? How often does it inspect or modify the URL ''fragment'' part of the URL? (It's not immediately obvious to me how Test Pilot or Firefox could answer that last question, except by deeply invasive and heuristic instrumentation of the ~JavaScript interpreter itself.)
* If someone wants to share with their friend and send their friend a link to a resource, do they: (a) visit the resource themselves and then copy the link out of the URL widget, (b) right-click on the link and choose "Copy Link Location", (c) select the link text (if it is displayed anywhere) and choose "Edit -> Copy", (d) write to their friend and type in the name or title of the resource (e.g. //a troll mother walked out one day by selma lagerlof//) or the name or description of the host (e.g. //the new story on story of the day//), (e) other?

//''Mom:'' I'm trying to figure out how to make a Strong Web—a World Wide Web where nobody can take down anyone else's web pages. In order to do this, we need to know how people use the Web, in terms of how they find things and how they share things with others.//

I still think that the fact that Python 3 is backwards incompatible with Python 2 may impede users so much that Python really loses steam and Ruby, ~JavaScript, and newer languages like Clojure take its place. We'll see.

But, I'm heartened to see that it ''is'' actually possible to maintain a codebase which runs on both Python 2 and Python 3:

http://mail.python.org/pipermail/python-list/2010-July/1249312.html

http://mail.python.org/pipermail/python-list/2010-July/1249421.html

Since a user recently requested a Python 3 port of one of the packages that I maintain, I will actually start the first tentative steps toward supporting Python 3 soon.

(Note that the package in question, http://tahoe-lafs.org/trac/pycryptopp is actually a native extension package, so the compatibility hacks listed above for pure-Python code won't help. I'll have to read [[Porting Extension Modules to 3.0|http://docs.python.org/howto/cporting.html]] and probably add a bunch of horrible #ifdef's.)

P.S. I have to mention that I'm a lot more interested in ~JavaScript for future work than I am in Python, partially because [[Python 3 added an encapsulation hole|http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-python-30.html]] but [[JavaScript is trying to close a comparable hole|http://code.google.com/p/google-caja/wiki/GlobalScopeViaThis]] (by turning it off in ~ES5 strict mode).

Here are some tickets on open source projects that I'm interested in tracking.  See also [[issue tickets closed]].
* [[nautilus (launchpad #372287)|https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/372287]]: If you drag and drop a file whose name is invalidly encoded and whose mangled name collides with an extant file then the window doesn't refresh. (sometimes!?)
* python
** [[python #6280|http://bugs.python.org/issue6280]]: calendar.timegm() belongs in time module, next to time.gmtime()
** [[python #5720|http://bugs.python.org/issue5720]]: ctime: I don't think that word means what you think it means.
* tahoe
** [[tiddly_on_tahoe #9|http://allmydata.org/trac/tiddly_on_tahoe/ticket/9]]: can't save from Konqueror-4.2.0
** [[tahoe #615|http://allmydata.org/trac/tahoe/ticket/615]]: Can ~JavaScript loaded from Tahoe access all your content which is loaded from Tahoe?
** [[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail
** [[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff
** [[tahoe #605|http://allmydata.org/trac/tahoe/ticket/605]]: delayed connection on Windows
** [[tahoe #591|http://allmydata.org/trac/tahoe/ticket/591]]: "make quicktest" could be quicker and less noisy
** [[tahoe #556|http://allmydata.org/trac/tahoe/ticket/556]]: prepend 'application-version' with the name of this particular application
** [[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue
** [[tahoe #217|http://allmydata.org/trac/tahoe/ticket/217]]: ~DSA-based mutable files -- small ~URLs, fast file creation
** [[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read
** [[tahoe #424|http://allmydata.org/trac/tahoe/ticket/424]]: stdeb: push to upstream
** [[tahoe #423|http://allmydata.org/trac/tahoe/ticket/423]]: stdeb: use stdeb on tahoe itself
** [[tahoe #422|http://allmydata.org/trac/tahoe/ticket/422]]: stdeb: run from buildslaves
** [[tahoe #530|http://allmydata.org/trac/tahoe/ticket/530]]: use setuptools's """--multi-version""" mode
** [[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue
** [[tahoe #558|http://allmydata.org/trac/tahoe/ticket/558]]: kpreid says that the -SUMO tarballs don't exist
* (py)cryptopp
** [[pycryptopp #23|http://allmydata.org/trac/pycryptopp/ticket/23]]: can't build Python module for Python 2.6 with ~MinGW
** [[pycryptopp #2|http://allmydata.org/trac/pycryptopp/ticket/2]]: deterministic generation of private key from small seed
** [[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff
** [[pycryptopp #12|http://allmydata.org/trac/pycryptopp/ticket/12]]: automatic wrappers for all of Crypto++
** [[pycryptopp #13|http://allmydata.org/trac/pycryptopp/ticket/13]]: DSA "semi-private"/intermediate keys
* [[ubuntu p7zip #322481|https://bugs.launchpad.net/ubuntu/+source/p7zip/+bug/322481]]: took an order of magnitude longer than expected to compress
* [[amarok #184834|https://bugs.kde.org/show_bug.cgi?id=184834]]: amarok crashes on "Quit" 
* konqueror
** [[konqueror #184157|http://bugs.kde.org/show_bug.cgi?id=184157]]: crashed (I might have just hit the "back" button)
** [[konqueror #321636|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321636]]: kioslave crashes when logging into my issue tracker
* pyopenssl
** [[pyopenssl#238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: please provide binaries -- thanks Chris Galvan and JP Calderone
** [[pyopenssl#373470|https://bugs.launchpad.net/pyopenssl/+bug/373470]]: setup.py uses "/sw" by default
* [[tiddly_on_tahoe #7|http://allmydata.org/trac/tiddly_on_tahoe/ticket/7]]: wrong error message when server is unreachable
* [[pywin32 #1799934|https://sourceforge.net/tracker2/?func=detail&aid=1799934&group_id=78018&atid=551954]]: easy_install pywin32 -- @@almost fixed@@
* setuptools
** [[ubuntu #389865|https://bugs.launchpad.net/ubuntu/+source/python-setuptools/+bug/390965]]: Ubuntu package of setuptools has no version number in the filename of the {{{.egg-info}}} file
** [[setuptools #20|http://bugs.python.org/setuptools/issue20]]: package required at build time seems to be not fully present at install time?
** [[setuptools #57|http://bugs.python.org/setuptools/issue57]]: {{{develop}}} doesn't create {{{.pth}}} files and {{{site.py}}} if {{{--multi-version}}}
** [[setuptools #53|http://bugs.python.org/setuptools/issue53]]: respect the PYTHONPATH
** [[setuptools #54|http://bugs.python.org/setuptools/issue54]]: be more like distutils with regard to """--prefix="""
** [[setuptools #17|http://bugs.python.org/setuptools/issue17]]: easy_install will install a package that is already there; This issue should probably be renamed in light of the fact that it seems to cause a worse failure nowadays with the proposed Debian packages for {{{foolscap}}} and {{{tahoe-lafs}}}.
* nevow
** [[nevow #2713|http://divmod.org/trac/ticket/2713]]: setup.py installs tests, but not documentation
** [[nevow #2629|http://divmod.org/trac/ticket/2629]]: Nevow doesn't declare its dependency on Twisted in a machine-parseable way
** [[nevow #2699|http://divmod.org/trac/ticket/2699]]: build nevow without importing nevow
** [[nevow #2798|http://divmod.org/trac/ticket/2798]]: setup.py install """--home""" is broken :-(
* buildbot
** [[buildbot #266|http://buildbot.net/trac/ticket/266]]: I wish to tell my buildmaster: "restart yourself the next time you quiesce"
** [[buildbot #212|http://buildbot.net/trac/ticket/212]]: buildbot doesn't respond to darcs tags
** [[buildbot #395|http://buildbot.net/trac/ticket/395]]: when i change the vcs executable, buildslave stops being able to invoke it until I restart buildslave
** [[buildbot #396|http://buildbot.net/trac/ticket/396]]: Older builds
** [[buildbot #252|http://buildbot.net/trac/ticket/252]]: side-effecty operations (Force Builder) should be ~POSTs
** [[buildbot #407|http://buildbot.net/trac/ticket/407]]: {{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better @@patch submitted@@
* twisted
** [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory
** [[twisted #3649|http://twistedmatrix.com/trac/ticket/3649]]: more specific warning about plugin cache
** [[twisted #3238|http://twistedmatrix.com/trac/ticket/3238]]: patch: declare that twisted requires pywin32 if it is to offer process management or iocp reactor on Windows
** [[twisted #3878|http://twistedmatrix.com/trac/ticket/3878]]: build executables for trial, twistd, manhole, etc. on Windows (if setuptools is installed)
** [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory @@patch submitted@@
** [[twisted #3586|http://twistedmatrix.com/trac/ticket/3586]]: I want to install twisted without a c compiler
** [[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]]: Select default reactor based on platform and available libraries
** [[twisted #1956|http://twistedmatrix.com/trac/ticket/1956]]: Make a less sucky producer/consumer API
** [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]]: closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.
* darcs
** [[tailor #693|http://bugs.darcs.net/issue693]]: rmfile patch created without the actual content removal
** [[darcs #1477|http://bugs.darcs.net/issue1477]]: please speed up "darcs show contents"
** [[darcsver #2|http://allmydata.org/trac/darcsver/ticket/2]]: use "darcs query" to get count of patches faster
** [[darcs #1153|http://bugs.darcs.net/issue1153]]: darcs waits to hear back from servers unnecessarily
** [[darcs #1303|http://bugs.darcs.net/issue1303]]: proposal: make "darcs changes" interactive by default
** [[darcs #26|http://bugs.darcs.net/issue26]]: Darcs needs real MIME parsing, fails with Mail.app, Courier
** [[darcs #1255|http://bugs.darcs.net/issue1255]]: darcs put tries to convert to darcs-2-format?
* foolscap
** [[foolscap #107|http://foolscap.lothar.com/trac/ticket/107]]: exceptions.~KeyError: "unable to find reference for name
** [[foolscap #105|http://foolscap.lothar.com/trac/ticket/105]]: make it easy to distinguish server-side failures/exceptions from client-side
** [[foolscap #108|http://foolscap.lothar.com/trac/ticket/108]]: set base to "." if not running from source (so {{{flogtool}}} works on Windows)
** [[foolscap #109|http://foolscap.lothar.com/trac/ticket/109]]: make a "flogtool" executable that works on Windows
** [[foolscap #111|http://foolscap.lothar.com/trac/ticket/111]]: timestamps of incident files -- TZ indicator please
** [[foolscap #112|http://foolscap.lothar.com/trac/ticket/112]]: timestamps of incident files -- ~ISO-8601'ish
** [[foolscap #113|http://foolscap.lothar.com/trac/ticket/113]]: timestamps of incident files -- UTC
* [[ubuntu #314468|https://bugs.launchpad.net/hardy-backports/+bug/314468]]: Please backport setuptools-0.6c9 from Intrepid.

* [[twisted #4468|http://twistedmatrix.com/trac/ticket/4468]]: twisted.python.randbytes
* [[bzr #294709|https://bugs.launchpad.net/bzr/+bug/294709]]: Pushing to shared repo using FTP doesn't work: "Append/Restart not permitted, try again" @@Fixed -- this makes bzr work on top of ~TahoeLAFS!@@
* [[nevow #2830|http://divmod.org/trac/ticket/2830]]: setup.py incorrectly declares twisted.plugins to be a package @@fixed@@
* [[cryptopp #?+1|http://groups.google.com/group/cryptopp-users/browse_thread/thread/97570197f77cc80c]]: patch: initialize randpool with zeroes @@fixed@@
* [[pyflakes #2720|http://divmod.org/trac/ticket/2720]]: Release Pyflakes @@fixed@@
* [[konqueror #321656|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321656]]: iso-8859-1 and/or utf-8 character not decoded properly @@fixed@@
* [[pycryptopp #9|http://allmydata.org/trac/pycryptopp/ticket/9]]: link against existing (system) libcrypto++.so @@fixed@@
* [[twisted #3568|http://twistedmatrix.com/trac/ticket/3568]]: ERROR from conch test when pycrypto is not installed @@fixed@@
* [[pycryptopp #11|http://allmydata.org/trac/pycryptopp/ticket/11]]: submit patch for Brainpool ECC to Crypto++ @@fixed@@
* [[cryptopp #?+0|http://groups.google.com/group/cryptopp-users/browse_thread/thread/6d37437aa40fc135]]: patch: build libcryptopp.so @@invalid@@ rejected in [[this mailing list discussion|http://groups.google.com/group/cryptopp-users/browse_thread/thread/6d37437aa40fc135]]
* [[elisa #263697|https://bugs.launchpad.net/ubuntu/+source/elisa/+bug/263697]]: installing elisa breaks the unit tests of unrelated Python packages @@fixed@@
* [[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]]: python-foolscap: should advertise [secure_connections] feature to setuptools @@fixed!@@
* [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]]: ~SiteUrl as current document location @@fixed@@
* [[darcs #1217|http://bugs.darcs.net/issue1217]]: darcs put fails with 'darcs failed:  Malformed patch bundle: '{' is not 'Context:' ' -- @@superceded by darcs #1255@@
* [[pyopenssl #311600|https://bugs.launchpad.net/pyopenssl/+bug/311600]]: please update http://pypi.python.org/simple/pyOpenSSL -- @@fixed@@
* [[tahoe #555|http://allmydata.org/trac/tahoe/ticket/555]]: tahoe .deb cannot be installed on hardy: simplejson dependency is too new -- @@fixed@@
* [[tiddly_on_tahoe #2|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]: don't offer the option to save changes when you are viewing read-only -- @@fixed!@@
* [[tiddly_on_tahoe #3|http://allmydata.org/trac/tiddly_on_tahoe/ticket/3]]: offer a read-only cap to the user -- @@fixed!@@
* [[nevow #2527|http://divmod.org/trac/ticket/2527]]: easy_install compatibility -- @@hooray!  This issue is finally fixed!@@
* [[pyflakes #2709|http://divmod.org/trac/ticket/2709]]: Pyflakes svn doesn't install properly due to missing packages -- @@hooray!  Fixed!@@
* [[buildbot #236|http://buildbot.net/trac/ticket/236]]: show elapsed time for steps -- @@fixed@@

My brother Josh sent me a link to this New York Times article: //Benedict Carey: [[At the Bridge Table, Clues to a Lucid Old Age|../URI%3ADIR2-RO%3Aww5ty4rchjgwgh5qympsgrqc54%3Aryjsirro4fejrurrcladf5swhtfc5y2xr7guchqd7al4uuqlq2wq/22brain.html]]//.  I replied something like this:

This article says “So far, scientists here have found little evidence that diet [...] affects the risk of dementia in people over 90.”.

Then it goes on to mention that having the //~APOE2// and //CETP// genes confer mental acuity into old age, or confer longevity.  And what do //~APOE2// and //CETP// do?  Well, of course it's not easy to know //everything// that they do, but the obvious things that they do are all about improving cholesterol levels and reducing insulin (and therefore improving the breakdown of amyloid, among other things).  Isn't it interesting that a low-carb diet has these same effects? ([[Taubes|http://www.amazon.com/dp/1400033462]], p. 204, "Dementia, Cancer, and Aging")

Scientists have in general found little evidence of diet helping with dementia and aging, but they have never looked at the effects of //low-carb// diet on dementia or aging.  All experiments and observations to date have been comparing the control group of the current mainstream diet with the test group of a low-fat diet.  Low-fat diets should be expected (from what we now know about their effects on cholesterol levels) to either have little effect or to worsen dementia and aging.  (I believe I've read that low-fat dieting does indeed positively correlate with worse dementia in women, but I don't have a reference to hand -- it's probably from the large scale studies in the USA in the 2nd half of the 20th century, as described by Taubes.)  Low-fat diets have been the natural focus of such research because of the belief, now disproven but still widely held, that low-fat diets improve cholesterol levels and are generally "healthy".

There's a more insidious reason for this myopia, however.  Whoever owns the patent on a drug that prevents dementia will make billions and tens of billions of dollars over the next couple of decades.  Whoever discovers that you can get the same effect by quitting sugar, high-fructose corn syrup, grains and starches will get notoriety among a few thousand of their peers, and hopefully tenure.  You can hear echoes of this vastly disproportionate incentive whenever scientists in news articles talk about the ways that their ideas might be able to help people in the future:
<<<
Dr. Nir Barzilai of the Albert Einstein College of Medicine has found that lucid Ashkenazi Jewish centenarians are three times more likely to carry a gene called //CETP//, which appears to increase the size and amount of so-called good cholesterol particles, than peers who succumbed to dementia.
“We don’t know how this could be protective, but it’s very strongly correlated with good cognitive function at this late age,” Dr. Barzilai said. “And at least it gives us a target for future treatments.”
<<<
I think he's talking about a target for drugs.

And yes, low-carb dieting increases your so-called good cholesterol levels.  I think that it does so to approximately the same degree that being born with the //CETP// gene does, but I'm not sure.

I think Josh might be ideally placed to test the hypothesis that low-carb dieting has similar prolongevity and anti-dementia effects as the genes //~APOE2// and //CETP// (and, not mentioned in this article, //apoC-III//).  Maybe he can get tenure out of the deal!

moved from [[things to read]] to [[things I have read]] and also into [[studies on low-carb diet]]:
* R. Neil A. Black, Michelle Spence, Ross O. ~McMahon, Geraldine J. Cuskelly, Cieran N. Ennis, David R. ~McCance,Ian S. Young, Patrick M. Bell, and Steven J. Hunter: [[Effect of Eucaloric High- and Low-Sucrose Diets With Identical Macronutrient Profile on Insulin Resistance and Vascular Risk|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Black_et_al-2006-Effect_of_Eucaloric_High-_and_Low-Sucrose_Diets_With_Identical_Macronutrient_Profile_on_Insulin_Resistance_and_Vascular_Risk.pdf]] (as recommended by Brendan O'Connor)
This is a randomized, controlled, crossover trial, and it provides a result that is counter to my beliefs: it shows that increasing the portion of table sugar in your diet does ''not'' make you more insulin-resistant!
Disconfirmation, oh wonderful thing! As I [[recently blogged|2010-02-11]], ''disconfirmation'' is the stuff that you really ought to pay attention to, and especially when it comes from well-controlled experiments (as opposed to ex post facto analysis of complex, noisy data). Unfortunately, this study has enough limitations and weirdnesses that it isn't by itself enough to make me give up on the hypothesis that a diet of refined carbohydrates causes diabetes. (For background on why I would think that, please see chapter 6 of Taubes //Good Calories, Bad Calories// [[(my review)|Good Calories, Bad Calories]].)
The main weirdness in this study is that they deliberately replaced the table sugar ("sucrose"), with other highly refined carbohydrates instead of with complex carbohydrates. For example, the low-sucrose diet had white bread, the high-sucrose diet had whole wheat bread. The low-sucrose diet had strawberry jam (presumably the kind without added sucrose) the high-sucrose diet had none. The low-sucrose diet had cornflakes for breakfast, the high-sucrose diet had a combination of cornflakes and bran-based cereal. The effect was that the total amount of //refined, easily digestable carbs// consumed was the same in both diets.
This is a good way to target the experiment narrowly at whether table sugar—sucrose—specifically causes insulin-resistance compared with other easily digestable carbs such as those from potatoes, corn or strawberries. However, my primary hypothesis is that easily digestable carbs ''as such'' cause insulin-resistance, and the design of this experiment can't shed light on that question.
Even so, I'm surprised at the way that sucrose caused very litlte harm in this study. Sucrose is about half fructose and half glucose, so a diet high in sucrose is also a diet high in fructose. (//High fructose corn syrup// is roughly the same as sucrose in this regard.) The hypothesis of //fructose-induced lipogenesis// (Taubes chapter 12) says that a diet high in fructose should increase triglycerides. According to Taubes, Peter Mayes of King's College, London says that the liver and pancreas adapt over time to a diet high in fructose and start generating more and more insulin in response to fructose ingestion, but although there was indication of those effects in this experiment, the results were not statistically significant.
And that's where we get to the other limitations of this study. The subjects tried each diet for only six weeks. If there is an effect, as Peter Mayes claims, which appears with long-term exposure then this study wouldn't have shown it. Other limitations: there were only thirteen subjects (which is so few that it makes it easy for effects to be statistically insignificant) and they were all young, healthy, males of Western European descent, and finally that the food substitutions that the researchers designed to get the amount of sucrose they wanted accidentally made the amount of saturated fat in the high-sucrose diet 36% greater than the amount of saturated fat in the low-sucrose diet.
This study was funded by the Netherlands sugar manufacturers' assocation and the U.K. sugar manufacturers' association. I don't like to make too much of that sort of thing because scientists contribute to the world by doing verifiable, re-usable work and sharing it, not by persuading others to have faith in their character and their motives. If someone does verifiable, re-usable work and shares it then we can benefit from their work regardless of who funded it.
However, [[this web page from the U.K. sugar manufacturers' association|http://www.sugar-bureau.co.uk/diabetes.html]], which cites this study in order to encourage diabetics to eat more sugar, makes me angry.
The authors, or at least the sponsors, of this study would have us conclude that eating table sugar is no worse for your risk of diabetes than eating potatoes, corn, or strawberries. I'm not quite ready to agree with them—we need larger, longer, and more studies!—but on the other hand I wouldn't be surprised if it turned out that eating potatoes, corn, and strawberries were just as bad for your risk of diabetes as eating table sugar. :-)
----
P.S. Oh, hey you can read parts of //Good Calories, Bad Calories// on "google books", including [[the part of Chapter 12 that describes Peter Mayes's hypothesis of fructose-induced lipogenesis|http://books.google.com/books?id=Xdm40JUD9HwC&pg=PA200&lpg=PA200&dq=peter+mayes+king%27s+college&source=bl&ots=aLsfN7PAA8&sig=MiSIAVOO4IKsrvaoLienZe-KHYE&hl=en&ei=4Ml8S8SHPImcswP6lqW8Cw&sa=X&oi=book_result&ct=result&resnum=5&ved=0CBUQ6AEwBA#v=onepage&q=&f=false]]. If you were impressed by weighty authorities that the U.K. sugar manufacturer's association appealed to in its page on diabetes which I linked to above then you should go ahead and read to the end of Chapter 12. ;-)
----
P.P.S. Here are notes from my wife, Amber, which I consulted when writing this blog entry:
<<<
Well, I was surprised that sucrose didn't fare worse.  It seemed from the sample menus provided that they were comparing sucrose with white bread and potatoes.  White bread and potatoes do have higher glycemic indeces than table sugar.  I also found it odd that the 10% sugar group were getting white bread and the 25% sugar group were getting whole wheat, and the 10% group were getting potato chips, while the 25% group were getting apples.  They did report that overall fiber amounts were the same, though, even though they were adding fiber to the cereal of the 25% group, but not the 10% group.  I'd also like to know the quality of the carbohydrates that weren't sucrose or starch. Ten percent of the calories in both cases were non-sucrose, non-starch carbohydrates.  For example, what were the amounts of high-fructose corn syrup, which should act in the same detrimental way as sucrose.
I'm a little suspicious of these details, because of the affiliation of the writers with two sugar boards, but I understand that sometimes only interested parties will fund such things, and indeed, at least one of my favourite low carb studies was funded in part by Atkins Inc.

Anyway, still interesting.  I wonder if other comparable studies exist.
<<<

added to [[things I have read]]:
* C. Murray Skeaff, and Jody Miller: [[Dietary Fat and Coronary Heart Disease: Summary of Evidence from Prospective Cohort and Randomised Controlled Trials|http://content.karger.com/produktedb/produkte.asp?typ=fulltext&file=000229002]] (as recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]])
This meta-analysis offers detailed results and it seems like it followed a pretty rigorous process.  I wouldn't bet my life on it because it includes prospective cohort studies (which do not always tease apart correlation and causation) in addition to randomized controlled trials (which generally do) and because some of its findings are self-contradictory or inexplicable. Nonetheless these conclusions are almost certainly true:
* intake of //total fat// has ''no effect'' on heart disease
* intake of //saturated fat// has ''no effect'' on heart disease
* intake of //monounsaturated fat// (the so-called “good fats” that they are telling you to eat more of) has ''no effect'' on heart disease
* intake of //trans fat// ''greatly increases'' your risk of heart disease
* intake of //polyunsaturated fat//: ''unclear'' -- seems to increase your chance of //dying// from heart disease although it doesn't increase your chance of getting heart disease in the first place, and it seems to decrease your chance of dying from heart disease in a subset of the studies in which it was a substitute for //saturated fat//. My interpretation of this weird result is that it makes you unhealthy in other ways which makes it harder for you to survive and recover from a heart attack, or else that the data about this is just insufficient and unreliable.
* intake of //n-3 long-chain polyunsaturated fatty acids// (i.e. fish or fish oil supplements): ''unclear'' -- some studies indicate that eating fish a couple of times a week will reduce your risk of dying of heart disease by 36%.  Others show that taking fish oil supplements for a few months has no effect on your chances of getting heart disease. My guess is that eating fish makes you healthier in other ways so that you're better able to survive a heart attack.  Maybe people who eat a whole meal of fish (as opposed to taking a fish oil pill) therefore eat fewer carbs (which should greatly reduce their chances of getting heart disease and their chances of dying from it). Or maybe the data is just insufficient and unreliable.  
For a good demonstration of the bad advice you are getting from your expert health officials, see //The American Heart Association//'s web page called [[Know Your Fats|http://www.americanheart.org/presenter.jhtml?identifier=532]] which advises you to do almost everything exactly wrong.  The only part they get right is that you should avoid trans fats.  I find this ironic because one of the reasons that trans fats are such a big part of the modern American diet is because of the advice given by //The American Heart Association// from the late 1970's to 2000's saying that people should switch away from saturated fat foods like butter to alternatives like margarine.  (Check the labels of your food products for “partially hydrogenated vegetable oil”.)

I submitted the following proposal in response to [[the CodeCon RFP|http://www.codecon.org/2009/cfp.txt]]:

Project name: Tahoe, the ~Least-Authority Filesystem
Track: code
url of home page: http://allmydata.org
tagline: a secure, decentralized, fault-tolerant storage network //(whoops I forgot to say ''"cloud"'' again)//
presenter: Zooko ~Wilcox-O'Hearn, http://zooko.com
alternate/backup/co- presenter: Brian Warner, http://lothar.com

project history: In 2006 I got to start fresh on inventing a secure, decentralized storage network, after the failure of Mojo Nation (for which I was partially responsible), the failure of Mnet (for which I was primarily responsible), the observed failures of Freenet, and the ongoing failure of a proprietary commercial backup system written by allmydata.com (for which I was partially responsible), not to mention a few other failures that I also tried to learn from.  I tried to learn from the success of ~BitTorrent by starting fresh and limiting the scope.  Also, I was blessed with a supportive company and the kick-ass engineering skills of Brian Warner, and I finally got a secure, decentralized storage network that didn't fail!  allmydata.com deployed Tahoe a year ago and copied all of their customer data over to Tahoe from the old proprietary system.  Open source hackers are building on it.  It works!

novelty: Tahoe is comparable to Freenet, ~OceanStore, and Mojo Nation.  It avoids some of the trickier problems in this space by limiting the scope: Tahoe assumes that the set of storage servers is not too large or dynamic, and that there are enough servers that are at least moderately reliable.  This means it doesn't even *try* to solve the Very Hard Problem of sharing storage with millions of anonymous strangers, but on the other hand it does a fine job of sharing storage among a couple hundred moderately reliable servers, such as a "friendnet" (home computers operated by your friends and family) or the allmydata.com commercial grid.  On top of this pool of moderately-reliable servers, Tahoe adds encryption for confidentiality and integrity, erasure-coding for high reliability, and capabilities for file-sharing.  The "Principle of Least Authority" design means that the system relies on each component as little as possible -- security properties such as confidentiality, integrity, and access control are all guaranteed by the client on its own behalf using cryptography instead of relying on the servers to cooperate in providing those properties.  To get the reliability properties that it wants the client *does* require the help of the servers, but by the power of erasure coding, only a subset of the servers need to perform only moderately well for the reliability properties to hold.  Tahoe is the only open source project that I know of which offers these sorts of properties in a practical system that many people use every day.

demo: I haven't thought this through all the way, but at several hacker parties in the past we've had partiers install Tahoe on their laptops and form a "temporary autonomous zone" storage system on which to share music and movies.  When the laptops close up and go home, the temporary autonomous zone is destroyed and all of the files become unrecoverable (unless a quorum of the partiers were to later reconvene and reconnect their laptops).  Maybe we could figure out a way to have some such live audience participation in the demo.  It has worked at parties with dozens of attendees, but I'm not sure if it would fit into a ~CodeCon demo.  Failing that, I can always demo the user-facing applications that run from Tahoe, such as streaming movies and "gridapps", which are ~JavaScript applets that are stored in Tahoe and executed in your web browser.  Maybe I could cook up some sort of demo involving suddenly and violently destroying one of the storage servers and then demonstrating that all the content is still available because of the survival of the other ones.  Hey, that sounds like fun!  As you can see, I don't have a precise plan yet.  Nor money to spend on a sacrificial removable hard drive or two.  :-)

slides: I have none prepared specifically for ~CodeCon yet.  Here is the peer-reviewed short paper that I presented at the Storage Security and Survivability Workshop -- http://allmydata.org/~zooko/lafs.pdf .  Here are the slides that I used at that presentation: http://zooko.com/lafs/presentation/index.html .  At that presentation I did actually load each of the slides on demand from a live Tahoe grid so it was a demo as well as a presentation.

future plans: 1.  Support more and more people building on top of Tahoe, such as allmydata.com's backup business, and several open source projects that are currently building on top of Tahoe.  I'm especially interested in "gridapps", which might evolve into a distributed computation platform that can be built with the world's vast supply of web app development expertise.  "Gridapps" look exactly like web apps, but all of their storage is in the decentralized, secure tahoe grid, and they have access to the convenient capability-based file-sharing API (over HTTP), so they could do some interesting things.

Future plan #2: fix the glaring deficiencies that we already know about, plus all the new ones that will be revealed in the process of Future plan #1.

Future plan #3: document the file formats and protocols in sufficient precision that others could write a compatible implementation from the spec.

Future plan #4: design better-performing and safer cryptographic mechanisms and better-performing and more versatile filesystem semantics.

In 2008 the Comprehensive National Cybersecurity Initiative (CNCI) was [[created by a classified presidential order|http://www.nextgov.com/the_basics/tb_20090601_8569.php]], with a budget estimated to be a whopping $40 billion. This initiative is supposed to protect American government, military, and civilian networks from being penetrated by enemies.
[["Titan Rain"|http://en.wikipedia.org/wiki/Titan_Rain]], [["GhostNet"|http://en.wikipedia.org/wiki/GhostNet]] and most recently [["Operation Aurora"|http://en.wikipedia.org/wiki/Operation_Aurora]] in which someone -- [[probably Chinese cyberwarriors in the service of the Chinese government|http://arstechnica.com/security/news/2010/01/researchers-identify-command-servers-behind-google-attack.ars]], exploited United States national laboratories, human rights organizations, and dozens of high-tech companies like Adobe and Google are examples of the kind of threat that CNCI is supposed to defend against.
(It wasn't clear originally who got to be in charge, so there was a bureaucratic turf war between the Department of Homeland Security (DHS) and National Security Agency (NSA) [[which NSA won|http://www.govexec.com/pdfs/030909j1.pdf]].)
One of the things that the CNCI did was organize [["National Cyber Leap-Year"|http://www.nitrd.gov/leapyear/National_Cyber_Leap_Year_Background.pdf]].  This project was headed by the NSA and included every national security and computer science agency that I can think of except for DHS: DARPA, DOE, NASA, NIH, NIST, NSA, NSF, OSD, DOD. I think I recall hearing at the Association for Computing Machinery security conference in 2008 that the budget for the National Cyber ~Leap-Year project was $100 million.  They recruited all the best security researchers that they could find and asked them to dream up some great leap forward in national cyber defense. They were asked to envision something which would break out of the arms race between attackers and defenders and instead "change the terrain" so that defenders have a permanent advantage.
The report that the security experts produced after a year of thinking up novel cyber defense strategies predictably includes a myriad of ambitious, unproven ideas.  It also includes one specific proposal: follow the lead of [[Tahoe-LAFS|http://tahoe-lafs.org/trac/tahoe-lafs]]!
<<<
Systems like Tahoe are making these methods immediately usable for securely and availably storing files at rest; we propose that the methods be further reviewed, written up, and strongly evangelized as best practices in both government and industry.
<<<
—from [[Report on National Cyber Leap Year Summit 2009|http://www.qinetiq-na.com/Collateral/Documents/English-US/InTheNews_docs/National_Cyber_Leap_Year_Summit_2009_Co-Chairs_Report.pdf]]
Yeah!! It feels great to get some official recognition of what this little loosely-organized Free Software project is doing.
We've been doing this on a shoestring budget. (Estimated budget for running http://tahoe-lafs.org and sending thank-you gifts to volunteers was $3000 for 2009. We could use financial help to cover these costs for 2010! You can donate on [[the Tahoe-LAFS project home page|http://tahoe-lafs.org]].)
We've been "working in public", publishing all of our ideas, putting all of our source code under Free Software, Open Source licences, and documenting our successes and failures so that others can learn from them.  Our team of volunteers includes hackers in the U.S.A., Germany, the U.K., Pakistan, France, Austria, and probably many other places around the globe.
And what we've accomplished so far really ''is'' a step forward in safe, reliable Internet storage.  If you care about such things, please help us out. We need coders, testers, documentation writers, usability experts, and users.

One day I decided I needed to un-follow some people on twitter, even though I thought they were sometimes interesting, just because I had too little time to do that much twitter-reading.  I decided to simply write down their twitter names so that I could re-follow them, or at least just check their current twitter pages, in the future.  To see who I am still following, see [[my twitter page|http://twitter.com/zooko]].  Probably a lot of the people that I am still following are still there because they haven't tweeted anything, or at least not much, since I started this winnowing process.  ;-)

Then I stopped updating this list and started putting people that I unfollowed into this list hosted on twitter instead: http://twitter.com/#/list/zooko/interesting

* tahoe-lafs/file systems: [[joshuagross|http://twitter.com/joshuagross]], [[destari|http://twitter.com/destari]]
* datacenter/storage/cloud: [[datacenter|http://twitter.com/datacenter]], [[cloudbook|http://twitter.com/cloudbook]], [[scott_lowe|http://twitter.com/scott_lowe]], [[stu|http://twitter.com/stu]], [[ruv|http://twitter.com/ruv]], [[samj|http://twitter.com/samj]], [[davegraham|http://twitter.com/davegraham]], [[krishnan|http://twitter.com/krishnan]], [[lewmoorman|http://twitter.com/lewmoorman]], [[Rich_Bruklis|http://twitter.com/Rich_Bruklis]], [[Cirrhus9|http://twitter.com/Cirrhus9]], [[cloudbzz|http://twitter.com/cloudbzz]], [[Beaker|http://twitter.com/Beaker]], [[randybias|http://twitter.com/randybias]], [[jamesurquhart|http://twitter.com/jamesurquhart]], [[storageio|http://twitter.com/storageio]], [[stevie_chambers|http://twitter.com/stevie_chambers]]
* open source: [[vaughmuirhead|http://twitter.com/vaughmuirhead]], [[anantn|http://twitter.com/anantn]]
* security: [[jeremiahg|http://twitter.com/jeremiahg]], [[iamnowonmai|http://twitter.com/iamnowonmai]], [[stiennon|http://twitter.com/stiennon]], [[securityincite|http://twitter.com/securityincite]], [[jack_daniel|http://twitter.com/jack_daniel]], [[gattaca|http://twitter.com/gattaca]], [[domdingelom|http://twitter.com/domdingelom]], [[rmogull|http://twitter.com/rmogull]], [[paperghost|http://twitter.com/paperghost]], [[quine|http://twitter.com/quine]], [[weldpond|http://twitter.com/weldpond]], [[myrcurial|http://twitter.com/myrcurial]], [[2342|http://twitter.com/2342]], [[singe|http://twitter.com/singe]]
* programming languages: [[kumar303 |http://twitter.com/kumar303]] (python), [[jboner |http://twitter.com/jboner]] (scala), [[patrickdlogan|http://twitter.com/patrickdlogan]], [[mamund|https://twitter.com/mamund]] (erlang)
* startup: [[anandiyer|http://twitter.com/anandiyer]], [[brendonjwilson|http://twitter.com/brendonjwilson]]
* social: [[davewiner|http://twitter.com/davewiner]], [[Scobleizer|http://twitter.com/Scobleizer]], [[arrington|http://twitter.com/arrington]],
* unsure: [[sourcehosting|http://twitter.com/sourcehosting]], [[theqbn|http://twitter.com/theqbn]], [[awakeningaimee|http://twitter.com/awakeningaimee]],
* Boulder: [[kimbal|http://twitter.com/kimbal]], [[GeorgeGSmithJr|http://twitter.com/GeorgeGSmithJr]], [[mg|http://twitter.com/mg]], [[BoulderTriSkier|http://twitter.com/BoulderTriSkier]], [[tcabeen|http://twitter.com/tcabeen]], [[pugofwar|http://twitter.com/pugofwar]], [[jennyjenjen|http://twitter.com/jennyjenjen]], [[heyrich|http://twitter.com/heyrich]], [[Java1Guy|http://twitter.com/Java1Guy]], [[bpm140|http://twitter.com/bpm140]], [[ozskier|http://twitter.com/ozskier]], [[OneRiot|http://twitter.com/OneRiot]], [[andrewhyde|http://twitter.com/andrewhyde]], [[BrettGreene|http://twitter.com/BrettGreene]], [[Penguin|http://twitter.com/Penguin]], [[kirash4|http://twitter.com/kirash4]], [[lmckeogh|http://twitter.com/lmckeogh]], [[hwy12|http://twitter.com/hwy12]], [[DHendersonCO|http://twitter.com/DHendersonCO]], [[COStartupTrack|http://twitter.com/COStartupTrack]], [[jvaleski|http://twitter.com/jvaleski]], [[jrpowers|http://twitter.com/jrpowers]], [[tcabeen|https://twitter.com/tcabeen]]
* misc tech: [[stockrt|http://twitter.com/stockrt]]

I edited the following to fix a couple of major edit-o's that confused the meaning of my post. To see the original including edit-o's, look at [[the archive of cryptography@randombit.net|http://lists.randombit.net/pipermail/cryptography/2010-October/000203.html]], and also while you are there read the ensuing thread which has lots of good commentary by others.

From: Zooko O'Whielacronx
To: hash-forum
Date: 2010-10-14 03:46 UTC
	
Folks:

If a hash has 32-bit pre-image-resistance then this means an attacker
might spend about 2³² resources to find a pre-image.

If a hash has 64-bit pre-image-resistance then this means an attacker
might spend about 2⁶⁴ resources to find a pre-image.

What if a hash has 512-bit pre-image-resistance? What would that mean?
That an attacker might spend about 2⁵¹² resources to find a pre-image
in it? That is a meaningless possibility to discuss since 2⁵¹²
resources will never exist in the life of this universe, so it can't
mean that, or if it does mean that then there is no use in talking
about "512-bit pre-image-resistance". Maybe it means something else?

By analogy, suppose you considered the construction of a bridge that
withstood 10³ tons of pressure. You could also consider a bridge that
could withstand 10⁶ tons of pressure. If the bridge were to be
deployed in a situation where more than 10³ tons but less than 10⁶
tons might rest on it, then this would be a very important distinction
to make.

But what would it mean to discuss a design for a bridge that could
withstand 10¹⁵⁰ tons of pressure? Such an amount of pressure could
never be applied to the bridge. Would there be any value in a
distinction between one bridge design that would withstand 10¹⁵⁰ tons
of pressure and another that would withstand 10³⁰⁰? Even though
neither of them could ever experience as much as 10¹⁵⁰ tons of
pressure, perhaps the latter bridge would still be safer against some
other threat—an error on the part of the builders or designers or a
stressful event that was not included in the model which we used to
evaluate our bridges in the first place.

Or perhaps not. Perhaps the bridge which is designed to withstand
10³⁰⁰ tons of pressure is actually ''more'' likely to fail than the
other one when hit by this unpredicted, unmodelled event. Who can
tell?

One reasonable position to take is that it was a mistake for NIST to
specify that some of the ~SHA-3 hashes had to have 512-bit preimage
resistance. (If it ''was'' a mistake the I really have no idea what to
do about it at this juncture!)

That position says that there ''is'' a need for a hash function which
takes much more CPU time than ~SHA-3-256 does in order to provide much
less likelihood that an attacker will be able to find a pre-image in
it than in ~SHA-3-256, but that this "much less likelihood" is not in
any meaningful sense correlated with the idea of having "512-bit
pre-image resistance".

Another reasonable position to take is that a hash function which is
known to have at most 384-bit pre-image resistance is ''more likely to
fail'' than one which is known to have at most 512-bit pre-image
resistance. This is where my limited understanding of hash function
cryptanalysis comes to an end. Is that plausible? If I give you two
hash functions like that, are you confident that you could learn how
to find pre-images in the former before you find pre-images in the
latter? How sure are you? Is it possible that it would be the other
way around—that you would discover a method of finding pre-images in
the latter before discovering a method of finding pre-images in the
former?

If someone who has real hash function cryptanalysis expertise and who
takes the latter position could explain what they mean by "more likely
to fail", then I would be fascinated to hear it.

In any case, I'm pretty sure that as a ''user'' of hash functions what I
care about is "more likely to fail" (and efficiency), not about "bits
of security" for any bit-level greater than about 128 (including
consideration of quantum attacks, multi-target attacks, etc.)

Thank you for taking the time to read this.

Regards,

Zooko ~Wilcox-O'Hearn

[a letter I wrote to a discussion group]

During the meeting yesterday morning someone's voice -- I believe it was the voice of Bill Frantz -- said that he preferred to avoid public key cryptography when possible because it could be broken by quantum computers, if they existed.  This is true of all the big ones -- RSA, DSA, ECDSA, etc.  Fortunately, there are a few public key crypto systems which (as far as we know) aren't vulnerable to quantum computers: Merkle Hash-based signatures, ~McEliece code-based crypto, lattice-based crypto, and something that I completely don't understand called multivariate-quadratic-equations.  Here is Daniel J. Bernstein's overview at a conference of the topic of "post-quantum cryptography" in 2008:

http://math.uc.edu/~aac/pqcrypto2008/presentations/bernstein.pdf

The lattice-based ones are especially interesting to me because they are already implemented and used today, and because they are faster than RSA or ECDSA and have smaller key sizes than RSA.  See the part about "NTRU" in here:

http://postquantum.cr.yp.to/pqcrypto2006record.pdf

Unfortunately they have much larger key sizes and costs of key generation than ECDSA, and there are also many patents currently in force on lattice-based techniques, so I don't expect to use them any time soon.

However, when thinking about the longer-term, it seems like there is a good chance that public key crypto will continue to work even if real quantum computers are built.  On the other hand, don't forget the time Blueshell had a humor fit at Pham's faith in public-key cryptography.

Regards,

Zooko

//Looking for {{{pyutil.jsonutil}}}? Download [[pyutil|http://pypi.python.org/pypi/pyutil]].//

Yesterday a coworker at [[SimpleGeo|http://simplegeo.com]], Derek Smith, mentioned that he was converting real numbers (latitude/longitude coordinates) to and from string representation (e.g. "102.01198501").  This sort of representation has been standardized as [[GeoJSON|http://geojson.org]].  We noticed that the Python {{{json}}} module was converting these into floats, which introduces some error and some weirdness.
For example, observe that {{{0.1²≠0.01}}} when using floats:
{{{
$ python
>>> 0.1**2 == 0.01
False
>>> 0.1**2
0.010000000000000002
}}}
{{{0.1²}}} comes out to {{{0.010000000000000002}}} (that's fifteen zeroes in the middle) even though the exact answer is {{{0.01}}} and the IEEE 754 number that is closest to {{{0.01}}} is {{{0.01000000000000000020816681712}}} (that's sixteen zeroes in the middle) which is off by only one-tenth as much! (I learned this from [[this cool IEEE 754 calculator|http://babbage.cs.qc.edu/IEEE-754/Decimal.html]]).
Python has a nice standard datatype for decimal numbers like this: [[the decimal module|http://docs.python.org/library/decimal.html#module-decimal]].  So I decided I would just configure the Python {{{json}}} module to convert to and from JSON numbers using the Python decimal module instead of using the Python float implementation. It wasn't as easy as I had hoped, but here it is: [[pyutil.jsonutil|http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/jsonutil.py]] (download [[pyutil|http://pypi.python.org/pypi/pyutil]]).
To use {{{jsonutil}}}, just replace {{{import json}}} in your code with {{{from pyutil import jsonutil as json}}}.  Even though my hack is [[less than ten lines of code|http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/jsonutil.py]] I [[copied the json unit tests|http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/test/json_tests]] from the Python 2.6 source code and added [[my own unit tests|http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/test/test_jsonutil.py]] and sure enough I found a couple of bugs in my hack.  (It's amazing how a bug or two can hide in even a small amount of code.  And how a few simple tests can find them.)  I had to change the json unit tests just a little -- the major change was removing the assertion that if you try to encode {{{23456789012E666}}} it will fail and replacing it with an assertion that if you try to encode that number it will succeed.  That was satisfying.  :-)
There is an open ticket for the {{{json}}} module [[asking for decimal support|http://code.google.com/p/simplejson/issues/detail?id=34]]. From reading that ticket it sounds like some of the folks involved (e.g. Bob Ippolito, the author of simplejson and a good hacker) think that JSON has a float type. That's incorrect -- JSON has a number type which specifies arbitrary precision decimal, see: [[the JSON spec|http://json.org]]. For example this JSON document:
{{{
{"price_of_eggs": 0.1}
}}}
is distinct from this JSON document:
{{{
{"price_of_eggs": 0.10000000000000001}
}}}
Even though both of them would result in the same data if parsed as Javascript.
In any case JSON was designed to be a language-independent format and it works well for that purpose -- in our application there is no Javascript to be found.
It also sounds like some people (e.g. Donovan Preston, a good hacker and a personal friend) think that producing and consuming decimal numbers (not floats) in JSON could lead to interoperability problems.  I can't imagine how that could happen -- if there is a program (say, in Javascript) which works when you send it a JSON document containing {{{0.10000000000000001}}} then I assume it will still work if you send it {{{0.1}}}.  Basically I expect that the extra {{{0.00000000000000001}}} is not a required part of the program's semantics but is just error, introduced by the use of floats, that immediately gets rounded off.  Certainly any program that parses its JSON numbers into floats will equate {{{0.10000000000000001}}} and {{{0.1}}}.  It would have to be doing something pretty funny if it broke when you sent it {{{0.1}}} and it was expecting {{{0.10000000000000001}}}.  However, I'm always open to the possibility that programs will break in even stupider ways than we imagined, so please let me know if you come across some situation where using Decimals instead of floats for your JSON causes a problem.
(Thanks to ~David-Sarah Hopwood for posting [[this note on es-discuss|https://mail.mozilla.org/pipermail/es-discuss/2009-January/008614.html]] which informed my opinions about this topic.)

There are several ~JavaScript crypto libraries with Free Software licences.  This is a table of pointers to the ones that I've looked at.  It would be great if everyone could start converging on a common API and common crypto algorithms so that the builders of ~JavaScript runtimes (i.e. the authors of web browsers) could consider implementing a native-code implementation of that API and those algorithms.  Until then, here are the 100%-pure-~JavaScript libraries.

(This list is sorted in descending order of the most important criterion.)

|!~JavaScript crypto library|!algs|!tests|!notes|!licence|
|[[SJCL|http://crypto.stanford.edu/sjcl/]]|AES,~SHA256,CMAC,some sort of PBKDF,semi-Fortuna,CCM,OCB,PMAC|good|fast; ECC "coming soon"|BSD or ~GPLv2+|
|[[clipperzlib|http://www.clipperz.com/open_source/javascript_crypto_library]]|AES,~SHA256,SRP,SRP,Fortuna|good|has preliminary ECC|GNU Affero|
|[[pidCrypt|https://www.pidder.com/pidcrypt/]]|AES,RSA,~SHA256|poor||GPL|
|[[jsbn|http://www-cs-students.stanford.edu/~tjw/jsbn/]]|RSA,a PRNG|none||BSD|
|[[jct|http://ats.oka.nu/titaniumcore/js/crypto]]|AES,Serpent,RSA,~SHA256,~RC4,|none|async,derived from jsbn,[[bug|http://lists.randombit.net/pipermail/botan-devel/2010-July/001183.html]]|LGPL|

There seem to be a lot of Python libraries which are for holding miscellaneous tools.  I'm the maintainer of one myself -- [[pyutil|http://pypi.python.org/pypi/pyutil]].  How many others are there?  I've decided to start accumulating pointers to the ones that I notice:

|!Python utility library|!licence|
|[[Voidspace Pythonutils|http://www.voidspace.org.uk/python/pythonutils.html]]|BSD|
|[[pyflu|http://pypi.python.org/pypi/pyflu]]|BSD|
|[[boduch|http://pypi.python.org/pypi/boduch]]|GNU Affero|
|[[pyutil|http://pypi.python.org/pypi/pyutil]]|GPL or TGPPL|
|[[itools|http://pypi.python.org/pypi/itools]]|GPL|
|[[RO|http://pypi.python.org/pypi/RO]]|GPL, except for one module|

A friend asked for reading suggestions for him to do a deep investigation of ketogenic dieting. I responded that there aren't any 

(Here's the link to [[the conversational context|http://www.overcomingbias.com/2009/01/open-thread.html#comment-144823758]].  See also my earlier post -- a book review: [[Good Calories, Bad Calories]].)

Hal:

You raise a good objection that you can't tell if Taubes is omitting important contrary data or arguments. How can one find out if such important contraries exist but are omitted? An efficient way to do so is debate, which for large written works like this takes the form of a rebuttal. If Taubes has omitted significant evidence or important argument, then people who know a lot about that evidence and argument and who believe Taubes is wrong can be relied upon to inform us about them.

I'm aware of one such rebuttal of "Good Calories, Bad Calories". (Only one!! If anyone has more, please let me know.) It is by Dr. George Bray, who is, according to low-carb guru Dr. Mike Eades "probably the most renowned figure in the field of obesity research today", and whose contributions to the field are mentioned in the book itself. Here's the link to the full rebuttal:

[[bray-review-of-gcbc.pdf|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/bray-review-of-gcbc.pdf]]

Unfortunately, Dr. Bray seems to have misunderstood or even failed to read important parts of the work he is rebutting, since he claims that the book omits the distinction between low-density lipoproteins and high-density lipoproteins, which it does not, and that it evinces a misunderstanding of the First Law of Thermodynamics, which it does not.

That last part is really the key: Dr. Bray and his colleagues are committing the classic error of looking at a relation and assuming the direction of causation. The First Law of Thermodynamics dictates that delta energy storage (roughly, weight gain), equals energy in minus energy out (given a few plausible assumptions about what counts as a "closed system" in this case). Everyone in the debate agrees on that point. What the First Law of Thermodynamics does not tell us is the direction of causation. Does energy imbalance cause obesity, or does obesity cause energy imbalance? (Or more complex combinations of causation?) Dr. Bray and company intuitively believe the former direction: they think the causation must flow from human decisions to eat more or less food, and human decisions to exercise more or less, to deposition of fat in human fat cells. This is not the only causal explanation which is consistent with the First Law of Thermodynamics, but Dr. Bray appears to think that it is. We can tell, because he seems to think that if Taubes disagrees with this causal direction, then Taubes must misunderstand the First Law of Thermodynamics. We can also tell by the way Bray asserts that direction of causation without justification, perhaps because he thinks it is too obvious to require justification or that it is the only logical explanation -- search in the text of his rebuttal for the phrase "result of".

I have a hypothesis about why so many well-versed researchers make this unjustified assumption: it is because of their belief in Free Will. If the arrow of causation has the pointy end aiming at human decisions, then this violates the notion that humans are free to choose their own fate, and this is either inconceivable or abhorrent. Therefore, the arrow of causation must have the blunt end towards human decisions and the pointy end towards weight gain. Taubes doesn't really explore the notion of Free Will in his book -- too bad. Room for follow-up work.

By the way, here is Taubes's rebuttal to Bray's rebuttal. Now that you've read mine, you don't need to read Taubes's so much. ;-)

[[taubes-response-to-bray-ob-reviews.pdf|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/taubes-response-to-bray-ob-reviews.pdf]]

Anyway, back to Hal's original question: how can you tell if Taubes is omitting some important pieces? I think rebuttal is the best way to tell. This rebuttal by Bray does point out some omissions in "Good Calories, Bad Calories", although unfortunately it also (I think) incorrectly alleges some other omissions. This points up the problem with this approach -- how do we know that Dr. Bray hasn't failed to notice more important omissions in "Good Calories, Bad Calories"? Especially since he made those two huge blunders I described above. That's why I'm hoping for more better rebuttals. But ultimately, we can't know. Taubes could be omitting tremendously important aspects in his book. Bray could be omitting to point out omissions. I'm still totally willing to put down cash on Taubes being the righter of the two (although I might want to first dig into those meta-studies the Bray mentions which covered five studies of law-carb diets). Too bad there's no legal, high-volume open market for such a bet.

(from [[a post to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-December/003254.html]])
I've been thinking that maybe the security community is the wrong market.  Most users, I've come to believe, will instinctively reach for the ''other'' tool if one of the tools is labelled as "secure".  This may sound strange, but I think it is true and that there is a good reason for it.  Users know that a tool which comes with a "security" sticker on it means more hoops they have to jump through before they can get their work done: pop-up dialogs asking "Are you sure?", key-management hassle, access-denied errors, etc..  They also know that most of the time bad guys aren't going to be attacking them and most of the time this tool isn't going to be the weakest link in the chain anyway.  In short, users are rational and correct when they pass over the products with "security" in favor of the products with "get your job done today".
Now we have always tried with ~Tahoe-LAFS to make something which provides security ''without'' introducing lots of hassle.  I think we've at least partially succeeded (although I'm still alert for more evidence from the field to indicate what's working and what isn't).  So maybe we should find some way to appeal to those people who just want a reliable and easy-to-use cloud storage tool and don't want an extra helping of "security".

write to me: [[zooko@zooko.com|mailto:zooko@zooko.com]]

blogroll:
[[Mark Seaborn|http://lackingrhoticity.blogspot.com]], [[Wes Felter|http://blog.felter.org/]], [[Trevor Stone|http://flwyd.livejournal.com]], [[Chris Hibbert|http://pancrit.blogspot.com]], [[Brad Templeton|http://ideas.4brad.com]], [[David-Sarah Hopwood|http://davidsarah.livejournal.com]], [[Ben Hyde|http://enthusiasm.cozy.org]], [[Nick Szabo|http://unenumerated.blogspot.com]], [[Danny O'Brien|http://www.oblomovka.com]], [[Mike Eades|http://proteinpower.com/drmike]], [[J.P. Calderone|http://jcalderone.livejournal.com]], [[Stephan Guyenet|http://wholehealthsource.blogspot.com]]

reading:
[[things to read]]; [[things read|things I have read]]
open source hacking:
[[issue tickets]]; [[tickets closed|issue tickets closed]]
trying to learn from failures: [[collection of failures]]

[img[Flattr this|/file/URI:CHK:5pw7deqgl67ds2dkpmn5jyk32e:cmjoyfjzkqennxouicimg5ivnkdmxjjtwiknxzrtg5ew4yxvu6sq:3:10:4724/@@named=button-compact-static-100x17.png][http://flattr.com/thing/68935/Zookos-klog]]
[[about this button|Flattr me]]

[img[RSS feed|/file/URI:CHK:2jfor3byn52fuay7wivxruniii:vpcxn23bb3romjbhglbonu2goe2rl32perflhzkamjqk3jbsi2lq:3:10:689/@@named=feed-icon-14x14.png][http://pubgrid.tahoe-lafs.org/uri/URI%3ADIR2-RO%3Aixqhc4kdbjxc7o65xjnveoewym%3A5x6lwoxghrd5rxhwunzavft2qygfkt27oj3fbxlq4c6p45z5uneq/blog.xml]] [[RSS feed for this blog|http://pubgrid.tahoe-lafs.org/uri/URI%3ADIR2-RO%3Aixqhc4kdbjxc7o65xjnveoewym%3A5x6lwoxghrd5rxhwunzavft2qygfkt27oj3fbxlq4c6p45z5uneq/blog.xml]]

Inuit. I've read these articles:
* [["Traditional plant foods of Canadian indigenous peoples: nutrition, botany ..."|http://books.google.com/books?id=fPDErXqH8YYC&source=gbs_navlinks_s]] By Harriet V. Kuhnlein, Nancy J. Turner; much uncertainty, but some reason to believe that various Inuit tribes ate a few plant foods sometimes.
* [[Guts and Grease: The Diet of Native Americans|http://www.westonaprice.org/Guts-and-Grease-The-Diet-of-Native-Americans.html]] by Sally Fallon Morell and Mary Enig
* [[War|http://www.nunatsiaqonline.ca/archives/nunavut000131/nunani.html]]; newspaper series on Inuit psychology of violence
* [[Examining the Inuit, Thule and Dorset Using Current Hunter-Gatherer Risk Models|http://www.associatedcontent.com/pop_print.shtml?content_type=article&content_type_id=1377748]] subtitled "Retrospective Inuit Anthropology Since Man the Hunter, 1966" By Leo Pucklis; infanticide
* [[Cancer Among the Inuit|http://wholehealthsource.blogspot.com/2008/07/cancer-among-inuit.html]] blog post by Stephan Guyenet, 2008-06-04
* [[Mortality and Lifespan of the Inuit|http://wholehealthsource.blogspot.com/2008/07/mortality-and-lifespan-of-inuit.html]] blog post by Stephan Guyenet, 2008-06-05
----
Inuit. Books I'd be interested in.
* Vilhalmur Stefannson: //[[The Fat of the Land|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Stefansson_1956-The_Fat_of_the_Land.pdf]]// (book, 1956) (<-- PDF version hosted on ~Tahoe-LAFS)
* [[Overland to Starvation Cove|http://openlibrary.org/b/OL7873668M/Overland_to_Starvation_Cove]] by Heinrich Klutschak; first-hand account of some Inuit in the 1880's
* [[White Lies About the Inuit|http://www.utphighereducation.com/product.php?productid=880]] by John Steckley
----
Hunza. I've read these articles:
* [[Death Rides a Slow Bus in Hunza|http://www.alkalizeforhealth.net/Lhunzadiet2.htm]] by Jane Kinderlehrer; secondary source; mostly included in this list as an example of the sort of myth and speculation that one finds on the topic of Hunza diet and health; but it does cite a few sources
* [[The Truth, Myths, and Lies About the Health and Diet of the "Long-Lived" People of Hunza|http://www.biblelife.org/hunza.htm]]; a detailed secondary source with a strong bias and off-putting style. He relies words like "prove" and "scientific", which are signals of an unscientific approach! Also his domain name is "biblelife.org", which sounds unscientific to me. :-) Nonetheless, his view on Hunza lifestyle and health appears quite plausible to me. It appears that he is drawing mostly from three books: John Clark ''Hunza - Lost Kingdom of the Himalayas'', 1957 [[free on-line in PDF|http://biblelife.org/Hunza%20-%20Lost%20Kingdom%20of%20the%20Himalayas.pdf]] and Dr. Allen E. Banik and Renee Taylor ''Hunza Land'' 1960 and Renee Taylor ''Hunza Health Secrets For Long Life and Happiness'' 1964.
----
----
Health theory. To-read:
* Stefania Piconi, Daria Trabattoni, Cristina Luraghi, Edoardo Perilli, Manuela Borelli, Michela Pacei, Giuliano Rizzardini, Antonella Lattuada, Dorothy H. Bray, Mariella Catalano, Antonella Sparaco and Mario Clerici: [[Treatment of periodontal disease results in improvements in endothelial dysfunction and reduction of the carotid intima-media thickness|http://www.fasebj.org/cgi/content/abstract/23/4/1196]] ([[as recommended|http://twitter.com/DrEades/status/10185869017]] by Dr. Mike Eades)
Health theory. I haven't really read these, but I've looked at the abstract of each one enough to think that I want to keep a note of it.
* [[The Effects of Macronutrient Intake on Total and High-molecular Weight Adiponectin: Results From the OMNI-Heart Trial|http://www.nature.com/oby/journal/vaop/ncurrent/abs/oby2009402a.html]] (referenced by [[Dr. Mike Eades|http://www.proteinpower.com/drmike/]] who tweeted "High-fat diet increases adiponectin more than does a high-carb or high-protein diet.")
* [[Würzburg Diet for Cancer Patients|http://comfort-eaters-diet.blogspot.com/2010/01/wurzburg-diet-for-cancer-patients.html]]; Not a study but a blog post about a low-carb diet programme for cancer patients published by the University of Würzburg.
* David N. Ruskin, Masahito Kawamura, Jr, Susan A. Masino: [[Reduced Pain and Inflammation in Juvenile and Adult Rats Fed a Ketogenic Diet|http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0008349]] (recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]] who posted on twitter: "Ketogenic diet reduces pain and inflammation in rodents. My experience is that it does in humans, too.")
----
''Good Calories, Bad Calories''
* ''Good Calories, Bad Calories'', 2006, by Gary Taubes, is the thing to read first if you are seriously interested in diet and health. The book is a tour de force which chronicled the history of the subject for the first time and which also made history itself -- future histories of human diet and health will be divided into "before GCBC" and "after GCBC". :-) You have to be seriously interested in reading a lot of details, though. If you aren't ''that'' excited about studying the topic, then nevermind. I'll get back to you when I find a book which is quick and fun and also useful. Here's my review: [[Good Calories, Bad Calories]].
----
Studies I've read
''There are a whole bunch of studies that I've looked at or in some cases actually read that I forgot to enter into this tiddler.  Sorry about that. Hopefully I'll get around to linking them all in...''
Click on the date stamp to see my journal for that day which includes my notes on the study.
* [[2010-02-17]]
** R. Neil A. Black, Michelle Spence, Ross O. ~McMahon, Geraldine J. Cuskelly, Cieran N. Ennis, David R. ~McCance,Ian S. Young, Patrick M. Bell, and Steven J. Hunter: [[Effect of Eucaloric High- and Low-Sucrose Diets With Identical Macronutrient Profile on Insulin Resistance and Vascular Risk|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Black_et_al-2006-Effect_of_Eucaloric_High-_and_Low-Sucrose_Diets_With_Identical_Macronutrient_Profile_on_Insulin_Resistance_and_Vascular_Risk.pdf]] (as recommended by Brendan O'Connor)

The dominant business model of the consumer web is that you give service to consumers in return for their private data which you can use to profit, such as through advertising.  Google is a pre-eminent example of this model, and Facebook is a promising new entrant.

So, purveyors of web services aimed at consumers so far assume that consumers are happy to give up privacy and control over their data in return for the services offered, and so far they appear to be right.

But, the nebulous storm of "cloud computing" is partially caused by businesses wanting to get the same convenience and cost-savings that consumers currently have.  And businesses are a lot less happy to give up confidentiality and control of their private data just so that their ''service provider'' can profit from it!

Currently they have unpleasant alternatives -- they can keep incurring the expense and limitations of owning and operating their own service, or they can give up on confidentiality and control in order to benefit from "the cloud".

[[Tahoe-LAFS|http://allmydata.org]] is all about the third alternative -- the convenience and cost-savings of leasing service from a pro, but the confidentiality and control of keeping your own data in your own house.

The best business model that I can think of for ~Tahoe-LAFS is simply that you pay me and I give you [[good storage service|http://allmydata.com]].  Pretty boring.  :-)  ~Tahoe-LAFS's architecture even makes it harder for me to lock you in!  What a stupid architecture!

P.S.  I guess the flip side of my observation that the model doesn't currently extend to businesses would be to try to make it do so!  Maybe I'll give your company free payroll services because then I get to learn all about your employees so I can advertise to them or something.  Crazy!  But who knows.

I saw two new studies today both of which are comprehensive "meta-analyses" which re-examine all of the studies from the past, comparing them to one another and doing careful statistical tests.
* Patty W ~Siri-Tarino, Qi Sun, Frank B Hu and Ronald M Krauss: [[Meta-analysis of prospective cohort studies evaluating the association of saturated fat with cardiovascular disease|http://www.ajcn.org/cgi/content/abstract/ajcn.2009.27725v1?papetoc]]: "There is no significant evidence for concluding that dietary saturated fat is associated with an increased risk of [heart disease]."
* C. Murray Skeaff, and Jody Miller: [[Dietary Fat and Coronary Heart Disease: Summary of Evidence from Prospective Cohort and Randomised Controlled Trials|http://content.karger.com/produktedb/produkte.asp?typ=fulltext&file=000229002]]: "... results showed no association between [saturated fat] intake and [heart disease]"
Yes, yes indeed. Eating saturated fat does not increase your risk of heart disease. Nor does the amount of total fat that you eat. I'm unsurprised because of [[Good Calories, Bad Calories]] (2007) as well as because of a pile of other evidence that's turned up in the last ten years. These are just a couple more nails in the coffin of the “diet-heart” hypothesis.

Okay now what? This means that a whole bunch of ideas about human health that we've relied on for the past 40 years were wrong, and now we have a chance to revisit those issues and come up with better theories and improve everyone's lives. Let's get on with it!

The human costs of this particular set of wrong beliefs extend beyond heart disease to include obesity, diabetes, alzheimer's, most cancers, and perhaps more.  Cardiovascular disease probably kills about 16 million people each year //(source: [[1|http://www.americanheart.org/downloadable/heart/1077185395308FS06INT4(ebook).pdf]])//.  There are probably about 15 million people living with alzheimer's //(sources: [[2|http://www.alz.co.uk/research/statistics.html]], [[3|http://www.alz.org/alzheimers_disease_facts_figures.asp]])//.  There are probably something on the order of 200 million people living with diabetes //(source: [[4|http://www.who.int/diabetes/facts/world_figures/en/]])//.  There are probably about 700 million people living with obesity //(source: [[5|http://www.who.int/mediacentre/factsheets/fs311/en/index.html]])//. (For comparison, there are about 33 million people living with HIV //(source: [[6|http://www.avert.org/worlstatinfo.htm]])//.)

These epidemics of obesity and diabetes, which began in the late 20th century, are probably in part ''caused'' by the bad advice given by scientists and officials based on this now-discredited theory that consumption of fat and in particular saturated fat is bad for you.  Unfortunately changing the course of a vast, centralized scientific bureaucracy is like changing the course of a supertanker.

For example, my source for the statistic about global obesity, above, is the //World Health Organization//.  Their web page about the obesity epidemic comes with [[this dietary advice|http://www.who.int/dietphysicalactivity/diet/en/index.html]], which is based on the discredited theory and which if heeded will subject the reader to ''increased'' risks of the devastating diseases mentioned above. (Exception: the line about eating less sugar is right.  The rest is wrong and is either useless or harmful.)

For another example my father had a major heart attack in 2007 and his cardiologist and nutritionist have instructed him to reduce consumption of fat and especially of saturated fat. This is certainly not good for his health, and it is probably bad for it.

Once they get established, good scientific theories die hard.  Bad ones die harder.  This is a bad one -- it was never justified by the evidence that was available at the time. I wonder for how many more years the health authorities will cling to it, and how many millions more will suffer and die during that period.

 ([[My notes on Skeaff and Miller 2009|my notes on Skeaff and Miller 2009]], which can serve as an appendix to this blog entry.)

Here are some things I have read.  Click on the date stamp before each item to see if I commented about it on my journal that day.
(Note: I often read things and don't get around to adding them to this list.)

* [[2010-12-10]]
** interesting short fiction story: //[[A Troll Story|http://nicolagriffith.com/troll.html]]// by Nicola Griffith
* [[2010-11-27]]
** Timo E. Strandberg, MD; Veikko V. Salomaa, MD; Vesa A. Naukkarinen, MD; Hannu T. Vanhanen, MD; Seppo J. Sarna, ~PhD; Tatu A. Miettinen, MD [[Long-term Mortality After 5-Year Multifactorial Primary Prevention of Cardiovascular Diseases in Middle-aged Men|http://jama.ama-assn.org/cgi/content/abstract/266/9/1225]] ([[PDF copy hosted on Tahoe-LAFS|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Strandberg-1991-Long-term_Mortality_After_5-Year_Multifactorial_Primary_Prevention_of_Cardiovascular_Diseases_in_Middle-aged_Men.pdf]])

…
I read many things during this interval and did not note them here…
…

* [[2010-10-17]]:
** David S. H. Rosenthal, Thomas Lipkis, Thomas S. Robertson, and Seth Morabito (the LOCKSS project) [[Transparent Format Migration of Preserved Web Content|http://www.dlib.org/dlib/january05/rosenthal/01rosenthal.html]]

…
I read many things during this interval and did not note them here…
…

* [[2010-04-12]]:
** Hu et al. [[Dietary Fat Intake and the Risk of Coronary Heart Disease in Women|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Hu-1997-Dietary_Fat_Intake_and_the_Risk_of_Coronary_Heart_Disease_in_Women.pdf]] (1997)
** Oh et al. [[Dietary Fat Intake and Risk of Coronary Heart Disease in Women: 20 Years of Follow-up of the Nurses' Health Study|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Oh-2005-Dietary_Fat_Intake_and_Risk_of_Coronary_Heart_Disease_in_Women:_20_Years_of_Follow-up_of_the_Nurses'_Health_Study.pdf]] (2005)
* [[2010-03-18]]
** Daniel Suarez: //[[Daemon|http://thedaemon.com/]]// (book)
** Jonathan Lange: [[Your Code Sucks and I Hate You: The Social Dynamics of Code Reviews|http://mumak.net/stuff/your-code-sucks.html]] ([[as recommended|http://twitter.com/dreid/status/10690810313]] by David Reid)
* [[2010-03-02]]
** Ellen Ferrante, Bobbie Mixon: [[Breakthrough in Electron Spin Control Brings Quantum Computers Closer to Reality|http://www.nsf.gov/discoveries/disc_summ.jsp?cntn_id=116456]] (press release)
* [[2010-03-01]]:
** Leila Gray, Dr. Philippe P. Hujoel: [[If A Diet Is Bad For The Teeth It Is Also Bad For The Body|http://www.medicalnewstoday.com/articles/157105.php]] (press release) ([[as recommended|http://twitter.com/DrEades/status/9692921690]] by [[Dr. Mike Eades|http://www.proteinpower.com/drmike/]])
* [[2010-02-24]]:
** [[Stephan Guyenet|http://wholehealthsource.blogspot.com/]]: [[Magnesium and Insulin Sensitivity|http://wholehealthsource.blogspot.com/2010/02/magnesium-and-insulin-sensitivity.html]] (blog entry)
** Inna Slutsky, Nashat Abumaria, ~Long-Jun Wu, Chao Huang, Ling Zhang, Bo Li, Xiang Zhao, Arvind Govindarajan, ~Ming-Gao Zhao, Min Zhuo, Susumu Tonegawa, and Guosong Liu: [[Enhancement of Learning and Memory by Elevating Brain Magnesium|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Slutsky-2010-Enhancement_of_Learning_and_Memory_by_Elevating_Brain_Magnesium.pdf]] and [[the press release about it|http://www.sciencedaily.com/releases/2010/02/100222162011.htm]] (as recommended by some commenters on that blog entry of Stephan Guyenet's)
* [[2010-02-17]]
** R. Neil A. Black, Michelle Spence, Ross O. ~McMahon, Geraldine J. Cuskelly, Cieran N. Ennis, David R. ~McCance,Ian S. Young, Patrick M. Bell, and Steven J. Hunter: [[Effect of Eucaloric High- and Low-Sucrose Diets With Identical Macronutrient Profile on Insulin Resistance and Vascular Risk|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Black_et_al-2006-Effect_of_Eucaloric_High-_and_Low-Sucrose_Diets_With_Identical_Macronutrient_Profile_on_Insulin_Resistance_and_Vascular_Risk.pdf]] (as recommended by Brendan O'Connor)
* [[2010-02-09]]
** Jennifer B Keogh, Grant D Brinkworth, Manny Noakes, Damien P Belobrajdic, Jonathan D Buckley, and Peter M Clifton: [[Effects of weight loss from a very-low-carbohydrate diet on endothelial function and markers of cardiovascular disease risk in subjects with abdominal obesity|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Keogh-2008-Effects_of_weight_loss_from_a_very-low-carbohydrate_diet_on_endothelial_function_and_markers_of_cardiovascular_disease_risk_in_subjects_with_abdominal_obesity.pdf]] <-- that link is a copy of the article stored on this ~Tahoe-LAFS grid; Here is the page for the article [[at the American Journal of Clinical Nutrition|http://www.ajcn.org/cgi/content/abstract/87/3/567]]
* [[2010-01-14]]
** Patty W ~Siri-Tarino, Qi Sun, Frank B Hu and Ronald M Krauss: [[Meta-analysis of prospective cohort studies evaluating the association of saturated fat with cardiovascular disease|http://www.ajcn.org/cgi/content/abstract/ajcn.2009.27725v1?papetoc]] (as recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]]) (I read only the abstract)
** C. Murray Skeaff, and Jody Miller: [[Dietary Fat and Coronary Heart Disease: Summary of Evidence from Prospective Cohort and Randomised Controlled Trials|http://content.karger.com/produktedb/produkte.asp?typ=fulltext&file=000229002]] (as recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]])
* [[2010-01-12]]
** Jian Guo, San Ling, Christian Rechberger, and Huaxiong Wang: [[Advanced Meet-in-the-Middle Preimage Attacks: First Results on Full Tiger, and Improved Results on MD4 and SHA-2|http://eprint.iacr.org/2010/016.pdf]]
* [[2010-01-11]]
** Kazuyuki Kobayashi, Jun Ikegami, Shin’ichiro Matsuo, Kazuo Sakiyama and Kazuo Ohta: [[Evaluation of Hardware Performance for the SHA-3 Candidates Using SASEBO-GII|http://eprint.iacr.org/2010/010.pdf]]
* [[2010-01-04]]
** Jonathan Lehrer writing in Wired Magazine: [[Accept Defeat: The Neuroscience of Screwing Up|http://www.wired.com/magazine/2009/12/fail_accept_defeat/all/1]] (recommended by [[Dr. Mike Eades|http://proteinpower.com/drmike]] on his twitter stream)
** Nate Lawson: [[Side channel attacks on cryptographic software|http://rdist.root.org/2009/12/30/side-channel-attacks-on-cryptographic-software/]]
* [[2010-01-02]]
** Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno: a draft of the next edition of //[[Practical Cryptography|http://www.schneier.com/book-practical.html]]//
** Michael Pollan: //[[The Omnivore's Dilemma|http://www.michaelpollan.com/omnivore.php]]//
* [[2009-12-28]]
** Chris Palmer's [[Open Letter to the Free Software Community|http://noncombatant.org/blog/2009-12-29-01:40.html]]
** Trygve Tollefsbol, Gerald Weissmann, et al.: [[Calorie Restriction: Scientists Take Important Step Toward 'Fountain of Youth'|http://www.sciencedaily.com/releases/2009/12/091222105219.htm]]
* [[2009-12-25]]
** Chris Williams writing for The Register: [[Software fraudster 'fooled CIA' into terror alert|http://www.theregister.co.uk/2009/12/24/cia_montgomery/]]
* [[2009-12-09]]
** Eduardo Pinheiro, ~Wolf-Dietrich Weber, and Luiz André Barroso (all of Google Inc.): [[Failure Trends in a Large Disk Drive Population|http://labs.google.com/papers/disk_failures.pdf]]
* [[2009-12-02]]
** Kevin D. Bowers, Ari Juels, Alina Oprea: [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]]
** Vadim Lyubashevsky, Adriana Palacio, Gil Segev: [[Public-Key Cryptographic Primitives Provably as Secure as Subset Sum|http://eprint.iacr.org/2009/576]]
** Tom Knox: [[Do these mysterious stones mark the site of the Garden of Eden?|http://www.dailymail.co.uk/sciencetech/article-1157784/Do-mysterious-stones-mark-site-Garden-Eden.html]] (as recommended by Kris Nuttycombe)
* [[2009-11-19]]
** Jen Lexmond, Richard Reeves: [[Building Character|../../file/URI%3ACHK%3Arev3ooozvxyxcm3ek56pxe5qza%3Amzwkmziexzwjyrly3p2wvzxu7hy6bxevwumitz32hem3ux3y4meq%3A3%3A10%3A749145/@@named=/Building_Character_Web.pdf]]
** Anil Dash: [[The Web in Danger|../../file/URI%3ACHK%3Aymeendyhlglzbqomu7c2ppo2re%3Azaigwylxh3gyt6lrfzhwm4n3v534nn6zl3z5hnefybklikopep5a%3A3%3A10%3A63751/@@named=/the-web-in-danger.html]]
* [[2009-09-11]]
** Ian Bicking: [[Toward a new self-definition for open source|http://blog.ianbicking.org/2009/09/10/a-new-self-definition-for-foss]]
* [[2009-09-05]]
** Daniel J. Bernstein: [[Understanding brute force|http://cr.yp.to/snuffle/bruteforce-20050425.pdf]] (short answer: use 256-bit keys! Too bad that following this advice would make ~Tahoe-LAFS read-caps unwieldy.)
* [[2009-08-30]]
** Apple Computer: [[Grand Central Dispatch: A better way to do multicore|http://images.apple.com/macosx/technology/docs/GrandCentral_TB_brief_20090608.pdf]]
** Steven R. Hetzler: [[The Storage Chasm: Implications for the Future of HDD and Solid State Storage|http://www.caiss.org/docs/DinnerSeminar/TheStorageChasm20090205.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/08/28]])
* [[2009-08-25]]
** Richard K. Morgan: //[[The Steel Remains|http://books.google.com/books?id=AsRF3s1MK08C&dq=the+steel+remains&printsec=frontcover&source=bl&ots=keAI_8SMcl&sig=DKVlGa63-Q0r9pI5PLsQKGNQnLI&hl=en&ei=mrOSSs2xAonuswOUo4wM&sa=X&oi=book_result&ct=result&resnum=8#v=onepage&q=&f=false]]// (a gift from Tom Christiansen)
* [[2009-08-13]]:
** ~David-Sarah Hopwood: [[Jacaranda programming language specification, draft v0.42|http://jacaranda.org/jacaranda-spec-0.44.txt]]
** [[Doug Crockford|http://www.crockford.com]]: //[[JavaScript: The Good Parts|http://oreilly.com/catalog/9780596517748/]]//
* [[2009-08-11]]
** Hugo Krawczyk: [[On Extract-then-Expand Key Derivation Functions and an HMAC-based KDF|http://webee.technion.ac.il/~hugo/kdf/]]
* [[2009-08-04]]
** Alex Biryukov, Orr Dunkelman, Nathan Keller, Dmitry Khovratovich, Adi Shamir: [[Key Recovery Attacks of Practical Complexity on AES Variants With Up To 10 Rounds|http://eprint.iacr.org/2009/374]]
** Christopher Soghoian [[Manipulation and abuse of the consumer credit reporting agencies|http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2583/2246]]
* [[2009-07-29]]
** Ron Rivest: [[All-Or-Nothing Encryption and The Package Transform (1997)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.1059]]
* [[2009-07-28]]
** Norm Hardy: [[The Confused Deputy (or why capabilities might have been invented)|http://www.cis.upenn.edu/~KeyKOS/ConfusedDeputy.html]] (I just now re-re-read this paper.  It is short, informal, and ancient, but it is the best summary of a critical modern security issue.)
** The Economist: [[Financial economics: Efficiency and beyond|http://www.economist.com/displaystory.cfm?story_id=14030296]] (note: fragile URL; I should copy the document onto a ~Tahoe-LAFS grid and link to the copy.)
* [[2009-07-25]]
** William Vambenepe: [[REST in practice for IT and Cloud management (part 1: Cloud APIs)|http://stage.vambenepe.com/archives/863]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/07/16]])
* [[2009-07-24]]
** Tillich and Martin Feldhofer, Wolfgang Issovits, Thomas Kern, Hermann Kureck, Michael Mühlberghuber, Georg Neubauer, Andreas Reiter, Armin  Köfler, Mathias Mayrhofer: [[Compact Hardware Implementations of the SHA-3 Candidates ARIRANG, BLAKE, Grøstl, and Skein|http://eprint.iacr.org/2009/349]]
* [[2009-07-15]]:
** //[[The Warriors book 4: Rising Storm|http://www.amazon.com/Rising-Storm-Warriors-Book-4/dp/0060525630/ref=sr_1_1?ie=UTF8&s=books&qid=1244570985&sr=8-1]]// by Erin Hunter
** Leslie Lamport: [[Paxos Made Simple|http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf]]
* [[2009-06-16]]:
** Hal Finney's [[summary of Craig Gentry's deep homomorphic encryption|http://www.mail-archive.com/cryptography@metzdowd.com/msg10571.html]]
** [[the slides and transcripts from the first SHA-3 conference|http://csrc.nist.gov/groups/ST/hash/sha-3/Round1/Feb2009/program.html]]
* [[2009-05-30]]:
** Gautham Sekar, Souradyuti Paul, Bart Preneel: [[Related-key Attacks on the Py-family of Ciphers and an Approach to Repair the Weaknesses|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.92.3976]]
* [[2009-05-27]]
** George Bray: [[Obesity: a failure of homeostasis because of hedonic rewards: response to the letter from Gary Taubes|http://testgrid.allmydata.com:3567/file/URI:CHK:q3cwozbit4be2jxmxfre65wzdq:h6rm26yuvcjttexlwhdi6lfcgqoxkordv52xxkwyhxphkytcfuva:3:10:116345/@@named=/Bray_Taubes_rebuttal_rebuttal_rebuttal.pdf]] //(thanks to Gary Taubes for bringing it to my attention as I requested in [[rebuttals to "Good Calories, Bad Calories"]])//
* [[2009-05-24]]
** Henry Harpending: [[Henry And The Cape Buffalo|http://the10000yearexplosion.com/henry-and-the-cape-buffalo]]
** Gardner Dozois (ed.): [[The Year's Best Science Fiction: Twenty-Second Annual Collection|http://www.amazon.com/exec/obidos/ASIN/0312336608]]
* [[2009-05-09]]
** Vijay Vasudevan, Jason Franklin, David Andersen, Amar Phanishayee, Lawrence Tan, Michael Kaminsky, Iulian Moraru: [[FAWNdamentally Power-efficient Clusters|http://www.cs.cmu.edu/~vrv/papers/hotos2009.pdf]], as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/05/07]]
* [[2009-05-05]]
** John C. Wright: //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// trilogy
** Jeff Bonwick (inventor of ZFS): [[on timeouts|http://storagemojo.com/2008/11/25/stupid-storage-failures/#comments]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/11/29]])
** Oscar Bonilla: [[Visualizing Bayes' Theorem|http://oscarbonilla.com/2009/05/visualizing-bayes-theorem]]
* [[2009-04-29]]
** Stefan Tillich: [[Hardware Implementation of the SHA-3 Candidate Skein|http://eprint.iacr.org/2009/159]]; See journal entry on [[2009-04-29]] for comments.
* 2009-01-20 (By the way, I read a lot of things between 2009-01-20 and 2009-04-29, but I forgot to note them here.  Sorry.)
** Ewan Fleischmann, Christian Forler, and Michael Gorski: [[Classification of the SHA-3 Candidates (2009-09-19 edition)|http://eprint.iacr.org/2008/511]]
* 2009-01-18
** Huseyin Hisil, Kenneth ~Koon-Ho Wong, Gary Carter, and Ed Dawson: [[Twisted Edwards Curves Revisited|http://eprint.iacr.org/2008/522]]; I don't really understand much of the math, but it looks like elliptic curve cryptography is going to get more efficient and safer as results like these trickle down to practice.
** James Hamilton: [[The Case For Low-Cost, Low-Power Servers|http://perspectives.mvdirona.com/2009/01/15/TheCaseForLowCostLowPowerServers.aspx]] (via [[Wes Felter's blog|http://wmf.editthispage.com/2009/01/16]]), and posted a question about it to James Hamilton's blog
* 2009-01-17
** Peter Gutmann: [[The Crypto Gardening Guide and Planting Tips|http://www.cs.auckland.ac.nz/~pgut001/pubs/crypto_guide.txt]]
* earlier
** James Hamilton: [[The Cost of Bulk Cold Storage|http://perspectives.mvdirona.com/2008/12/22/TheCostOfBulkColdStorage.aspx]] as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/27]]
** [[Some Results from EDOS on Dependency Checking|http://mancoosi.org/edos]]

*  Scott Contini, Ron Steinfeld, Josef Pieprzyk, Krystian Matusiewicz: [[A critical look at cryptographic hash function literature (2007)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]]

* finish letter to Shawn, Josh, tahoe-dev //Re: [tahoe-dev] Fwd: On the value of "proofs"...//
* read ~David-Sarah Hopwood //[[Jacaranda Language Specification, draft 0.3|http://www.eros-os.org/pipermail/e-lang/2008-August/012974.html]]// and twenty-six other [[things to read]]
* write letter to friam about provenance of the git ~Cryptographic-Hash-Function-Directed-Acyclic-Graph (~CHF-DAG) idea, starting with Graydon Hoare's recollections posted to uvc-reviewers
* reduce the number of incomplete drafts of e-mails in my Drafts folder -- currently twenty-six (hm, twenty-six Things To Read, twenty-six unfinished drafts of Things For Other People To Read...)
* catch up on reading other people's blogs
* talk about [[darcsver|http://allmydata.org/trac/darcsver]] with Brian

(Copied from [[2009-02-07]]:)
* //[[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail// (pushed out of the 1.3.0 milestone)
* write back on [[this thread on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001056.html]] and say "No, no, it really is almost as easy as Shawn originally thought.  Go, Shawn, go."
* figure out why setuptools/zetuptoolz is rebuilding things when {{{python ./setup.py test}}} after it just built them when {{{python ./setup.py build}}} //[[tahoe #591|http://allmydata.org/trac/tahoe/ticket/591]]: "make quicktest" could be quicker and less noisy// (added to [[issue tickets]])
* pycryptopp improvements -- link against system libcryptopp.so for Debian and Fedora packagers (//[[pycryptopp #9|http://allmydata.org/trac/pycryptopp/ticket/9]]: link against existing (system) libcrypto++.so//), add new improved ECDSA (//[[pycryptopp #13|http://allmydata.org/trac/pycryptopp/ticket/13]]: DSA "semi-private"/intermediate keys//, //[[pycryptopp #2|http://allmydata.org/trac/pycryptopp/ticket/2]]: deterministic generation of private key from small seed//, //[[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff//, //[[pycryptopp #11|http://allmydata.org/trac/pycryptopp/ticket/11]]: submit patch for Brainpool ECC to Crypto++//), build out more pycryptopp buildbots, etc., (all added to [[issue tickets]])
* accounting/garbage-collection/quotas/etc.
* write up all those documents described in [[2009-02-06]]
* write up my new idea for immutable crypto caps
* start backing up all my personal files with tahoe
* one zillion other things

Here are some things that I hope I'll have time to read someday.  I'm posting them here so that (a) I can easily find them again or be reminded of their existence, and (b) people who've already read them or who otherwise have useful things to share with me will know that I'm interested in them.

[[computer security]] and reliability:
* Amir Herzberg: [[Folklore, Practice and Theory of Robust Combiners|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Herzberg_2008-Folklore,_Practice_and_Theory_of_Robust_Combiners.pdf]]
* Nik Cubrilovic (in ~TechCrunch): [[The Anatomy of the Twitter Attack|../../file/URI:CHK:nm72blax6oqt3fui3dnrhahszq:wcpjaneyqzf4bw752izfey44abql6ywync2vweejsmnohyiwkkia:3:10:275196/@@named=/the-anatomy-of-the-twitter-attack.html]]
* Tim Nufire of Backblaze: [[Petabytes on a budget: How to build cheap cloud storage|http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/]] (blog entry)
* Algirdas Avizienis, ~Jean-Claude Laprie, Brian Randell, and Carl Landwehr: [[Basic Concepts and Taxonomy of Dependable and Secure Computing|http://www.cs.umass.edu/~ransford/srg/papers/avizienis--dependable-secure.pdf]] (recommended by Ludovic Courtès)
* Paul Crowley: [[Squaring Zooko’s Triangle, part two|http://www.lshift.net/blog/2007/11/21/squaring-zookos-triangle-part-two]] (blog entry)
* Kevin Bauer, Damon ~McCoy, Douglas Sicker, Dirk Grunwald: [[BitBlender: Light-Weight Anonymity for BitTorrent|http://systems.cs.colorado.edu/~bauerk/papers/alpaca2008.pdf]]
* Ken Thompson: [[Reflections on Trusting Trust|http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf]]
* Helen J. Wang, Xiaofeng Fan, Jon Howell, Collin Jackson: [[Protection and Communication Abstractions for Web Browsers in MashupOS|http://research.microsoft.com/en-us/um/people/helenw/papers/sosp07mashupos.pdf]] (also on the topic of [[operating systems]])
* Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]] and [[the ensuing conversation on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2009-January/012030.html]]
* [[Martin C. Atkins|http://www.mca-ltd.com/martin/]]: [[An Introduction to Ten15 - A personal retrospective.|http://www.mca-ltd.com/martin/Ten15/introduction.html]]
* [[Carl Hewitt|http://carlhewitt.info]] ([[on wikipedia|http://en.wikipedia.org/wiki/Carl_Hewitt]]): [[A historical perspective on developing foundations for privacy-friendly client cloud computing: The Paradigm Shift from “Inconsistency Denial” to “Semantic Integration”|http://perspective.carlhewitt.info]]
* Ben Laurie, Abe Singer: [[Choose the Red Pill and the Blue Pill|http://www.links.org/files/nspw36.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/08]])
* Ben Laurie, Eric Sachs: [[Usability of Stronger Authentication Options|http://sites.google.com/site/oauthgoog/UXFedLogin/strongauth]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/03]])
* google: [[Browser Security Handbook|http://code.google.com/p/browsersec/wiki/Main]]

[[cryptography]] (which means also [[computer security]] and safety engineering):
* Robert N. Charette: [[Automated to Death|http://spectrum.ieee.org/computing/software/automated-to-death/0]] (recommended by [[Brian Warner|http://en.wikipedia.org/wiki/Brian_Warner]])
* Brian Baldwin and Andrew Byrne and Mark Hamilton and Neil Hanley and Robert P. ~McEvoy and Weibo Pan and William P. Marnane: [[FPGA Implementations of SHA-3 Candidates:CubeHash, Grøstl, LANE, Shabal and Spectral Hash|http://eprint.iacr.org/2009/342]]
* Alexandra Boldyreva and Serge Fehr and Adam O'Neill: [[On Notions of Security for Deterministic Encryption, and Efficient Constructions without Random Oracles|http://eprint.iacr.org/2008/352]] (it is directly relevant to [[convergent encryption reconsidered|http://hacktahoe.org/drew_perttula.html]])
* Daniel R. L. Brown and Robert P. Gallant: [[The Static Diffie-Hellman Problem|http://eprint.iacr.org/2004/306]] (recommended by Dcoder)
* Aniket Kate and Ian Goldberg: [[Asynchronous Distributed Private-Key Generators for Identity-Based Cryptography|http://eprint.iacr.org/2009/355]]
* Dimitrios Poulakis: [[Small Solutions of Bivariant Modular Equations and the security of DSA and ECDSA|http://eprint.iacr.org/2009/363]]
* Nishanth Chandran and Vipul Goyal and Ryan Moriarty and Rafail Ostrovsky: [[Position Based Cryptography|http://eprint.iacr.org/2009/364]]
* Abhishek Parakh and Subhash Kak: [[Space Efficient Secret Sharing: A Recursive Approach|http://eprint.iacr.org/2009/365]]
* Boris Skoric: [[Quantum readout of Physical Unclonable Functions: Remote authentication without trusted readers and authenticated Quantum Key Exchange without initial shared secrets|http://eprint.iacr.org/2009/369]]
* Rosario Gennaro and Shai Halevi: [[More on Key Wrapping|http://eprint.iacr.org/2009/372]]
* Masao KASAHARA: [[Forgotten Secret Recovering Scheme and Fuzzy Vault Scheme Constructed Based on Systematic Error-Correcting Codes|http://eprint.iacr.org/2009/375]]
* Manoj Kumar: [[A Registration Scheme to Allocate a Unique Identification Number|http://eprint.iacr.org/2009/383]]
* Joppe W. Bos and Marcelo E. Kaihara and Thorsten Kleinjung and Arjen K. Lenstra and Peter L. Montgomery: [[On the Security of 1024-bit RSA and 160-bit Elliptic Curve Cryptography|http://eprint.iacr.org/2009/389]]
* Qian Wang and Cong Wang and Jin Li and Kui Ren and Wenjing Lou: [[Enabling Public Verifiability and Data Dynamics for Storage Security|http://eprint.iacr.org/2009/281]]
* Ueli Maurer and Stefano Tessaro: [[Computational Indistinguishability Amplification: Tight Product Theorems for System Composition|http://eprint.iacr.org/2009/396]]
* Francesco Davì and Stefan Dziembowski: [[Leakage-Resilient Storage|http://eprint.iacr.org/2009/399]]
* ~Jean-Sébastien Coron, Thomas Icart: [[A Random Oracle into Elliptic Curves|http://eprint.iacr.org/2009/340]]
* Thomas Icart: [[How to Hash into Elliptic Curves|http://eprint.iacr.org/2009/226]]
* Fabian Kuhn, René Struik: [[Random Walks Revisited: Extensions of Pollard’s Rho Algorithm for Computing Multiple Discrete Logarithms|../../named/URI:CHK:wu2kjacvelvgzula6ojafumrru:ynj4xnlwfp35obni7ydxjcc6winzdyc4tlyj5e4ooh5prn5oh2ca:3:10:195703/Random_Walks_Revisited.pdf]]
* Yvonne Hitchcock, Paul Montague, Gary Carter, Ed Dawson: [[The efficiency of solving multiple discrete logarithm problems and the implications for the security of fixed elliptic curves|/file/URI:CHK:qxugzb4esi6b6clupozci7x6hu:jscs6fjjtm5jmqh3rchurubi5qmz7pxc2bf54ixt7vn64zmy5abq:3:10:305839/@@named=/multipledlog.pdf]]
* Phil Rogaway: [[Formalizing Human Ignorance|http://www.cs.ucdavis.edu/~rogaway/papers/ignorance.html]] ([[on citeseerx|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.65.291]], [[on eprint.iacr.org|http://eprint.iacr.org/2006/281]])
* Whitfield Diffie, Martin Hellman: [[New Directions in Cryptography|http://groups.csail.mit.edu/cis/crypto/classes/6.857/papers/diffie-hellman.pdf]], 1976
* Kevin D. Bowers, Ari Juels, Alina Oprea: [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]]
* Gaetan Leurent, Phong Q. Nguyen: [[How Risky is the Random-Oracle Model?|http://eprint.iacr.org/2008/441]]
* Michal Rjaško: [[Properties of Cryptographic Hash Functions|http://eprint.iacr.org/2008/527]]
* Abhishek Parakh, Subhash Kak: [[A Recursive Threshold Visual Cryptography Scheme|http://eprint.iacr.org/2008/535]]; (A good feature of papers about visual cryptography is that they always come with pictures!)
* Yehuda Lindell: [[Adaptively Secure Two-Party Computation with Erasures|http://eprint.iacr.org/2009/031]]

[[programming languages]], tools, protocols, patterns, engineering, decentralization:
* [[The Agile Skills Project|http://sites.google.com/site/agileskillsprojectwiki/]] (via Jack Lloyd)
* Jason Yip: [[It's Not Just Standing Up: Patterns of Daily Stand-up Meetings|http://martinfowler.com/articles/itsNotJustStandingUp.html]]
* Mike Cohn: [[An Overview of Scrum for Agile Software Development|http://www.mountaingoatsoftware.com/scrum/overview]]
* Agile Manifesto Authors: [[Principles behind the Agile Manifesto|http://agilemanifesto.org/principles.html]]
* Abby Fichter: [[Are We Agile Yet?|http://www.thehackerchickblog.com/2010/02/are-we-agile-yet.html]]
* Sean Eron Anderson: [[Bit Twiddling Hacks|http://graphics.stanford.edu/~seander/bithacks.html]] (mentioned by Brad Neuberg)
* Rohit Khare, Richard N. Taylor: [[Extending the REpresentational State Transfer (REST) Architectural Style for Decentralized Systems|http://www.ics.uci.edu/~rohit/ARRESTED-ICSE.pdf]] ([[indexed on CiteSeerX|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.8033]]) ([[copy hosted on Tahoe-LAFS decentralized secure storage grid|http://../../file/URI%3ACHK%3Alwd3qsbxzrnqkaabdrzunshuqe%3A6gipjscuop2r3hrcb47svxc3xehj6c2ptcaoplhflv7smn3vcx4q%3A3%3A10%3A4550069/@@named=/ARRESTED-ICSE.pdf]]) (as recommended [[by James Urquhart|http://twitter.com/jamesurquhart/status/5738731163]] and [[by Mike Kelly|http://twitter.com/mike__kelly/status/5689070113]])
* J. Lacan, V. Roca, J. Peltotalo, and S. Peltotalo: [[RFC 5510: Reed-Solomon Forward Error Correction (FEC) Schemes|http://tools.ietf.org/html/rfc5510]]
* Peter Seibel: [[Coders at Work|http://www.codersatwork.com]]
* Gilles Dubochet: [[Computer Code as a Medium for Human Communication: Are Programming Languages Improving?|http://infoscience.epfl.ch/record/138586/files/dubochet2009coco.pdf]] (as recommended by Bryan O'Sullivan); At last!  A real psychology paper studying programming languages!  Such research is about thirty years overdue.
* Dhananjay Nene: [[CRUD is not only good for, but is the only consistent way to build REST over HTTP|http://blog.dhananjaynene.com/2009/08/crud-is-not-only-good-for-but-is-the-only-consistent-way-to-build-rest-over-http/]]
* Mauricio Fernández: [[Introducing extprot: extensible binary protocols for cross-language communication and long-term serialization|http://eigenclass.org/R2/writings/extprot-extensible-protocols-intro]] (recommended by Paul Snively)
* AMQP standards writers: [[Advanced Message Queueing Protocol spec, v0-10|http://jira.amqp.org/confluence/download/attachments/720900/amqp.0-10.pdf?version=1]]
* Dan Reverri: [[Writing an AMQP connector with Python, Twisted, Trial and txAMQP|http://app.arat.us/blog/?p=112]]
* w3schools.com: [[XML Schema Tutorial|http://www.w3schools.com/Schema/default.asp]]
* Nate Foster, Benjamin C. Pierce, Michael Greenberg, et al.: [[Harmony/Boomerang|http://www.seas.upenn.edu/~harmony]], A bidirectional programming language for ad-hoc, textual data

[[computer performance]]:
* Sachin Katti, Hariharan Rahul, Wenjun Hu, Dina Katabi, Muriel Medard, Jon Crowcroft: [[XORs in The Air: Practical Wireless Network Coding|http://127.0.0.1:8123/file/URI%3ACHK%3A5v6bgzvcdovykxluylk5omyp5a%3Anzf5v2wmv2ox3ajb5gvd7f7photy43eb2zp724ls5qz6oaphqhhq%3A3%3A10%3A452699/@@named=/XORs_in_the_air-practical_wireless_network_coding.pdf]]
* Daniel J. Bernstein, Adam Langley: [[Crit-bit Trees|http://www.imperialviolet.org/binary/critbit.pdf]]
* Bryan Cantrill, Jeff Bonwick: [[Real-World CONCURRENCY|http://mags.acm.org/queue/200809/?folio=16&CFID=19616687&CFTOKEN=90220359]] //(thanks to Kragen Sitaker for bringing it to my attention)//
* IEEE Spectrum: [[A Fairer, Faster Internet Protocol|http://spectrum.ieee.org/dec08/7027]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/04]])
* Greenan, Long, Miller, Schwarz, Wylie: [[A Spin-Up Saved is Energy Earned: Achieving Power-Efficient, Erasure-Coded Storage|http://www.usenix.org/events/hotdep08/tech/full_papers/greenan/greenan_html]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
* Andreas Merkel, Frank Bellosa: [[Memory-aware Scheduling for Energy Efficiency on Multicore Processors|http://www.usenix.org/events/hotpower08/tech/full_papers/merkel/merkel_html]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])

[[science]], [[health and anthropology]], [[politics and economics]], [[hacking community]] (and culture and history and linguistics and psychology) (see also [[studies on low-carb diet]]):
* Tom Standage: //The Victorian Internet//
* John M. Freeman, Eric H. Kossoff and Adam L. Hartman: [[The Ketogenic Diet: One Decade Later|../URI:CHK:45fi2wtqfpxbhyqwcczbcbauee:g4mqh6zcmhf4pt4hl2qbmnp7b4s3c2mf2nsetjmplh36prfjqzxq:3:10:335331]]
* Vilhalmur Stefannson: //[[The Fat of the Land|../URI%3ADIR2-RO%3Audi3loyke42azk6wkhfuhhicde%3A6y6dvdbrhynlpmpsx2trms367ixy6q72hxfnjcnfeuhh56jtakna/Stefansson_1956-The_Fat_of_the_Land.pdf]]// (book, 1956) (<-- PDF version hosted on ~Tahoe-LAFS)
* Stefania Piconi, Daria Trabattoni, Cristina Luraghi, Edoardo Perilli, Manuela Borelli, Michela Pacei, Giuliano Rizzardini, Antonella Lattuada, Dorothy H. Bray, Mariella Catalano, Antonella Sparaco and Mario Clerici: [[Treatment of periodontal disease results in improvements in endothelial dysfunction and reduction of the carotid intima-media thickness|http://www.fasebj.org/cgi/content/abstract/23/4/1196]] ([[as recommended|http://twitter.com/DrEades/status/10185869017]] by Dr. Mike Eades)
* John Jacob Cannell MD: [[Vitamin D, Vitamin A, and Cancer|http://www.vitamindcouncil.org/newsletter/vitamin-d-vitamin-a-and-cancer.shtml]] (as recommended [[by Dr Mike Eades|http://twitter.com/DrEades/status/9825372497]] and [[by Amber|http://twitter.com/ambimorph/status/9831705925]])
* Joshua Wolf Shenk: [[What Makes Us Happy?|http://www.theatlantic.com/magazine/archive/2009/06/what-makes-us-happy/7439/]] (as [[recommended|http://twitter.com/bos31337/status/2162906886]] by [[Bryan O'Sullivan|http://serpentine.com]])
* [[Steve Solomon|http://www.soilandhealth.org/05steve%27sfolder/05aboutmeindex.html]]: [[Longevity and Nutrition Library|http://www.soilandhealth.org/02/0203CAT/0203longevitylibcat.html]]; Wow! It is a treasure trove of out-of-print books about human health and longevity, curated by some kind of radical Tasmanian farmer philosopher!
* Jonathan K. Pritchard: Haplotype Analysis Human Evolution, circa 2002-2007, University of Chicago
* [[Extra Terrestrial Jaynes|http://bayes.wustl.edu]]: [[Probability Theory: The Logic of Science|http://books.google.com/books?id=tTN4HuUNXjgC&printsec=frontcover&dq=e.t.+jaynes&source=bl&ots=H3OwmvHyW3&sig=HNoyD74_wgvJbUSwsD-d2wb51IQ&hl=en&ei=ISB2S_X_HY6otgPHuKS8Cw&sa=X&oi=book_result&ct=result&resnum=7&ved=0CCgQ6AEwBg#v=onepage&q=e.t.%20jaynes&f=false]]
* The Memory Bank: [[Building economic democracy with community currencies|http://thememorybank.co.uk/2010/01/16/building-economic-democracy-with-community-currencies/]] (blog post) (recommended by [["openmoney" on twitter|http://twitter.com/openmoney]])
* Oscar ~Fernandez-Capetillo: [[Intrauterine programming of ageing|http://www.nature.com/embor/journal/v11/n1/full/embor2009262.html]]
* John Timmer, writing in arstechnica.com: [[Anti-"publication bias" efforts not panning out for science|http://arstechnica.com/science/news/2009/09/for-clinical-trials-design-and-results-dont-always-match.ars]]
* Natalie Angier, writing in The New York Times: [[Skipping Spouse to Spouse Isn’t Just a Man’s Game|../../file/URI%3ACHK%3Ala6cyuvjlmk7jf4e42ywvakfxe%3Acdol7iz3bytmglqtympwrre6jyrbgqowyzj5piogl56jzvqrsxoq%3A3%3A10%3A21737/@@named=/01angi.html]]
* Michael Nielsen: [[Doing science in the open|http://physicsworld.com/cws/article/print/38904]]
* NIH "Research Matters" newsletter: [[Low-Fat Diet May Cut Ovarian Cancer Risk|http://www.nih.gov/news/research_matters/october2007/10222007diet.htm]]
* Emily Yoffe: [[Seeking: How the brain hard-wires us to love Google, Twitter, and texting. And why that's dangerous.|http://slate.com/toolbar.aspx?action=print&id=2224932]]
* The Economist: [[The state of economics: The other-worldly philosophers|http://www.economist.com/displaystory.cfm?story_id=14030288]] (note: fragile URL; I should copy the document onto a ~Tahoe-LAFS grid and link to the copy.)
* Dave Neary: [[Open Source Community Building – Barriers to Entry|http://dneary.free.fr/articles/Community_barriers_to_entry_checklist.pdf]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2009/07/24]])
*  [[Kevin Carson|http://mutualist.blogspot.com]]: //[[Organization Theory|http://www.abebooks.com/Organization-Theory-Kevin-A-Carson-BookSurge/1252202295/bd]]//
* Ursula K. ~LeGuin: //[[The Dispossessed|http://www.sfsite.com/01b/dis73.htm]]// (as recommended by Jake Appelbaum) (I've already read most of this book at least once.)
* Lera Boroditsky: [[How does our language shape the way we think?|http://edge.org/3rd_culture/boroditsky09/boroditsky09_index.html]] as recommended  by [[Kris Nuttycombe|http://nuttycombe.gaia.com/blog]]
* John Markoff: //[[What The Dormouse Said|http://www.amazon.com/dp/0670033820]]// (the dead tree copy I have was a gift from Sunah) 
* Stanford Encyclopedia of Philosophy: [[Bayes' Theorem|http://plato.stanford.edu/entries/bayes-theorem]]
* Leamer: [[on housing and the business cycle, posted in 1997|http://www.anderson.ucla.edu/Documents/areas/adm/media/leamer_housing_business_cycle.pdf]] (as [[recommended by Russ Roberts|http://cafehayek.typepad.com/hayek/2008/11/housing-and-the.html]])

See also [[things I have read]].

I've recently noticed people reading Bruce Schneier's classic //[[Applied Cryptography|http://www.schneier.com/book-applied.html]]// (784 pages, first edition published in 1995). It is a fun book—stuffed with descriptions of a variety of interesting protocols, written in an easy style, and of course it is historically important since it taught a generation of engineers that crypto was something that they could learn how to use and rely on. For these reasons alone, //Applied Cryptography// is still worth reading for fun and intellectual stimulation.
But //Applied Cryptography// is very dated now, and even when it was fresh it was imprecise and sometimes wrong. I get the impression that Schneier wrote //Applied Cryptography// first and //then// had a long, successful career in cryptography and security consulting and became an expert. (I mean no disrespect—Schneier is an excellent writer and a good cryptographer and security engineer. Today he certainly knows more than I do about most of these topics.)
If you are an engineer looking to use and rely on crypto today, you should start with //[[Cryptography Engineering|http://www.schneier.com/book-ce.html]]// by Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno (384 pages). This book is focused, thorough, and was written by true experts with extensive experience. (Disclosure: I contributed review and suggestions to the most recent edition and I am a personal friend of both Niels and Yoshi.)
You should also keep a copy of //[[Handbook of Applied Cryptography|http://www.cacr.math.uwaterloo.ca/hac/]]// (816 pages) available. It is of the same vintage as //Applied Cryptography//, but since it is more precise it has weathered better. It is good for looking things up, encyclopedia-like, rather than for reading right through. The whole thing is available for free in PDF form from the authors' site, but I personally appreciate having a dead tree version that I can lay open on a table while working.
Finally, if you want to spend a lot of time on this sort of thing and become a pro, then you ought to read Ross Anderson's //[[Security Engineering|http://www.cl.cam.ac.uk/~rja14/book.html]]// (1080 pages). Anderson was a pioneer of the idea that cryptography has to be seen as only one component of the security system which includes software engineering, usability, psychology, social and economic organization, military and criminal organization, etc. (Today this is obvious but we might not understand this as well as we do if it hadn't been for Ross Anderson's research and publications. It is often the fate of good research to become obvious.)
//Security Engineering// is Anderson's ambitious attempt to distill a distinguished career's worth of research and experience. It is broader in scope than the three books mentioned above and richly detailed. It can be difficult to slog through even though Anderson is a good writer who does not shy away from pronouncing succinct judgments on his subjects.
You occasionally find yourself wondering “If I'm supposed to be writing a secure web app in ~JavaScript, why am I reading the details of how different kinds of physical locks can be picked, or how banks manage employees, or how nuclear missile launches are controlled?” The promise is that lessons learned from security engineering in various domains can teach you how to work better on your particular domain. Also, the holes that get exploited in practice often lie at the borders between disciplines, where the good guys don't understand one another's work and so their systems don't match up. It isn't for the faint of heart, but if you are serious about the subject then //Security Engineering// is a treasure trove.

For my morning's crypto education I learned a little bit more about [[the LANE hash function|http://www.cosic.esat.kuleuven.be/publications/article-1182.pdf]].  The message expansion part of LANE can be seen as an error-detection code.  This is interesting to me, and it reminds me that the notions of collision-resistance and (especially) pre-image resistance ought, perhaps, to be clarified to include targets.

Informally, you care about the notion of pre-image resistance because you want to be able to give someone //H(x)// without revealing to them information about //x//.  But cryptographers seem to say that an attacker wins if he is able to learn the pre-image //y// about ''any'' image //H(y)// (excepting those that he already knew before he started -- i.e. excluding the trivial move of calculating //H(y)// from //y// and then saying you know the pre-image of //H(y)//).  This "general pre-image-resistance" notion isn't necessarily a stronger notion than the more natural "targetted-image pre-image-resistance" notion, although it might seem like it is at first.

That is: there could be some hash function for which an attacker can't come up with a pre-image for some arbitrarily chosen image, but which he //can// come up with a pre-image for your image.  :-)  (Because, the images you care about may be a special, non-random, non-uniformly distributed subset of all images, and because by telling the attacker an image, you are giving him information.)

It seems unlikely that any real hash function would be like this -- but the message-expansion step in LANE makes me wonder.

It seems to me, intuitively, that using an error-detecting code for the message-expansion step of LANE strengthens LANE against the attacker who wants to find any pre-image that he can, but weakens it against the attacker who wants to find //your pre-image// of some image that you told him.  This is because the message expansion code is adding redundancy but not information, and it is therefore //reducing the fraction of internal states which have pre-images at all//.  So if you are a cryptographer trying to make a name for yourself by coming up with a pre-image attack on LANE, and you've laboriously (with the help of complex algorithms and high-powered computers) worked out some prospective images which are reached from some internal states, then you might be disappointed to find that none of those internal states are valid according to the error-detection code, so there are no pre-images that lead to those internal states.  On the other hand, if you are an attacker who has been given an actual image by someone who hopes that you won't learn their pre-image, then the error-detection code ought to help you, by letting you cheaply determine whether a given internal state is right.  The error-detection property of the message expansion can't actually //hinder// your attempt to find the user's original pre-image, because by definition that pre-image is a valid input.  It could hinder your attempt to find a //different// pre-image that maps to the user's image.

A related puzzler is this: suppose you have an important secret //y//, and you compute //h = H(y)// and tell //h// to someone else.  Now suppose that other person is able to find a different pre-image //y′//, unrelated to your //y//, such that //h = H(y′)//.  Good for him!  He has violated the pre-image resistance property of //H//.  But he hasn't violated your confidentiality.  Hm.  Isn't your confidentiality the actual motivation for caring about pre-image resistance in the first place?

I know that cryptographers have formalized a notion of "target collision resistance", a.k.a. "universal one-way hash functions".  Have they done the same for "target pre-image resistance"?  Oh, maybe [[Herding Hash Functions and the Nostradamus Attack|http://www.cs.washington.edu/homes/yoshi/papers/EC06/herding.pdf]] by Kelsey and Kohno is relevant.  (See also [[the amusing demonstration|http://www.win.tue.nl/hashclash/Nostradamus]] in which some Dutch mathematicians used a ~PlayStation 3 to correctly predict the outcome of the 2008 USA Presidential Election.)

Thanks to Ghoti from IRC for challenging me to think more carefully about LANE.  Thanks to Ruptor from IRC for teaching me cryptography in the mornings.

Post script: ooh, this paper looks perfect for me: [[A Critical Look at Cryptographic Hash Function Literature|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]] by Scott Contini, Ron Steinfeld, Josef Pieprzyk, and Krystian Matusiewicz.  However, reading it will have to wait until I've finished the Repairer for ~Tahoe-LAFS!

(Also in this mornings crypto exploration: [[performance of the crypto hardware in the Ultrasparc T2|http://blogs.sun.com/sprack/entry/ultrasparc_t2_crypto_performance]].)

//My mom has pointed out that my blog is chock full of acronymy goodness and she can get only a vague idea of what I'm going on about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.//

Wow!  The [[list of accepted papers for Crypto 2009|http://www.iacr.org/conferences/crypto2009/accepted-papers.txt]] includes one named: //Alex Biryukov, Dmitry Khovratovich, Ivica Nikolic: [[Distinguisher and Related-Key Attack on the Full AES-256|http://google.com/search?q="Distinguisher and Related-Key Attack on the Full AES-256"]]//.  Does anybody have a pre-print I can see!?  Update: here are [[slides|http://eurocrypt2009rump.cr.yp.to/410b0c56029d2fa1d686823e3a059af8.pdf]], thanks to Ghoti.

"Distinguisher and ~Related-Key Attack" means that it probably won't threaten anyone's actual confidentiality right now, but it shows weaknesses in the algorithm (~AES-256) which could in some rare circumstances, actually threaten confidentiality.  If this means what I think it does, then this will spark cryptographers to question the reliability of ~AES-256.  That would be big news within the world of cryptography, as ~AES-256 is the most widely used, widely trusted cipher.  It is the only cipher that the U.S. Government is allowed to use for Top Secret confidential documents and transmissions, for example.

~AES-256 is widely believed to be unbreakable, so if there really are distinguisher and related-key attacks on it then this means there must be some new technique of cryptanalysis at work that the world's cryptographers hadn't understood until now.

//''Mom:'' scientists are gradually figuring out how to make algorithms which can guarantee the confidentiality and integrity of data or of conversations.  Unfortunately, the only confidentiality and integrity currently protected by these algorithms is that of major governments and giant corporations.  Applying these techniques to protect the data and conversations of actual people such as yourself remains an unsolved problem, and there are sadly few people trying to solve it.//

{{{
From: zooko
Date: December 22, 2008 15:02:40 PM MST
To: Multiple recipients of list
Subject: will SHA-3 replace the current standard secure hash algorithm -- MD5?
}}}

Folks:

Below, I re-post [[a letter|SHA-256 is too slow]] that I wrote to this list last February.  I think that this letter, which some of you may not have seen, casts light on why we have different assumptions about valid security/performance trade-offs.  It is because secure hashes are used today for more than just their original purpose.

Since I wrote this letter, I did some more snooping around, and I learned that the situation is even more extreme than I thought -- for some areas of endeavour, ~MD5 is actually the standard secure hash algorithm in 2008.

I chatted with a couple of friends who are information security consultants -- they get paid big bucks by household-name corporations to audit source code and systems for security flaws.  I asked them what kinds of secure hash functions they see used in the wild.  They answered that ~MD5 was the most common, occasionally ~SHA-1, in large part because it is a default value on the Java Cryptography Extensions, and they have never seen any other secure hash functions in client systems.

I chatted with a friend who works at the Internet Archive -- all files stored at the Internet Archive are identified by their ~MD5 hashes.

I noticed that there was a new release of the Haskell compiler GHC.  One of the new features is that it uses ~MD5 to identify code modules.

I learned more about the "computer forensics" field.  ~MD5 appears to be the standard mechanism to identify files in that field.  I read discussion forums in which computer forensics practitioners asked each other whether the cryptographic attacks on ~MD5 that they had heard about meant that they needed to change their practice.  The consensus seemed to be that they could continue using ~MD5 for now.

Finally, I was intrigued to see that NIST, of all organizations, uses and recommends the use of ~MD5 (in addition to ~SHA-1), as part of its "National Software Reference Library", which supports digital forensics.  This document explaining why NIST believes that this is safe is fascinating:

http://www.nsrl.nist.gov/Documents/analysis/draft-060530.pdf


The wide gap between the performance needs of using a secure hash function for public key cryptography versus using it for bulk data identification and integrity checking (which is what I use it for at my day job), make me wonder if ~SHA-3 should include variants or officially recommended tuning parameters so that people identifying large files can use a ~SHA-3 which is at least as fast as ~MD5 or Tiger, while people who are signing thousand-year documents can use a ~SHA-3 which is more expensive but safer.  (By the way, I tend to think that HMAC shouldn't be weighted heavily as a use case for ~SHA-3 simply because people should stop using HMAC and start using ~Carter-Wegman ~MACs instead such as Poly1305 or VMAC.)


Regards,

Zooko O'Whielacronx