excluded from Debian because of no-source-code-included files

[zooko]

According to the Tails developers (​https://labs.riseup.net/code/issues/6227#note-23) Tahoe-LAFS v1.10 is about or already has been excluded from the upcoming new Debian release because of the presence of some code with no accompanying source: ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735940

[zooko]

For the record, we wrote down our thoughts about this packaging issue when we first introduced those files:

​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1200#comment:8

Would it satisfy the Debian policy if the files were stored in source form instead of minified form?

[zooko]

Let's start by writing a patch that removes the minified js files and adds the source js files. (Because that's easy, and it might help.)

The other possible things to do here I don't necessarily fully understand:

​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735940#40

[daira]

I suggest the following:

1. Rename src/allmydata/web/static/d3-2.4.6.min.js to d3.min.js and src/allmydata/web/static/jquery-1.6.1.min.js to jquery.min.js.
2. Replace d3-2.4.6.time.min.js with ​https://raw.github.com/jiahuang/d3-timeline/master/src/d3-timeline.js (not minified).
3. Upgrade d3 to version 3.3.13 and jquery to 1.7.2. (This is to ensure that we're compatible with those versions.)
4. Make the Debian package depend on libjs-d3 >= 3.3.13 and libjs-jquery >= 1.7.2.
5. Patch the Debian package to remove src/allmydata/web/static/d3.min.js and src/allmydata/web/static/jquery.min.js.
6. Patch the Debian installation script to symlink d3.min.js to /usr/share/javascript/d3/d3.min.js and jquery.min.js to /usr/share/javascript/jquery/jquery.min.js.

[daira]

Note that steps 1-3 inclusive also fix #2122.

[zooko]

Shouldn't we take the ".min" out of the filename if the content is not minified?

[anarcat]

i would say yes: not-minified code shouldn't have the '.min' infix.

as for the steps, they make sense! however, i wonder what step 5 means, if those files are removed upstream already...

[daira]

Replying to zooko:

Shouldn't we take the ".min" out of the filename if the content is not minified?

The files provided by the Debian libjs-d3 and libjs-query packages are minified.

[daira]

Replying to anarcat:

i would say yes: not-minified code shouldn't have the '.min' infix.

Indeed not (and d3-timeline.js won't).

as for the steps, they make sense! however, i wonder what step 5 means, if those files are removed upstream already...

They wouldn't be removed upstream.

[daira]

It appears (from smoke-testing) that d3-timeline.js is not the correct replacement for d3-2.4.6.time.min.js. I don't know where to find pre-minified source for the latter. (It is not part of d3 3.3.13.)

[warner]

Right, it's from d3 2.4.6 (they must have merged it into the core codebase between 2.4.6 and the current 3.4.4).

I copied it from ​https://github.com/mbostock/d3.git (the official d3.js repo), tag "v2.4.6", the file named d3.time.min.js . So d3.time.js in the same repo+tag is the unminified source.

I'm cool with switching to non-minified versions. I'd suggest going all the way to the current release (v3.4.4) instead of stopping at 3.3.13 unless.. ohh, debian currently has 3.3.13? Or some specific debian target version has 3.3.13?

Anyways, as long as we switch to a sufficiently-new version of d3, we can get rid of the separate d3.time file.

[daira]

Yes, the problem is that the libjs-d3 package in Debian provides only provides version 3.3.13 of d3 (see ​https://packages.debian.org/jessie/libjs-d3), and that is prior to d3.time having been merged in. So that package does not contain d3.time at all.

Therefore, although we might decide to upgrade to more recent d3 in upstream Tahoe-LAFS anyway, that wouldn't solve this ticket.

[daira]

Replying to daira:

Therefore, although we might decide to upgrade to more recent d3 in upstream Tahoe-LAFS anyway, that wouldn't solve this ticket.

Or more precisely, it wouldn't solve this ticket if it is a requirement to use a Debian-packaged version of d3. If Debian is happy with us including unminified source, there's no problem. I will ask on the ​Debian ticket.

[warner]

Huh, d3-v3.3.13 *does* have d3.time merged into the main d3.js file (​https://github.com/mbostock/d3/blob/v3.3.13/d3.js#L8626). However the debian packaging of it does not (/usr/share/javascript/d3/d3.js looks like it's been partially minified, but not as much as the d3.min.js sitting next to it). I'm still trying to figure out how the debian package gets built: perhaps there's some bug therein which is failing to include everything that it ought to. In that case, the easiest path may be to get debian to fix the packaging.

[daira]

Replying to warner:

Huh, d3-v3.3.13 *does* have d3.time merged into the main d3.js file (​https://github.com/mbostock/d3/blob/v3.3.13/d3.js#L8626).

Oh. Well then why does the timeline fail to render correctly with the d3.min.js from ​here? (It renders just as a bunch of overlapping text in the top-left corner of the view.)

[daira]

It does render correctly when d3-2.4.6.time.min.js is loaded after d3.min.js version 3.3.13 (which is the main reason why I thought d3.time wasn't included in the latter).

[warner]

Yeah, the way d3 "plugins" work is to add properties to a pre-existing d3 object (basically monkeypatching), so order matters.

I assume this would fail if patched to use a system-provided (debian) libjs-d3 copy of d3.js, but I haven't actually tried it to see. It's possible that the debian copy *does* include d3.time, but the way it's built defies my grep abilities to see it.

[daira]

Putting this in the 1.11 milestone is optimistic because it may depend on cooperation and information from Debian developers and/or other package maintainers (libjs-d3 at least).

[daira]

Zooko found the following comment in trac's spam bucket:

First, an aside: I am Ramakrishnan, a Debian developer with some interest in crypto and Functional Programming and file systems, which brought me to Tahoe-LAFS. I hope to look at this from a Debian perspective and work with you folks and get it resolved. As a first step, I filed a wishlist bug against libjs-d3 to upgrade it to the latest upstream release, which was promptly done by the libjs-d3 debian maintainer. I wish that we base further work on top of it, as it is the latest D3 release. If the debian package of D3 has anything lacking, let us fix that as well on the way. I hope to start looking at it from tomorrow.

[daira]

Review needed for ​https://github.com/tahoe-lafs/tahoe-lafs/pull/89.

[daira]

$ diff --ignore-space-change ~/d3-3.4.6/d3.js src/allmydata/web/static/d3.js >d3.diff
$ wc --lines d3.diff
15291 d3.diff

Sigh. I don't understand this -- how can there be 15291 lines of differences between upstream d3, and the Debian package with the same version number?

[warner]

The d3 git repo includes three things: individual source files (275 .js files in src/), a merged "d3.js" file (330K, about 9k lines), and a merged+minimized "d3.min.js" (150K, 4 really long lines). Each time the author makes a change (or accepts somebody else's change), they also rebuild d3.js and d3.min.js and check in all three things at the same time.

I'm guessing that debian rebuilds their own merged d3.js from src/ , and manage to do it differently than the upstream author does. It'd be nice to fix that, to remove the difference that you see.

[daira]

But why?

Why would anyone not understand the importance of being able to trace differences from upstream? This is not comprehensible to me.

[zooko]

Replying to daira:

But why?

Why would anyone not understand the importance of being able to trace differences from upstream? This is not comprehensible to me.

So, as far as I understand we can avoid the need to understand such things by just not using minified, compressed, compiled, or bundled, versions of anything, unless in the future we discover a compelling need to do so. This also lets us avoid Debian's uncertainty about what source code we're shipping, in addition to our uncertainty about what source code they're shipping.

[vu3rdd]

Perhaps one very simple to use and quickest fix in the Debian package would be to use the CDN:

<​https://cdnjs.cloudflare.com/ajax/libs/d3/2.7.4/d3.time.min.js>

and remove the ones in src/allmydata/web/static altogether. This way, we can:

• live without any doubt on the js file included with the OS packages.
• stop shipping a particular version of the JS file with the Tahoe-LAFS.
• circumvent silly bugs like this one, which is holding off an otherwise great software from being distributed with a distro like Debian.

I can't find the version 2.4.6 that is used by the "download-status-timeline.xhtml". JQuery versions are here:

<​https://cdnjs.com/libraries/jquery/>

If this solution is acceptable, would any of you, authors of Tahoe-LAFS verify a version of d3 that works fine? I am asking you that instead of doing it myself because I am unfamiliar with Tahoe-LAFS in general and don't even have it setup on my end (which I am going to do this week).

After that I can easily incorporate the patch into the 1.10.0 debian package and upload it.

Thanks Ramakrishnan

edit: fixed a typo.

[daira]

Using the CDN isn't acceptable for privacy reasons. Tahoe-LAFS shouldn't be making any requests to third parties (i.e. not servers that it has found via the configured introducer) at run-time.

[tsygrl]

Tahoe-lafs should run with full functionality detached from the main internet.

[vu3rdd]

Daira and I had a chat on the IRC today. I tried a few things today, here is the summary:

• The d3.js shipped with Debian's libjs-d3 package is different from the upstream d3.js.
• If I download the source and build it with 'npm install' and then 'make' (along with the Debian's nodejs package), it produces exactly the same js file as the one shipped with the upstream.
• Upon further inspection, looks like the uglify.js (I don't know what it does!) shipped with Debian and used by the libjs-d3 is quite old. D3 upstream needs 2.4.0 where as Debian has 1.3.4. Perhaps this needs to be fixed. I will raise a debian bug regarding this.
• I tried changing the dev dependency of the upstream sources of d3 to match that of the debian's dependency and build it but it didn't build for me with some weird error, which I didn't pursue further.

I wonder what can be done at this point. If Debian's current version of D3 and JQuery libraries work fine, then I could quickly change/patch the XHTML source to use those files in Debian (/usr/share/javascript/*), add dependencies on those libraries and do an upload. Alternatively, I could add a patch that removes the d3.min.js and jquery.min.js and add the non-minified versions.

Comments?

[daira]

+1 on raising the Debian bug about uglify.js.

At this point I'm inclined to ship the upstream unminified versions of d3 and jquery, and ask on the Debian bug whether that is sufficient. For jquery there's a choice between 1.7.2 and the latest upstream 1.x release (1.11.0) which I'm agnostic about.

[vu3rdd]

Replying to daira:

+1 on raising the Debian bug about uglify.js.

Done: ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745687

[warner]

I'm +1 on shipping unminified versions. I'd lean towards updating to the latest upstream (I expect v2.1.0 should be fine, nothing in the timeline code I wrote uses anything fancy, and it should be pretty quick to test that the newer version didn't break anything).

[warner]

I'll work on this. Preliminary tests suggests that moving to current d3-3.4.6 causes zoom to break. Also I'm going to stick with jquery-1.x (1.11.0) because jquery-2.x dropped support for IE6/7/8, which I'm willing to do but maybe after the next release.

[warner]

I pushed a branch (using d3-2.4.6 and current upstream jquery-1.11.0): ​https://github.com/tahoe-lafs/tahoe-lafs/tree/2208-unminified-js

.. and then I noticed that daira had already pushed a branch with d3-3.4.6 and jquery-1.7.2 . The latter should work ok, but in my tests the newer d3 doesn't.

Daira: did your branch handle zoom/pan properly? Any thoughts about which branch to use?

[Brian Warner <warner@…>]

In b6c0297deea77cce2aa22458f0645635cb1aa50d/trunk:

Include unminified versions of d3+jquery. Addresses #2208.

This should hopefully satisfy the Debian requirement to include original
sources. The old minified files for d3 and jquery were 63k and 91k
respectively, while the new unminified files are 133k and 293k.

[Brian Warner <warner@…>]

In b6c0297deea77cce2aa22458f0645635cb1aa50d/trunk:

Include unminified versions of d3+jquery. Addresses #2208.

This should hopefully satisfy the Debian requirement to include original
sources. The old minified files for d3 and jquery were 63k and 91k
respectively, while the new unminified files are 133k and 293k.

[daira]

warner: my branch did indeed cause zoom to break, so I reviewed and merged yours.

[zooko]

Okay, I believe this fixes the original problem of "no-source-code-included files", because now we bundle a copy of the original source (identical to upstream) instead of a minified version. Closing this ticket.

[vu3rdd]

Uploaded into Debian unstable: ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735940#106