provide binary .egg's for pycryptopp for all supported platforms and versions of Python

[arch_o_median]

Some System Configuration Information

arc@showertimedepot:~/tahoe$ darcs --version
2.0.2 (release)
arc@showertimedepot:~/tahoe$ uname -a
Linux showertimedepot 2.6.18.8-x86_64-linode7 #1 SMP Fri Aug 14 17:14:08 UTC 2009 x86_64 GNU/Linux
arc@showertimedepot:~/tahoe$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 8.10
Release:        8.10
Codename:       intrepid

Downloads

Per ​these instructions I ran:

darcs get --lazy http://allmydata.org/source/tahoe/trunk-hashedformat tahoe

I've deviated from the prescribed protocol since the default install docs do not recommend acquiring the source from the darcs repo.
Perhaps differences between the tarball and the repo versions generate this bug.
To check this I also attempt to build from the tarball in a separate location.

I.E. I ran:

 wget http://allmydata.org/source/tahoe/releases/allmydata-tahoe-1.5.0.zip

Build From Repo

Per ​these instructions I ran:

~/tahoe$ python ./setup.py build &> from_repo_build_log.txt

in the "from-repo" directory.

Upon inspection of the output (attached) I see several error-types.

I guess these two to be not-fatal:

1. "darcs failed: Not a repository: ..." and
   
2. "cc1plus: warning:..." lines are not fatal.
   

Following those two error-types the next error is:

pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory

Difference Between First and Second Build Attempts in From-Tarball Version

In the "from-tar" directory I ran:

python ./setup.py build &> from_tar_build_log_initial.txt

(Actually I changed the log names to be more descriptive, after the fact.)

I do not notice obvious differences between repo and tar.

I do notice differences between initial and subsequent build-attempts.

(Perhaps I should be writing "setup-" vice "build-" attempts.)

arc@showertimedepot:~/tahoe/allmydata-tahoe-1.5.0$ grep zope from_tar_build_log_initial.txt | grep Python.h
src/zope/interface/_zope_interface_coptimizations.c:15:20: error: Python.h: No such file or directory
arc@showertimedepot:~/tahoe/allmydata-tahoe-1.5.0$ grep zope from_tar_build_log_second.txt | grep Python.h

(Log names changed after the fact for clarity.)

Intriguingly "zope" appears to be missing "Python.h" on the first build attempt, this is not reported on subsequent build attempts.
The missing Python.h in pycryptopp is reported in initial and subsequent build attempts:

arc@showertimedepot:~/tahoe/allmydata-tahoe-1.5.0$ grep pycryp from_tar_build_log_initial.txt | grep Python.h
pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory
arc@showertimedepot:~/tahoe/allmydata-tahoe-1.5.0$ grep pycryp from_tar_build_log_second.txt | grep Python.h
pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory

First Versus Second From Repo Setup

Is the difference between initial and subsequent setup bugs also
seen in the from repo version? To find out I tested a fresh from-repo setup attempt.

arc@showertimedepot:~/tahoe-darcs$ darcs get --lazy http://allmydata.org/source/tahoe/trunk-hashedformat tahoe
Finished getting.                           

arc@showertimedepot:~/tahoe-darcs/tahoe$ grep -iEe"zope|pycryp" | grep Python.h from_repo_*
from_repo_initial_setup.txt:src/zope/interface/_zope_interface_coptimizations.c:15:20: error: Python.h: No such file or directory
from_repo_initial_setup.txt:pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory
from_repo_second_setup.txt:pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory

So the difference between zope and pycryptopp initial vs.
second installs generalizes to installs from tar and from repo.

Bug Interpretation

It looks like Python.h of pycryptopp is not found so I guess that I lack some necessary dependency.

Calling All Dependencies

(Since my bug does not seem to vary between from-repo and from-tar, I stick to from repo from here on.)

Where're these elusive Python.h's? Sound like C headers to me....

Naive:

arc@showertimedepot:~/tahoe-darcs/tahoe$ find . -iname "*Python.h*"
arc@showertimedepot:~/tahoe-darcs/tahoe$ 

OK, not surprising since docs say all repo is python, and what's missing is C. So, my belief that I have a dependency problem grows stronger.

Following ​More Details, I learn that I can view "the canonical list of dependencies" in _auto_deps.py.

Of course, I cannot run "tahoe --version" yet so I skip that advice...

Aha! A "complete list of all dependencies..."

PYTHONPATH=${PYTHONPATH}:./support/lib/python2.5/site-packages/ python -c "import pkg_resources;print ', '.join([d.project_name+': '+d.version for d in set(pkg_resources.require('allmydata-tahoe'))])"

arc@showertimedepot:~/tahoe-darcs/tahoe$ PYTHONPATH=${PYTHONPATH}:./support/lib/python2.5/site-packages/ python -c "import pkg_resources;print ', '.join([d.project_name+': '+d.version for d in set(pkg_resources.require('allmydata-tahoe'))])"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arc/tahoe-darcs/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/arc/tahoe-darcs/tahoe/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pycryptopp>=0.5.15
arc@showertimedepot:~/tahoe-darcs/tahoe$ 

Hmmm.... next step... submit and wait... if no help then dissect above expression...

OK, I did try to dissect a tiny bit:

In ipython interpreter:

In [1]: import sys

In [2]: sys.path
Out[2]: 
['',
 '/usr/bin',
 '/usr/lib/python2.5',
 '/usr/lib/python2.5/plat-linux2',
 '/usr/lib/python2.5/lib-tk',
 '/usr/lib/python2.5/lib-dynload',
 '/usr/local/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages',
 '/var/lib/python-support/python2.5',
 '/var/lib/python-support/python2.5/IPython/Extensions',
 '/home/arc/.ipython']

In [3]: sys.path.append('./support/lib/python2.5/site-packages/')

In [4]: import pkg_resources
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/home/arc/tahoe-darcs/tahoe/<ipython console> in <module>()

ImportError: No module named pkg_resources

(And just to be sure I tried above with explicit path name from root to relevant site-packages.)

Which is:

arc@showertimedepot:~/tahoe-darcs/tahoe$ ls -la ./support/lib/python2.5/site-packages/
total 32
drwxr-xr-x 3 arc arc 4096 2009-09-14 22:32 .
drwxr-xr-x 3 arc arc 4096 2009-09-14 22:30 ..
-rw-r--r-- 1 arc arc   35 2009-09-14 22:32 allmydata-tahoe.egg-link
-rw-r--r-- 1 arc arc  297 2009-09-14 22:30 easy-install.pth
drwxr-xr-x 4 arc arc 4096 2009-09-14 22:32 setuptools-0.6c12dev.egg
-rw-r--r-- 1 arc arc   27 2009-09-14 22:32 setuptools.pth
-rw-r--r-- 1 arc arc 2327 2009-09-14 22:30 site.py
-rw-r--r-- 1 arc arc 1833 2009-09-14 22:30 site.pyc

[arch_o_median]

OK, solved.

For my system:

sudo apt-get install python-dev python-openssl
wget http://pypi.python.org/packages/source/p/pyutil/pyutil-1.4.1.tar.gz
...
sudo python setup.py install (in pyutil directory)

Unfortunately when I ran setup.py install without sudo, I received a complaint about test_suite needing to be a list...

Anywho, it looks like basic install's OK now. Maybe I should write up an amateur install doc, for ubuntu people?

[zooko]

Thank you for the bug report. This isn't fixed yet though, until either the process in the current install.html works, or the install.html is changed to describe a process that works.

I'm not sure exactly how to accomplish that, yet...

[arch_o_median]

I see two distinct "bugs".

1. We need a doc that catalogs how to get dependencies (in this case for old Ubuntu).
2. We need to know why sudo is necessary for installing pyutil, and/or change the docs to reflect this.

[zooko]

sudo? You had to use sudo? I'm sorry, I haven't read your bug report thoroughly enough yet. You definitely don't need to use sudo.

[zooko]

As to your issue "i.", the goal is to make it so that the list of dependencies in ​http://allmydata.org/source/tahoe/trunk/docs/install.html is really sufficient:

• Python >= 2.4.2 and < 3.0
• if on Windows, pywin32

(Indeed, the goal would be to automate the installation of pywin32 too, so that the list of dependencies would be just Python. See ​http://twistedmatrix.com/trac/ticket/3238 (patch: declare that twisted requires pywin32 if it is to offer process management or iocp reactor on Windows), #756 (if pywin32 has been manually installed, setuptools still doesn't detect it), #142 (pywin32 can't be installed with setuptools), ​http://bugs.python.org/setuptools/issue18 (can't install pywin32), ​http://sourceforge.net/tracker/?func=detail&atid=551954&aid=1799934&group_id=78018 (easy_install silently fails).)

Anyway, my point is that the problem you encountered is that pycryptopp couldn't be automatically installed, and there are two ways to improve the situation: either (a) make pycryptopp be automatically installed on your platform and/or (b) document pycryptopp as something that has to be manually installed on your platform.

The former would be accomplished by building a binary egg of pycryptopp for your platform. I intend to configure the pycryptopp buildslaves to build eggs and upload them to ​http://allmydata.org in the same way that they currently build .deb's and upload those. One problem with this is ​http://bugs.python.org/setuptools/issue78 (egg platform names don't reflect unicode variant (UCS2, UCS4)).

The latter would be accomplished by editing the InstallDetails wiki page to tell Ubuntu users to do what you did. (sudo apt-get install python-dev)

[zooko]

Assigning to arc.

arc: what shall we do? I think we should narrow this ticket to be about the lack of a binary egg for pycryptopp on Linux. What do you think? If you agree, please change the title of this ticket and set it to be assigned to "nobody" instead of yourself. Perhaps we could *also* open another ticket saying that the InstallDetails page should include information about binary eggs of pycryptopp. Or instead of spending time opening that ticket, just spend the same time editing the wiki.

[imhavoc]

This bug has surfaced again.

System info:

tahoe@cat:~/allmydata-tahoe-1.5.0$ uname -a Linux cat 2.6.24-24-generic #1 SMP Fri Sep 18 16:49:39 UTC 2009 i686 GNU/Linux

tahoe@cat:~/allmydata-tahoe-1.5.0$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 8.04.3 LTS Release: 8.04 Codename: hardy

tahoe@cat:~/allmydata-tahoe-1.5.0$ cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 79 model name : AMD Athlon(tm) 64 Processor 3500+ stepping : 2 cpu MHz : 2210.208 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow up pni cx16 lahf_lm svm extapic cr8_legacy ts fid vid ttp tm stc bogomips : 4420.41 clflush size : 64

I tried using the solutions listed in this ticket. No joy.

[imhavoc]

This bug has surfaced again.

System info:

tahoe@cat:~/allmydata-tahoe-1.5.0$ uname -a
Linux cat 2.6.24-24-generic #1 SMP Fri Sep 18 16:49:39 UTC 2009 i686 GNU/Linux

tahoe@cat:~/allmydata-tahoe-1.5.0$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 8.04.3 LTS
Release:        8.04
Codename:       hardy

tahoe@cat:~/allmydata-tahoe-1.5.0$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 79
model name      : AMD Athlon(tm) 64 Processor 3500+
stepping        : 2
cpu MHz         : 2210.208
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow up pni cx16 lahf_lm svm extapic cr8_legacy ts fid vid ttp tm stc
bogomips        : 4420.41
clflush size    : 64

I tried using the solutions listed in this ticket. Build still failed.

[zooko]

Okay I'm changing this ticket to say that we should take strategy (a) from comment:5 -- make binary .egg's of pycryptopp be installed for supported systems. arc's and imhavoc's systems are both amd64 with UCS4, and we have a pycryptopp buildbot that fits that description: ​http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu-karmic-yukyuk . So to close this ticket, configure that buildbot to generate binary eggs and upload them to the tahoe-deps repository. (Then check if the builds start working for arc and imhavoc.) Oh, but arc was using Python 2.5 and imhavoc was using Python 2.6. Actually I think arc has subsequently upgraded his Ubuntu to Karmic so he has 2.6 as well. At least for now I'm defining this ticket as "provide binary .egg's for pycryptopp for linux-amd64-py2.6-ucs4".

[imhavoc]

Replying to zooko:

[snip] Oh, but arc was using Python 2.5 and imhavoc was using Python 2.6. Actually I think arc has subsequently upgraded his Ubuntu to Karmic so he has 2.6 as well....

Actually, on this system, Python is at version 2.5.2. Sorry.

[zooko]

Okay, to close this ticket we need to generate binary .egg's for both Python 2.5 and Python 2.6 on Linux-amd64-ucs4.

[imhavoc]

It turns out that my /home is a separate partition.

That partition was being mounted as 'noexec.'

When I edited fstab to make the partition mount as 'exec' and rebooted, tahoe compiled fine on the system in question.

[zooko]

Okay, so basically it turn out that you did have a C compiler, once you allowed it to be executable. Wait, was the cc1plus executable on your home partition?

In any case this ticket is about making precompiled pycryptopp binaries available for people who don't have a compiler.

[imhavoc]

I may have needed to install build-essential also.... I think I did. So, yes, I did have a C compiler available. So my problem may have been miss-files in this ticket.

[zooko]

So, the problem wasn't that your home directory was mounted no-exec, it was that you hadn't installed build-essential? That would explain the error message saying that cc1plus didn't work. Hey waitasecond, where is the error message? I thought I saw a paste from you showing cc1plus failing, but scanning back up through this ticket I don't see it.

What was the exact error message that you got?

[imhavoc]

I emailed it to you. I think that I never pasted it here.

[zooko]

Oh, okay here was the original error message that imhavoc mailed to me:

gcc: error trying to exec 'cc1plus': execvp: No such file or directory
error: Setup script exited with error: command 'gcc' failed with exit status 1

Yep, that looks like build-essential wasn't installed. Now, once this ticket is fixed, then it will just get the binary .egg of pycryptopp and it won't even try to execute gcc.

[imhavoc]

One of my grid participants has run into this bug again.

Ubuntu 8.10

AMD (32-bit CPU)

Looks like pycrypto is not available. There is also no python-pycryptopp available in apt.

We're trying to get a grid online, but installing tahoe-lafs seems to be, by far, the hardest part of getting a grid online.

[zooko]

The workaround for Ubuntu is to install build-essential so that a pycryptopp will be built automatically from source.

[imhavoc]

python -V

 2.5.2

gcc -v

Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.3.2-1ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu12) 

cat /proc/cpuinfo

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 44
model name	: Mobile AMD Sempron(tm) Processor 3000+
stepping	: 2
cpu MHz		: 800.000
cache size	: 128 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt 3dnowext 3dnow up pni lahf_lm
bogomips	: 1600.04
clflush size	: 64
power management: ts fid vid ttp tm stc

build fail output:

someUser@someUser-ubuntu:~/Desktop/allmydata-tahoe-1.5.0$ python setup.py build
running darcsver
setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone.
running develop
Not found: tahoe-deps
Not found: ../tahoe-deps
running egg_info
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
running build_ext
Processing setuptools-0.6c12dev.egg
creating /home/someUser/.local/share/Trash/files/allmydata-tahoe-1.5.0/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg
Extracting setuptools-0.6c12dev.egg to /home/someUser/.local/share/Trash/files/allmydata-tahoe-1.5.0/support/lib/python2.5/site-packages
Adding setuptools 0.6c12dev to easy-install.pth file
Installing easy_install script to support/bin
Installing easy_install-2.5 script to support/bin

Installed /home/someUser/.local/share/Trash/files/allmydata-tahoe-1.5.0/support/lib/python2.5/site-packages/setuptools-0.6c12dev.egg
Creating /home/someUser/.local/share/Trash/files/allmydata-tahoe-1.5.0/support/lib/python2.5/site-packages/allmydata-tahoe.egg-link (link to src)
allmydata-tahoe 1.5.0 is already the active version in easy-install.pth
Installing tahoe script to support/bin

Installed /home/someUser/.local/share/Trash/files/allmydata-tahoe-1.5.0/src
Processing dependencies for allmydata-tahoe==1.5.0
Searching for pycryptopp>=0.5.15
Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/
Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/
Best match: pycryptopp 0.5.17
Downloading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/pycryptopp-0.5.17.tar.gz
Processing pycryptopp-0.5.17.tar.gz
Running pycryptopp-0.5.17/setup.py -q bdist_egg --dist-dir /tmp/easy_install-pzTdsC/pycryptopp-0.5.17/egg-dist-tmp-7fK0dS
setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and pycryptopp/_version.py already exists, so leaving it alone.
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or directory
In file included from pycryptopp/_pycryptoppmodule.cpp:5:
pycryptopp/publickey/rsamodule.hpp:5: error: variable or field ‘init_rsa’ declared void
pycryptopp/publickey/rsamodule.hpp:5: error: ‘PyObject’ was not declared in this scope
pycryptopp/publickey/rsamodule.hpp:5: error: ‘module’ was not declared in this scope
pycryptopp/publickey/rsamodule.hpp:7: error: ‘PyMethodDef’ does not name a type
pycryptopp/publickey/rsamodule.hpp:9: error: expected initializer before ‘*’ token
pycryptopp/publickey/rsamodule.hpp:13: error: expected initializer before ‘*’ token
pycryptopp/publickey/rsamodule.hpp:17: error: expected initializer before ‘*’ token
In file included from pycryptopp/_pycryptoppmodule.cpp:6:
pycryptopp/hash/sha256module.hpp:5: error: variable or field ‘init_sha256’ declared void
pycryptopp/hash/sha256module.hpp:5: error: ‘PyObject’ was not declared in this scope
pycryptopp/hash/sha256module.hpp:5: error: ‘module’ was not declared in this scope
In file included from pycryptopp/_pycryptoppmodule.cpp:7:
pycryptopp/cipher/aesmodule.hpp:5: error: variable or field ‘init_aes’ declared void
pycryptopp/cipher/aesmodule.hpp:5: error: ‘PyObject’ was not declared in this scope
pycryptopp/cipher/aesmodule.hpp:5: error: ‘module’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:9: error: expected constructor, destructor, or type conversion before ‘(’ token
pycryptopp/_pycryptoppmodule.cpp:20: error: ‘PyMethodDef’ does not name a type
pycryptopp/_pycryptoppmodule.cpp: In function ‘void init_pycryptopp()’:
pycryptopp/_pycryptoppmodule.cpp:32: error: ‘PyObject’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:32: error: ‘module’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:34: error: ‘_pycryptopp_functions’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:34: error: ‘_pycryptopp__doc__’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:34: error: ‘Py_InitModule3’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:39: error: ‘init_rsa’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:40: error: ‘init_sha256’ was not declared in this scope
pycryptopp/_pycryptoppmodule.cpp:41: error: ‘init_aes’ was not declared in this scope
error: Setup script exited with error: command 'gcc' failed with exit status 1
someUser@someUser-ubuntu:~/Desktop/allmydata-tahoe-1.5.0$ 

[imhavoc]

Replying to zooko:

The workaround for Ubuntu is to install build-essential so that a pycryptopp will be built automatically from source.

We tried installing build-essential, and it continued to fail.

[zooko]

The informative line in that voluminous error output is:

 pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or
 directory

So the work-around is to install build-essential and python-dev. :-)

[imhavoc]

Replying to zooko:

The informative line in that voluminous error output is:

 pycryptopp/_pycryptoppmodule.cpp:2:20: error: Python.h: No such file or
 directory

So the work-around is to install build-essential and python-dev. :-)

It would be immensely helpful if this information was included in the download/install documentation. Python is not throwing useful errors for people who are not familiar with python builds.

So, let me start a list of system requirements that should be in place if you expect to see a successful build: build-essential, python-dev...

I see that I also installed python-openssl on my 8.04 system before getting a successful build.

Am I missing anything?

[zooko]

There are some notes about such things on ​http://allmydata.org/trac/tahoe/wiki/InstallDetails . Those notes aren't on ​http://allmydata.org/source/tahoe/trunk/docs/install.html because those steps aren't supposed to be necessary. The fact that they are necessary (on some platforms) is a bug, and is the subject of this ticket. Improving the documentation for those platforms where binaries of pycryptopp and the other dependencies are not available would be the subject of... let's see... #282 (more detailed and targeted docs about installing from source). Hm, see also #701 (The instructions in "Installing Tahoe" do not mention C compiler requirement.) which is pretty much the same issue as you are having but opened by my brother Nathan.

[imhavoc]

I stumbled across this page from the InstallDetails page: ​http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages

That's useful information. It should be linked from the install.html page.

[zooko]

It is!

[zooko]

Err, sorry, I meant that the InstallDetails page is linked from the install.html page. The DownloadDebianPackages page is too platform-specific to be linked from the install.html page.

Since imhavoc's friend is using 32-bit, I hereby enlarge this ticket to also provide 32-bit binaries of pycryptopp.

[zooko]

Okay, we've now fixed #1094 which is to automatically build bdist_eggs of pycryptopp on all supported platforms. Now the question is whether the supported platforms (see ​the pycryptopp buildbot and ​the tahoe-dep-eggs directory on tahoe-lafs.org) include the platforms that the users on this ticket needed...

[zooko]

Here is a nice ​table which shows which binary packages we are currently hosting for which platforms.

[zooko]

Any platform in ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/README.html which has a row that goes all the way across with every cell filled in is a platform where Tahoe-LAFS can be easily installed by someone who doesn't have a compiler or development libraries.

[midnightmagic]

here's a binary egg for netbsd-5.0.2 i386, from the non-syslib buildslave.

[zooko]

I'm going to call this good enough. We have binary pycryptopp eggs for every platform on our Supported Builder list (go to ​http://tahoe-lafs.org/buildbot/ and click on "Supported Builders") except for Solaris. I guess I'll open a new ticket asking for binary eggs for all of the needed dependencies for Solaris (those are: Twisted, pyOpenSSL, pycrypto, pycryptopp, pywin32, zfec, zope.interface as shown on ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/README.html .) There, the Solaris row of the matrix is now #1294.

[tilllt]

The bug still occurred for me on my raspberry pi running Linux 3.6.11+ armv6l and Python 2.7.3 where there is no pycryptoapp binary package. installing the suggested dependencies (python-dev python-openssl) solved this for me.

[zooko]

Thanks for the bug report, tilllt. Could you please paste in the output from python misc/build_helpers/show-tool-versions.py? That will tell us what kind of binary someone would need to provide to make it install on your platform without needing the dev packages.

[tilllt]

well, i was too quick in assuming the build had worked. actually the python setup.py build command finishes without error, but then tahoe won't run, complaining it cannot find it's dependency pycryptopp >=0.6. which is kind of weird. as even the first half hour of python setup.py trial did not have complains until i stopped it.

this is the output ​http://pastebin.com/eASzrxzk you asked for. so apparently somehow an outdated pycryptopp was installed along the way.

[tilllt]

now i did easy_install for pycryptopp from this page:

​https://tahoe-lafs.org/trac/pycryptopp

and now tahoe fails with

Traceback (most recent call last):
  File "/usr/local/bin/tahoe", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Twisted>=11.0.0

so i guess it is a different bug than the others in here had.

[tilllt]

ok, final update, after running:

sudo easy_install Twisted>=11.0.0

the build helpers look like this (and tahoe runs):

locale.locale: (None, None)

buildbot: no such file or directory
buildslave: no such file or directory
gcc: gcc (Debian 4.6.3-14+rpi1) 4.6.3
g++: g++ (Debian 4.6.3-14+rpi1) 4.6.3
cryptest: no such file or directory
git: git version 1.7.10.4 
openssl: OpenSSL 1.0.1c 10 May 2012 (Library: OpenSSL 1.0.1e 11 Feb 2013) 
darcs: no such file or directory
darcs-exact-version: no such file or directory
flappclient: Foolscap version: 0.6.4 Twisted version: 13.0.0 
valgrind: no such file or directory
lzip: no such file or directory
as: GNU assembler (GNU Binutils for Debian) 2.22 Copyright 2011 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or later. This program has absolutely no warranty. This assembler was configured for a target of `arm-linux-gnueabihf'.  
setuptools: [distribute 0.6.24dev-r0 (/usr/lib/python2.7/dist-packages)]

coverage: DistributionNotFound

trialcoverage: DistributionNotFound

pyflakes: DistributionNotFound

zope.interface: [zope.interface 3.6.1 (/usr/lib/python2.7/dist-packages)]
zope.interface module: <module 'zope.interface' from '/usr/lib/python2.7/dist-packages/zope/interface/__init__.pyc'>

setuptools_darcs: DistributionNotFound

darcsver: DistributionNotFound

Twisted: [Twisted 13.0.0 (/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-armv6l.egg), zope.interface 3.6.1 (/usr/lib/python2.7/dist-packages)]
Twisted module: <module 'twisted' from '/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-armv6l.egg/twisted/__init__.pyc'>
Twisted __version__: 13.0.0

TwistedCore: DistributionNotFound
TwistedCore module: <module 'twisted.python' from '/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-armv6l.egg/twisted/python/__init__.pyc'>

TwistedWeb: DistributionNotFound
TwistedWeb module: <module 'twisted.web' from '/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-armv6l.egg/twisted/web/__init__.pyc'>
TwistedWeb __version__: 13.0.0

TwistedConch: DistributionNotFound
TwistedConch module: <module 'twisted.conch' from '/usr/local/lib/python2.7/dist-packages/Twisted-13.0.0-py2.7-linux-armv6l.egg/twisted/conch/__init__.pyc'>
TwistedConch __version__: 13.0.0

pycryptopp: [pycryptopp 0.6.0.1206569328141510525648634803928199668821045408958 (/usr/local/lib/python2.7/dist-packages/pycryptopp-0.6.0.1206569328141510525648634803928199668821045408958-py2.7-linux-armv6l.egg), distribute 0.6.24dev-r0 (/usr/lib/python2.7/dist-packages)]
pycryptopp module: <module 'pycryptopp' from '/usr/local/lib/python2.7/dist-packages/pycryptopp-0.6.0.1206569328141510525648634803928199668821045408958-py2.7-linux-armv6l.egg/pycryptopp/__init__.pyc'>
pycryptopp __version__: 0.6.0.1206569328141510525648634803928199668821045408958

[zooko]

Replying to tilllt:

ok, final update, after running:

sudo easy_install Twisted>=11.0.0

the build helpers look like this (and tahoe runs):

locale.locale: (None, None)

buildbot: no such file or directory

Hey wait a minute. The part of the output that I wanted is what comes before locale.locale and buildbot. The first line should be platform and the second machine. Here's what it looks like on my system:

$ python misc/build_helpers/show-tool-versions.py  |  head
platform: Linux-3.9.3cz9-x86_64-with-LinuxMint-14-nadia
machine:  x86_64
linux_distribution: ('LinuxMint', '14', 'nadia')

python: 2.7.3 (default, Sep 26 2012, 21:51:14)  [GCC 4.7.2]
maxunicode: 1114111

locale: LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= 
filesystem.encoding: UTF-8
locale.getpreferredencoding: UTF-8

[tilllt]

ah sorry, i am running the raspberry pi headless and had "screen" running in the terminal which cut off the beginning of the output:

platform: Linux-3.6.11+-armv6l-with-debian-7.0
machine:  armv6l
linux_distribution: ('debian', '7.0', '')

python: 2.7.3 (default, Jan 13 2013, 11:20:46)  [GCC 4.6.3]
maxunicode: 1114111

locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
locale: LANG=en_GB.UTF-8 LANGUAGE= LC_CTYPE=UTF-8 LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_PAPER="en_GB.UTF-8" LC_NAME="en_GB.UTF-8" LC_ADDRESS="en_GB.UTF-8" LC_TELEPHONE="en_GB.UTF-8" LC_MEASUREMENT="en_GB.UTF-8" LC_IDENTIFICATION="en_GB.UTF-8" LC_ALL= 
filesystem.encoding: ANSI_X3.4-1968
locale.getpreferredencoding: ANSI_X3.4-1968
got exception from locale.getdefaultlocale():  unknown locale: UTF-8
locale.locale: (None, None)

[zooko]

Cool, thanks! Okay, I think you will be able to install Tahoe-LAFS on Raspberry Pi without compiling anything from source as soon as there is a new row on ​https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html in the "Python 2.7" table which row is labeled "linux-armv6l" and which has an egg in each of its six columns.

Okay, so which Raspberry Pi owner wants to build these six eggs for us?

[daira]

Perhaps we should have a ticket for each platform that needs eggs building for it?

Note that some eggs need newer versions because of these dependency increases for 1.10:

• Twisted >= 11.0.0
• pycryptopp >= 0.6.0
• zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5

[zooko]

Replying to daira:

Perhaps we should have a ticket for each platform that needs eggs building for it?

Good idea. I created #1983.

[daira]

Replying to daira:

Perhaps we should have a ticket for each platform that needs eggs building for it?

Filed as #1978 to #1985.

[daira]

Now we need to recruit some egg-building volunteers to assign the tickets to.