Loading a huge RSA key is _extremely_ slow on Ubuntu 22.04

[itamarst]

On Ubuntu 22.04, the test in test_crypto.py that loads a huge RSA key takes a ludicrous amount of time.

This is on my new, twice-as-fast-cores computer. Like, a minute, maybe more, it just sits there.

Maybe OpenSSL 3? Except OpenSSL is probably packaged in the relevant wheels?

I suggest either skipping the test on Ubuntu 22.04, or maybe just deleting it.

[itamarst]

I'm using cryptography 37.0.4.

[exarkun]

On CircleCI, this is reported as the slowest test, taking more than 4 minutes to run on at least 95% of test runs.

I can reproduce this locally with tox where these versions are installed:

appdirs==1.4.4
attrs==22.1.0       
autobahn==22.7.1
Automat==22.10.0        
bcrypt==4.0.1
beautifulsoup4==4.11.1 
boltons==21.0.0       
cbor2==5.4.5     
certifi==2022.9.24     
cffi==1.15.1                                                                                                                                                
cfgv==3.3.1      
charset-normalizer==2.1.1
click==8.1.3   
click-default-group==1.2.2
collections-extended==2.0.2
constantly==15.1.0
coverage==5.5      
cryptography==38.0.4
decorator==5.1.1  
distlib==0.3.6           
distro==1.8.0
eliot==1.14.0     
exceptiongroup==1.0.4
extras==1.0.0           
filelock==3.8.0 
fixtures==4.0.1    
flake8==3.8.4      
foolscap==21.7.0
future==0.18.2
greenlet==2.0.1      
hkdf==0.0.3     
html5lib==1.1                                                                 
humanize==4.4.0
hyperlink==21.0.0
hypothesis==6.59.0
identify==2.5.9
idna==3.4
incremental==22.10.0                                                                                                                                [30/129]
iniconfig==1.1.1
Jinja2==3.1.2
junitxml==0.7
klein==21.8.0
magic-wormhole==0.12.0
MarkupSafe==2.1.1
mccabe==0.6.1
mock==4.0.3
netifaces==0.11.0
nodeenv==1.7.0
packaging==21.3
paramiko==2.8.1
Parsley==1.3
pbr==5.11.0
platformdirs==2.5.4
pluggy==1.0.0
pre-commit==2.20.0
prometheus-client==0.11.0
psutil==5.9.4
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycddl==0.2.2
pycodestyle==2.6.0
pycparser==2.21
pyflakes==2.2.0
PyNaCl==1.5.0
pyOpenSSL==22.1.0
pyparsing==3.0.9
pyrsistent==0.19.2
pytest==7.2.0
pytest-timeout==2.1.0
pytest-twisted==1.14.0
python-subunit==1.4.2
pyutil==3.3.0
PyYAML==6.0
requests==2.28.1
service-identity==21.1.0
six==1.16.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
spake2==0.8
subunitreporter==22.2.0
tahoe-lafs @ file:///home/exarkun/Work/python/tahoe-lafs/.tox/.tmp/package/1/tahoe-lafs-1.18.1.dev172%2Bgb56f53b77.zip
tenacity==8.1.0
testtools==2.5.0
toml==0.10.2
tomli==2.0.1
towncrier==22.8.0
tox==3.27.1
tqdm==4.64.1
treq==22.2.0
Tubes==0.2.1
Twisted==22.10.0
txaio==22.2.1
txi2p-tahoe==0.3.7
txtorcon==22.0.0
typing_extensions==4.4.0
urllib3==1.26.13
virtualenv==20.17.0
webencodings==0.5.1
Werkzeug==2.2.2
zfec==1.5.7.2
zope.interface==5.5.2

but I fail to reproduce it in a different virtualenv (the test takes about 4 seconds there):

aniso8601==9.0.1
appdirs==1.4.4
attrs==21.4.0
autobahn==21.11.1
Automat==20.2.0
bcrypt==3.2.0
beautifulsoup4==4.10.0
black==22.3.0
boltons==21.0.0
build==0.9.0
cbor2==5.4.2
certifi==2021.10.8
cffi==1.15.0
characteristic==14.3.0
charset-normalizer==2.0.10
click==8.0.3
click-default-group==1.2.2
collections-extended==2.0.2
colorama==0.4.6
colored==1.4.3
compose==1.4.8
constantly==15.1.0
coverage==5.5
cryptography==36.0.1
decorator==4.4.2
distlib==0.3.4
distro==1.6.0
eliot==1.14.0
eliot-tree==21.0.0
execnet==1.9.0
extras==1.0.0
filelock==3.4.2
fixtures==3.0.0
flake8==3.8.4
foolscap==21.7.0
future==0.18.2
greenlet==1.1.2
hkdf==0.0.3
html5lib==1.1
humanize==3.13.1
hyperlink==21.0.0
hypothesis==6.36.0
idna==3.3
incremental==21.3.0
iniconfig==1.1.1
iso8601==1.0.2
isort==5.10.1
Jinja2==3.0.3
jmespath==1.0.1
junitxml==0.7
klein==21.8.0
magic-wormhole==0.12.0
MarkupSafe==2.0.1
mccabe==0.6.1
mock==4.0.3
mypy==0.981
mypy-clean-slate==0.1.5
mypy-extensions==0.4.3
mypy-zope==0.3.11
netifaces==0.11.0
packaging==21.3
paramiko==2.8.1
Parsley==1.3
pathspec==0.9.0
pbr==5.8.0
pep517==0.13.0
pip-tools==6.10.0
platformdirs==2.4.1
pluggy==1.0.0
prometheus-client==0.11.0
psutil==5.9.2
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycddl==0.2.2
pycodestyle==2.6.0
pycparser==2.21
pyflakes==2.2.0
PyHamcrest==2.0.3
PyNaCl==1.5.0
pyOpenSSL==21.0.0
pyparsing==3.0.6
pyrsistent==0.18.1
pytest==6.2.5
pytest-forked==1.4.0
pytest-timeout==2.1.0
pytest-twisted==1.13.4
pytest-xdist==2.5.0
python-challenge-bypass-ristretto==2022.6.30
pytz==2022.6
pyutil==3.3.0
PyYAML==6.0
requests==2.27.1
service-identity==21.1.0
setuptools-scm==7.0.5
six==1.16.0
sortedcontainers==2.4.0
soupsieve==2.3.1
spake2==0.8
sqlparse==0.4.3
-e git+ssh://git@github.com/tahoe-lafs/tahoe-lafs@b56f53b77c7c5f4f184cd82dca5d03c96664389a#egg=tahoe_lafs
tenacity==8.0.1
testtools==2.5.0
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
towncrier==21.3.0
tox==3.24.5
tqdm==4.62.3
treq==21.5.0
Tubes==0.2.0
Twisted==22.10.0
txaio==21.2.1
-e git+ssh://git@github.com/tahoe-lafs/txi2p@a2e23a2a38f1bb110f90eb77b60fd9a6be62e42d#egg=txi2p_tahoe
txtorcon==21.1.0
types-cryptography==3.3.23.2
types-mock==4.0.15.2
types-pkg-resources==0.1.3
types-pyOpenSSL==22.1.0.2
types-PyYAML==6.0.12.2
types-six==1.16.21.4
typing_extensions==4.0.1
urllib3==1.26.8
virtualenv==20.13.0
webencodings==0.5.1
Werkzeug==2.0.2
zero-knowledge-access-pass-authorizer==2022.8.21
zfec==1.5.5
zope.event==4.5.0
zope.interface==5.4.0
zope.schema==6.2.0

There's a *lot* that's different between the two envs. However, jumping straight to cryptography on the assumption that that's where a lot of the work is happening...

When I upgrade cryptography (36.0.1) and pyopenssl (21.0.0) in the fast env to the versions in the slow env (38.0.4 and 22.0.0 respectively), the fast env gets slow.

And with cryptography 37.0.0 it's still slow but with 36.0.2 it's fast again.

[exarkun]

Reading the cryptography changelog, the headline item for 37.0.0 is...

Updated Windows, macOS, and Linux wheels to be compiled with OpenSSL 3.0.2.

:/

[exarkun]

Here's a very narrow demonstration of the performance difference:

❯ time python -c 'from cryptography import __version__
print(__version__)
from cryptography.hazmat.primitives.serialization import load_der_private_key
from base64 import b64decode
with open("src/allmydata/test/data/pycryptopp-rsa-32768-priv.txt", "rb") as f:
  load_der_private_key(b64decode(f.read()), password=None)
'
36.0.2

real    0m4.702s
user    0m4.689s
sys     0m0.012s

❯ pip install --upgrade cryptography==37.0
Collecting cryptography==37.0
  Using cached cryptography-37.0.0-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
...

❯ time python -c 'from cryptography import __version__
print(__version__)
from cryptography.hazmat.primitives.serialization import load_der_private_key
from base64 import b64decode
with open("src/allmydata/test/data/pycryptopp-rsa-32768-priv.txt", "rb") as f:
  load_der_private_key(b64decode(f.read()), password=None)
'
37.0.0

real    3m8.899s
user    3m8.556s
sys     0m0.052s

OpenSSL 3 is basically 180x slower than OpenSSL 1.1.1 at decoding this key.

[exarkun]

This is a known slowdown. OpenSSL 3 introduces some "primality" tests. It is possible to disable these (pass unsafe_skip_rsa_key_validation=True to the above API). As the parameter name suggests, this option is "unsafe" (in the sense that setting it and also taking untrusted key material exposes you to unspecified behavior from OpenSSL).

The *test suite* is not exposed to untrusted key material so it would probably be fine to set the option there (and reclaim about 4 minutes from each test run). However, I'm not entirely confident that we can say the same thing about all production uses of the function.

[exarkun]

Oops, unsafe_skip_rsa_key_validation was added in the yet-to-be released cryptography 39. So hard to make use of it quite yet...

​https://github.com/pyca/cryptography/pull/7667

[exarkun]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/1239, someday

[exarkun]

39 was released a couple days ago so we can now optionally take advantage of this feature, I think.

[GitHub <noreply@…>]

In 3c3697d/trunk:

Merge pull request #1239 from tahoe-lafs/3914.faster-rsa-tests

Speed up that one RSA test

Fixes: ticket:3914