Mailman 3 December 2015 - Cryptography-dev

[Proposal] Intent to deprecate and remove: Python 2.6
by Alex Gaynor Jan. 20, 2016

Jan. 20, 2016

Hi all, I'd like to propose we formally deprecate (in our next release) and remove (in the release thereafter) support for Python2.6. It is no longer under support from its developers (which makes it a security risk), its use is more or less confined to users of CentOS and RHEL (which means SCL is available to them, and they have a corporate benefactor to complain to if it doesn't work), and its seriously funking with our development workflow because it's so abysmally slow in CI (which means it's annoying me enough to want to solve this). There is genuinely no justifiable reason to use Python 2.6 anymore, both Django and Twisted have dropped it. If this would _seriously_ impact you, please reply to this thread. If possible I'd also like to hear from our downstreams. Cheers, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084

6 6

Keys/Certificates/CRLs/x509 in pyOpenSSL
by Hynek Schlawack Jan. 4, 2016

Jan. 4, 2016

Hi, we have quite a bit of pull requests on pyOpenSSL that revolve around improving the state of x509 objects in general as far as I understand it. Since I already got reprimanded by Alex G for merging one because cryptography has routines for that, I wonder if we should close them all as WONTFIX and instead add methods akin to `PKey.from_cryptography()`, `key_instance.to_cryptography()`. # Questions - Am I misunderstanding something completely and this can’t happen for practical reasons? - Does cryptography have everything in place to achieve this at all? # Upsides - Keeps pyOpenSSL from playing catch-up. - Saves work in the long term. - Gives users more power independently from pyOpenSSL releases. - Risks of breaking backward compatibility are rather low. - People would get used to do x509 work with cryptography in preparation for `cryptography.tls`. # Downsides - I would need *active* help from either Paul or Alex G or someone who really understands both x509 (I know, nobody actually understands that, please don’t go there) and the cryptography layer on both drafting this up and implementing it. Therefore this is absolutely *not my call* to make. - It would further delay improvements to pyOpenSSL and the people on the bug tracker are already rightfully rather restless. I’m afraid this might lead to months of nothing happening. *** I welcome any feedback. The current pyOpenSSL situation which is mostly a swamp of guilt is becoming unbearable to me. When I took over maintainership I made it clear that I see myself mostly as a repo janitor and Bad Ideas Deflector™. Sadly that’s not working out at all. Getting rid of the burden of actually moving forward a whole sub-system might alleviate that a bit I guess (this is not meant as an ultimatum, I have no idea if it’d help). Best, —h

4 5

Supporting "memory view" and stream output
by Jesus Cea Dec. 13, 2015

Dec. 13, 2015

When calling Cryptography with a memoryview I get this error: """ TypeError: initializer for ctype 'unsigned char *' must be a cdata pointer, not memoryview """ Memory view are useful to avoid copies, eating memory and CPU for no reason. In my personal situation, I have 4 MB blocks, each divided in 16 pieces of 256Kbytes independently encrypted. I must extract those 256Kbytes fragments to call cryptography, instead of just passing a memory view with the appropriate slice. A secondary request would be to provide a different option for output, for the same reason: provide a target for output: a bytearray/writable memory view, or a file-like object. For the same reason: avoiding copies and ease concatenation. Hope you find these suggestions useful. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

2 1

PyCA cryptography 1.1.2 released
by Paul Kehrer Dec. 10, 2015

Dec. 10, 2015

PyCA/cryptography (https://github.com/pyca/cryptography) 1.1.2 has been released. cryptography is a package which provides cryptographic recipes and primitives to Python developers. Our goal is for it to be your "cryptographic standard library". We support Python 2.6-2.7, Python 3.3+, and PyPy. Changelog: * Fixed a SIGBUS crash with the OS X wheels caused by redefinition of a method. * Fixed a runtime error "undefined symbol EC_GFp_nistp224_method" that occurred with some OpenSSL installations. * Updated Windows and OS X wheels to be compiled against OpenSSL 1.0.2e. -Paul Kehrer (reaperhulk)

1 0

Minimal Test Case, bad interaction between "cryptography" and "wsgi" (Re: [modwsgi] mod_wsgi and creating new SSL connections inside the "body" iterator)
by Jesus Cea Dec. 4, 2015

Dec. 4, 2015

Context for "Cryptography" people: Using "cryptography" and then trying to create a "ssl.SSLContext()" hangs the progress when running under mod_wsgi. I can not reproduce this issue with a standalone script, it must be run inside mod_wsgi. May be useful to know that, in daemon mode, mod_wsgi uses several threads to monitor health and do maintenance, beside the main wsgi application. Mail thread in the mod_wsgi mailing list in <https://groups.google.com/forum/#!topic/modwsgi/WFLao7kOqxE>. On 04/12/15 01:00, Graham Dumpleton wrote: > You don’t happen to be using PostgreSQL in the same web application > process are you? No. I have quite a few modules in my apache, but I am running WSGI in daemon mode. I have a minimal testcase. I can not reproduce this issue OUTSIDE of mod_wsgi, so it seems to be a bad interaction between mod_wsgi and "cryptography" library. The minimal test case, mod_wsgi compiled to use Python 3.5.0, is: """ from cryptography.hazmat.primitives.ciphers import Cipher from cryptography.hazmat.primitives.ciphers import algorithms as \ Cipher_algorithms from cryptography.hazmat.primitives.ciphers import modes as \ Cipher_modes from cryptography.hazmat.backends import default_backend as \ Cipher_backend import ssl def application(environ, start_response) : cipher = Cipher( Cipher_algorithms.AES(b'e'*16), Cipher_modes.ECB(), backend = Cipher_backend()) ssl.SSLContext(ssl.PROTOCOL_SSLv23) # <- HANGS HERE!!! crypto_iv = cipher.encryptor() start_response('200 OK', []) return [b'SUCCESS!\r\n'] """ This code hangs the HTTP request in my machine. I am using Cryptography 1.1.1 and mod_wsgi 4.4.21. This server has OpenSSL 1.0.1. It is an Ubuntu 12.04 fully patched. A "thread apply all bt" in GDB shows this: """ Thread 19 (Thread 0x7fa947867700 (LWP 11497)): #0 0x00007fa952a3a6f3 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fa952f4d815 in apr_sleep (t=<optimized out>) at time/unix/time.c:246 #2 0x00007fa94cb78ee0 in wsgi_monitor_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:8658 #3 0x00007fa952d13e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 18 (Thread 0x7fa9477e6700 (LWP 11498)): #0 0x00007fa952d180fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fa94c792aff in take_gil () at Python/condvar.h:103 #2 0x00007fa94c793099 in PyEval_RestoreThread () at Python/ceval.c:450 #3 0x00007fa94c7ccb2b in PyGILState_Ensure () from /usr/local/lib/libpython3.5m.so.1.0 #4 0x00007fa94cb7ac9a in wsgi_deadlock_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:8468 #5 0x00007fa952d13e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 17 (Thread 0x7fa947765700 (LWP 11499)): #0 0x00007fa952d17d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fa94cb8325d in wsgi_worker_acquire (id=0) at src/server/mod_wsgi.c:8044 #2 wsgi_daemon_worker (thread=<optimized out>, p=0x1788588) at src/server/mod_wsgi.c:8166 #3 wsgi_daemon_thread (thd=0x15cae30, data=<optimized out>) at src/server/mod_wsgi.c:8425 #4 0x00007fa952d13e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () [QUITE A FEW THREADS WITH THE SAME STACK] Thread 3 (Thread 0x7fa947057700 (LWP 11513)): #0 0x00007fa952d180fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fa94c792aff in take_gil () at Python/condvar.h:103 #2 0x00007fa94c793099 in PyEval_RestoreThread () at Python/ceval.c:450 #3 0x00007fa94c7ccb2b in PyGILState_Ensure () from /usr/local/lib/libpython3.5m.so.1.0 #4 0x00007fa9363c51d1 in invoke_callback () from /usr/local/lib/python3.5/site-packages/_cffi_backend.cpython-35m-x86_64-linux-gnu.so #5 0x00007fa9361aa819 in ffi_closure_unix64_inner () from /usr/lib/x86_64-linux-gnu/libffi.so.6 #6 0x00007fa9361aab7c in ffi_closure_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6 #7 0x00007fa94b7585e5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #8 0x00007fa94b7583c8 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #9 0x00007fa94b76efbb in EC_KEY_generate_key () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #10 0x00007fa94bab3998 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #11 0x00007fa94474d6d5 in _ssl__SSLContext () from /usr/local/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so #12 0x00007fa94c7190a4 in tp_new_wrapper () from /usr/local/lib/libpython3.5m.so.1.0 #13 0x00007fa94c7023e9 in PyCFunction_Call () from /usr/local/lib/libpython3.5m.so.1.0 #14 0x00007fa94c79d04d in PyEval_EvalFrameEx () at Python/ceval.c:4651 #15 0x00007fa94c79dab6 in _PyEval_EvalCodeWithName () at Python/ceval.c:3962 #16 0x00007fa94c79dba8 in PyEval_EvalCodeEx () at Python/ceval.c:3983 #17 0x00007fa94c6df076 in function_call () from /usr/local/lib/libpython3.5m.so.1.0 #18 0x00007fa94c6ade0a in PyObject_Call () from /usr/local/lib/libpython3.5m.so.1.0 #19 0x00007fa94c714ab5 in slot_tp_new () from /usr/local/lib/libpython3.5m.so.1.0 #20 0x00007fa94c7191e3 in type_call () from /usr/local/lib/libpython3.5m.so.1.0 #21 0x00007fa94c6ade0a in PyObject_Call () from /usr/local/lib/libpython3.5m.so.1.0 #22 0x00007fa94c79711e in PyEval_EvalFrameEx () at Python/ceval.c:4883 #23 0x00007fa94c79dab6 in _PyEval_EvalCodeWithName () at Python/ceval.c:3962 #24 0x00007fa94c79dba8 in PyEval_EvalCodeEx () at Python/ceval.c:3983 #25 0x00007fa94c6df076 in function_call () from /usr/local/lib/libpython3.5m.so.1.0 #26 0x00007fa94c6ade0a in PyObject_Call () from /usr/local/lib/libpython3.5m.so.1.0 #27 0x00007fa94c793787 in PyEval_CallObjectWithKeywords () at Python/ceval.c:4526 #28 0x00007fa94cb81822 in Adapter_run (object=0x7fa946f16ea0, self=0x7fa944723210) at src/server/mod_wsgi.c:2897 #29 wsgi_execute_script (r=0x7fa940004980) at src/server/mod_wsgi.c:3726 #30 0x00007fa94cb83c69 in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:12236 #31 wsgi_process_socket (bucket_alloc=0x7fa940006e40, sock=<optimized out>, p=0x7fa940005e90, daemon=<optimized out>) at src/server/mod_wsgi.c:8007 #32 wsgi_daemon_worker (thread=<optimized out>, p=0x17a7638) at src/server/mod_wsgi.c:8336 #33 wsgi_daemon_thread (thd=0x1795368, data=<optimized out>) at src/server/mod_wsgi.c:8425 #34 0x00007fa952d13e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #35 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #36 0x0000000000000000 in ?? () [MORE BORING THREADS] Thread 1 (Thread 0x7fa9537c2700 (LWP 11094)): #0 0x00007fa952a359d3 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fa952f49563 in apr_poll (aprset=0x7fffca6fd500, num=1, nsds=0x7fffca6fd55c, timeout=<optimized out>) at poll/unix/poll.c:120 #2 0x00007fa94cb859ab in wsgi_daemon_main (daemon=0x15ca3d8, p=0x159a138) at src/server/mod_wsgi.c:8944 #3 wsgi_start_process (p=0x159a138, daemon=0x15ca3d8) at src/server/mod_wsgi.c:9665 #4 0x00007fa94cb87494 in wsgi_start_daemons (p=0x159a138) at src/server/mod_wsgi.c:9877 #5 0x00000000004370f0 in ap_run_pre_mpm () #6 0x000000000048454c in worker_run () #7 0x0000000000433618 in ap_run_mpm () #8 0x000000000042b340 in main () """ It is looks like some kind of callback is registered by "cryptography" and when calling "ssl.SSLContext()" that callback tries to get the GIL when it already has it, hanging. My speculation, I am not familiar with the inner working of OpenSSL neither "cryptography", but the traceback of "Thread 3" looks quite suspicious. In recent Python releases, the GIL is a NOP if not threads are created. Lets try something... Lets create another idle thread. """ from cryptography.hazmat.primitives.ciphers import Cipher from cryptography.hazmat.primitives.ciphers import algorithms as \ Cipher_algorithms from cryptography.hazmat.primitives.ciphers import modes as \ Cipher_modes from cryptography.hazmat.backends import default_backend as \ Cipher_backend import ssl, threading, time def dummy() : cipher = Cipher( Cipher_algorithms.AES(b'e'*16), Cipher_modes.ECB(), backend = Cipher_backend()) ssl.SSLContext(ssl.PROTOCOL_SSLv23) crypto_iv = cipher.encryptor() while True : pass t = threading.Thread(target = dummy) t.setDaemon(True) t.start() print("SSSS") time.sleep(2) print("OK!") """ This code works right. I can only reproduce the issue inside mod_wsgi. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

2 2