Mailman 3 August 2016 - Cryptography-dev

v1.4: _openssl has no Cryptography_STATIC_CALLBACKS
by Robin Fishbein Sept. 28, 2016

Sept. 28, 2016

Hello, I imagine this may have a trivial explanation, or perhaps indicate some quirk on my machine, but it looks as though some desired functionality isn’t available on my machine with version 1.4. I wanted to share this in case it indicates some wider issue of interest, and helps me learn a little about how cryptography works under the hood. Windows 7, 64-bit, Python 3.5.1, and I believe OpenSSL 1.0.2. I use paramiko for SFTP, and when updating paramiko (on a whim) to 2.0.2 with Anaconda, it had me update cryptography as well, to version 1.4, or maybe it was 1.3 at first — either one experiences the issue. When attempting an SSH connect, I get a traceback ending with this: File "C:\Users\...\AppData\Local\Continuum\Anaconda3\lib\site-packages\cryptography\hazmat\bindings\openssl\binding.py", line 73, in wrapper if lib.Cryptography_STATIC_CALLBACKS: AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_STATIC_CALLBACKS' When I manually carry out some of the imports in the source... from cryptography.hazmat.bindings._openssl import ffi, lib ...and then look at what auto-complete shows for lib, I don’t see Cryptography_STATIC_CALLBACKS, which is consistent with the AttributeError. I don’t yet understand where to look for the source underneath the bootstrapped _openssl.*.pyd file. The SSH connection and all subsequent functionality work fine with the latest versions of paramiko and cffi as long as I revert cryptography to 1.0.2. -Robin Fishbein

2 2

pyca/cryptography 1.5 released
by Paul Kehrer Aug. 26, 2016

Aug. 26, 2016

PyCA cryptography 1.5 has been released to PyPI. cryptography is a package which provides cryptographic recipes and primitives to Python developers. Our goal is for it to be your "cryptographic standard library". We support Python 2.6-2.7, Python 3.3+, and PyPy. Changelog: * Added calculate_max_pss_salt_length for RSA PSS. * Added "one shot" DSAPrivateKey.sign andDSAPublicKey.verify methods to DSA keys. * Added "one shot" EllipticCurvePrivateKey.sign and EllipticCurvePublicKey.verify methods to ECDSA keys. * Switched back to the older callback model on Python 3.5 in order to mitigate a locking callback problem with OpenSSL <1.1.0. * x509.CertificateBuilder, x509.CertificateRevocationListBuilder, and x509.RevokedCertificateBuilder now accept timezone aware datetime objects as method arguments. * cryptography now supports OpenSSL 1.1.0 as a compilation target. As of 1.5 we are still linking 1.0.2h for the Mac and Windows wheels, but in the 1.6 release we will switch to linking 1.1.0. Thanks to all the contributors for their help on this release! -Paul Kehrer (reaperhulk)

1 0

on how (not) to chain certs with openssl + pyopenssl
by Joel Sandin Aug. 22, 2016

Aug. 22, 2016

I wanted to use pyOpenSSL to verify a certificate chain in a TLS Certificate Handshake message, and in the process stumbled upon a _disturbing_ amount of misinformation online about how to do this, both using openSSL and with pyOpenSSL. This isn't about a bug in pyOpenSSL, but the combination of a) lack of documentation and b) potentially misleading unit tests could result in serious vulnerabilities for users of the library. It's clear from what I've found online that developers are confused and may have introduced vulnerabilities into their code. I sent this to the pyOpenSSL security contact and was cleared to post this to the list to solicit feedback. FWIW there seems to be an ongoing effort to _properly_ support chain verification in pyOpenSSL: https://github.com/pyca/pyopenssl/issues/502 https://github.com/pyca/pyopenssl/pull/473 But earlier changes to pyOpenSSL claim to support some form of chain verification, and appear to be the source of confusion for developers: https://github.com/pyca/pyopenssl/pull/155 The rest is long-winded but captures some of the issues with existing documentation online as well as the pyOpenSSL verify_certificate() function for those interested. best wishes Joel Cert "Chain" Verification ------------------------- As part of a system I'm building, I'd like to independently verify the server certificate included in a TLS Certificate (0x0b) Handshake message, sent by the server after the ServerHello Handshake message. The Certificate Handshake message contains the server's certificate, intermediate certificates needed to verify the server certificate, and (optionally) the root certificate of the CA that issued the intermediate cert(s). >From the TLS RFC: https://tools.ietf.org/html/rfc5246#page-47 certificate_list This is a sequence (chain) of certificates. The sender's certificate MUST come first in the list. Each following certificate MUST directly certify the one preceding it. Because certificate validation requires that root keys be distributed independently, the self-signed certificate that specifies the root certificate authority MAY be omitted from the chain, under the assumption that the remote end must already possess it in order to validate it in any case. Lets frame the problem: My goal is to verify this chain against a root certificate that I already know about and trust. The intermediates I'm presented are certificates I may never have seen before, and may have been tampered with in transit or replaced by an attacker that is impersonating the server I'm trying to connect to. Thus _chain_ is an important term here - I need to verify the site certificate, but in the process, I need to check the validity of any intermediate certs as well. In this use case, the intermediate cert is UNTRUSTED data and nothing prevents an attacker from replacing it with a self-signed certificate (e.g. root cert) - Calling it an intermediate because thats what the server claims it is doesn't make it one. To me, this is what verifying a chain means. And IMHO, this is the 'common case' for using intermediates - there are contexts where a system directly chooses to trust an intermediate, but I do not. Unfortunately, many of the solutions posted online (including those for pyOpenSSL) mistakenly trust the intermediate and make bypassing the verification process trivial for my use case. The Command Line ---------------- Lets review some of the misinformation about the use of 'openssl verify' for this purpose, as this misinformation could be the source of future bugs if developers decide to re-implement verification based on the command-line behavior. How do we supply the chain cert(s) for our use case? The intermediates are untrusted, so the answer is to use -untrusted (check man verify) but this is not the only suggestion. Googling turns up harmful guidance on the issue. For example, the top hit: http://stackoverflow.com/questions/25482199/verify-a- certificate-chain-using-openssl-verify This thread fortunately mentions -untrusted, but also includes several contradictory recommendations*, including the following to verify UserCert.pem: openssl verify -verbose -CAfile <(cat Intermediate.pem RootCert.pem) UserCert.pem Here, the intermediate and root are passed via the -CAfile command line argument, and the certificate to verify is in UserCert.pem. Is the poster deliberately trusting the intermediate? A person asks 'Will this actually verify the intermediate cert against the root cert?' and someone responds saying yes, it does. Variations of this advice is repeated elsewhere in various forms, for example: http://superuser.com/questions/904859/why-cant-i- verify-this-certificate-chain recommending: $ cat root.pem intermediate.pem > concat.pem $ openssl verify -CAfile concat.pem john.pem john.pem: OK This discussion also seems to echo the above, depending on how enduser-example.com.chain is generated: https://raymii.org/s/tutorials/OpenSSL_command_ line_Root_and_Intermediate_CA_including_OCSP_CRL%20and_revocation.html openssl verify -CAfile enduser-certs/enduser-example.com.chain enduser-certs/enduser-example.com.crt So what happens if we follow this advice for our problem? -CAfile vs -untrusted --------------------- Lets compare the approach described above to the use of -untrusted with real certificates. When we connect to sometechcompany.com we get the server certificate sometechcompany.com.pem, the intermediate sometechcompany_ist_ca_2.pem, and the CA certificate geotrust_global_ca.pem. *After confirming that geotrust_global_ca.pem is a certificate we trust*, we use openssl verify to verify the chain (note that the RFC allows the root CA to be omitted): Both of these indicate that that the sometechcompany.com.pem is trusted: $ openssl verify -CAfile geotrust_global_ca.pem -untrusted sometechcompany_ist_ca_2.pem sometechcompany.com.pem $ openssl verify -CAfile <(cat geotrust_global_ca.pem sometechcompany_ist_ca_2.pem) sometechcompany.com.pem If we omit the chain cert, or replace it with an invalid chain certificate, verification fails (as it should): $ verify -CAfile <(cat geotrust_global_ca.pem) sometechcompany.com.pem sometechcompany.com.pem: CN = sometechcompany.com, OU = management:idms.group.105316, O = sometechcompany Inc., ST = California, C = US error 20 at 0 depth lookup:unable to get local issuer certificate $ verify -CAfile <(cat geotrust_global_ca.pem invalid_chain_cert.crt) sometechcompany.com.pem sometechcompany.com.pem: CN = sometechcompany.com, OU = management:idms.group.105316, O = sometechcompany Inc., ST = California, C = US error 20 at 0 depth lookup:unable to get local issuer certificate It's complaining about the missing chain cert, so it seems like it must be validating it against the root CA. These work too: $ openssl verify -CAfile <(cat sometechcompany_ist_ca_2.pem) sometechcompany.com.pem sometechcompany.com.pem: OK $ openssl verify -untrusted sometechcompany_ist_ca_2.pem sometechcompany.com.pem sometechcompany.com.pem: OK So on this system, openssl is using the trusted certificates for verification. Because sometechcompany_ist_ca_2.pem isn't self signed, this *is* actually "verifying" it. Unfortunately, an "intermediate" cert that is actually a root / self-signed _will be treated as a trusted CA_ when using the recommended command given above: $ openssl verify -CAfile <(cat geotrust_global_ca.pem rogue_ca.pem) fake_sometechcompany_from_rogue_ca.com.pem fake_sometechcompany_from_rogue_ca.com.pem: OK This makes it trivial for a malicious server to "impersonate" sometechcompany.com, at least from the point of view of our verification scheme. The attacker creates a root cert, sign fake_sometechcompany_from_rogue_ca.com.pem with it, and sends this root as a so-called "intermediate". As we've seen, this approach to verification will show the server cert as valid, when in fact it is not issued by any of the CAs we trust. To go back to the right solution: Specifying the intermediate as untrusted (via -untrusted) causes this ultimately-untrusted certificate to fail validation, as it should: $ openssl verify -CAfile ~/geotrust_global_ca.pem -untrusted rogue_ca.pem fake_sometechcompany_from_rogue_ca.com.pem fake_sometechcompany_from_rogue_ca.com.pem: C = US, ST = Vermont, L = Barre, O = Trusted CA Intermediate Signing Cert, CN = intermediate.trusted.ca, emailAddress = intermediate(a)yahoo.com error 19 at 1 depth lookup:self signed certificate in certificate chain A rogue intermediate produces the same results. Note that there are more subtleties to using verify (w.r.t. purpose), also discussed http://stackoverflow.com/questions/23304139/openssl- verify-gives-ok-for-bad-certificate-chain This link also points out another gruesome usability wart for the command line tool - openSSL ends up ignoring multiple certs in the file passed for verification, and only verifies the first cert (I saw this recommended somewhere but can't find the link). Chain verification in pyopenssl: -------------------------------- This is all a diversion though - My real goal was to use pyOpenSSL for this purpose. There's nothing in the documentation about doing this so looking at the unit tests and code (grepping chain) is the next step. I'm not the first to do so - Googling 'how to verify certificate chain in python' produces the following stackoverflow post as the first hit. The poster bases their recommendations on pyopenssl unit tests: http://stackoverflow.com/questions/30700348/how-to-validate-verify-an-x509- certificate-chain-of-trust-in-python Another hit, same recommendation: http://www.yothenberg.com/validate-x509-certificate-in-python/ This second at least calls the intermediate cert 'trusted', but that contradicts the idea of any 'chain' of verification, at least in the sense of the term for my use case. The recommended approach from these links, drawn from the unit tests, boils down to the following when verifying server_cert: store = X509Store() store.add_cert(root_cert) store.add_cert(intermediate_cert) store_ctx = X509StoreContext(store, server_cert) We see this code makes no distinction between the root_cert and the intermediate. If we look at the documentation, add_cert itself adds a *trusted* cert (maybe add_trusted_cert would be a better name?). So this is looking like the command-line example discussed above: openssl verify -verbose -CAfile <(cat Intermediate.pem RootCert.pem) UserCert.pem How NOT to do chain verification -------------------------------- Lets try it, using the same certs used above: from OpenSSL.crypto import load_certificate, load_privatekey, FILETYPE_PEM from OpenSSL.crypto import X509Store, X509StoreContext from six import u, b, binary_type, PY3 root_cert_pem = open('geotrust_global_ca.pem').read() intermediate_cert_pem = open('sometechcompany_ist_ca_2.pem').read() intermediate_server_cert_pem = open('sometechcompany.com.pem').read() rogue_ca_cert_pem = open('rogue_ca.pem').read() fake_server_cert_from_rogue_ca_pem = open('fake_sometechcompany_ from_rogue_ca.com.pem').read() root_cert = load_certificate(FILETYPE_PEM, root_cert_pem) intermediate_cert = load_certificate(FILETYPE_PEM, intermediate_cert_pem) intermediate_server_cert = load_certificate(FILETYPE_PEM, intermediate_server_cert_pem) rogue_ca_cert = load_certificate(FILETYPE_PEM, rogue_ca_cert_pem) fake_server_cert_from_rogue_ca = load_certificate(FILETYPE_PEM, fake_server_cert_from_rogue_ca_pem) # recommended approach from the links above, prints None: store = X509Store() store.add_cert(root_cert) store.add_cert(intermediate_cert) store_ctx = X509StoreContext(store, intermediate_server_cert) print(store_ctx.verify_certificate()) # BAD - attacker substitutes a rogue CA in place of the intermediate, this prints None: store = X509Store() store.add_cert(root_cert) store.add_cert(rogue_ca_cert) store_ctx = X509StoreContext(store, fake_server_cert_from_rogue_ca) print(store_ctx.verify_certificate()) # interestingly, this causes an exception to be raised, and thus the behavior is a bit better than the 'openssl verify' antipattern shown above: store = X509Store() store.add_cert(intermediate_cert) store_ctx = X509StoreContext(store, intermediate_server_cert) print(store_ctx.verify_certificate()) Again, This makes it trivial for a malicious server to "impersonate" sometechcompany.com and bypass our verification. As we saw, the attacker creates a root cert, sign fake_sometechcompany_from_rogue_ca.com.pem with it, and sends this root as a so-called "intermediate". Unit tests ---------- For completeness, here is one of the unit tests referenced by the stackoverflow post - here, as we mentioned, add_cert adds a *trusted* intermediate certificate. This is reflected in the documentation for that function, but not clear from usage, hence the confusion above. 3644 def test_valid(self): 3645 """ 3646 :py:obj:`verify_certificate` returns ``None`` when called with a 3647 certificate and valid chain. 3648 """ 3649 store = X509Store() 3650 store.add_cert(self.root_cert) 3651 store.add_cert(self.intermediate_cert) 3652 store_ctx = X509StoreContext(store, self.intermediate_server_cert) 3653 self.assertEqual(store_ctx.verify_certificate(), None) an explicitly flawed verify_chain() ----------------------------------- The ability to verify certificate chains in pyopenssl was originally discussed in the following issue (can be found using google): https://github.com/pyca/pyopenssl/issues/154 and first contributed in the following change which explicitly uses this dangerous antipattern of trusting the chain certs: https://github.com/sholsapp/pyopenssl/blob/f8b517803ecb812a2140e45e069b20 e0ec1c0389/OpenSSL/crypto.py 2307 def verify_chain(cert, trust, chain=None): 2308 """ 2309 Verify a chain of certificates. 2310 2311 :param cert: certificate to verify 2312 :param trust: certificate to trust 2313 :param chain: additional certificates needed to verify the chain 2314 :return: None if the chain is valid, raise exception otherwise 2315 """ 2316 store = X509Store() 2317 if chain: 2318 for _cert in chain: 2319 store.add_cert(_cert) 2320 store.add_cert(trust) 2321 2322 store_ctx = _lib.X509_STORE_CTX_new() 2323 _lib.X509_STORE_CTX_init(store_ctx, store._store, cert._x509, _ffi.NULL); 2324 2325 ret = _lib.X509_verify_cert(store_ctx) 2326 if ret <= 0: 2327 _raise_current_error() These changes were ultimately abandoned and verify_certificate() was added here: https://github.com/pyca/pyopenssl/pull/155 "untrusted" certs ----------------- There seem to be some references to "untrusted" certificates in open issues: https://github.com/pyca/pyopenssl/issues/502 And these changes seem to add support for chain certificates in the sense I described in my original use case: https://github.com/pyca/pyopenssl/pull/473 It will be good to call out the earlier changes mentioned above in these issues so folks are warned against using the existing verify_certificate. * Strangly, a comment in that thread actually recommends _against_ using untrusted: "-untrusted doesn't check whether certificate chain is fully valid. Please consider to pass both intermediate and root to command as -CAfile as other questions suggests. – Envek Jun 22 at 18:06"

1 0

ssh public key processing
by Chris Hines Aug. 19, 2016

Aug. 19, 2016

Hi List, I have a question about the function cryptography.hazmat.primatives.serialization.load_ssh_public_key Basically is the function inteornded to load only the public key or is it intended that it be able to process any like out of an authorized_keys_file Source code shows that the function is prepared to strip of the key-type (eg ssh-rsa) and use it for comparison against the inner_key_type but is not prepared to strip off any options that can be passed in an authorized_keys file (For example SSH_FORCE_COMMAND or no-port-forwarding). I ask because the downstream project OpenStack Nova uses load_ssh_public_key to verify contents intended for authorized_keys is valid. Its easy enough to remove ssh options in Nova before passing to load_ssh_public_key, but I though if load_ssh_public_key already deals with the key-type header, perhaps it should also deal with the other options. I can create issues and merge requests if that is helpful, just looking for clarification on the intention (i.e. does load_ssh_public_key load contents intended for authorized_keys or just the public key part) Cheers, -- Chris

3 3

ECDSA Interoperablity with Microsoft CNG-based peer
by André Caron Aug. 18, 2016

Aug. 18, 2016

Hi all, I'm dealing with a C++ client & server pair that uses ECDSA to verify the server's identity. I'm trying to write a new Python client that will exchange with the server without making any changes to the server. I've gotten quite a bit of this in place with cryptography (the Python package :-), but I'm incapable of getting the Python client to verify the signature sent by the server and I'd like to see if you can help me out. One of the problems here is that the serialization formats seem to be internal to Microsoft's CNG API. For example, the public key is the raw output of BCryptExportKey() and the signature is the raw output of BCryptSignHash(). These are Microsoft APIs, so... needless to say cryptography doesn't "just work" with these formats. I'm pretty sure I managed to nail the key format conversion as Microsoft makes an obscure reference to the format[1], but I'm still having trouble with signatures. The blob I get as output from BCryptSignHash() has 64 bytes, but signatures for the same algorithm using cryptography are usually 70-72 bytes, so I'm confused. Cryptography's ECC signature computation clearly documents the format: "The signature is formatted as DER-encoded bytes, as specified in RFC 3279." However, Microsoft doesn't seem to output record an equivalent anywhere. They're usually pretty consistent with their APIs and storage formats, so I assume some sort of storage similar to the keys where we have two 32-byte octet streams in big endian format containing the values for R and S, but I haven't had any luck with this. I also know that the DER encoding for two integer fields will normally add 6 bytes of overhead, which gets us up to 70, but there is still the occasional extra 1 or 2 bytes, so I'm obviously missing something and may not be on the right track. [1] :https://msdn.microsoft.com/library/aa375520.aspx Anyways, I managed to extract the BCrypt* function calls from the server and client into a pair of C++ program the contain only the signing and signature verification code to reproduce the flow. The total is ~30 lines of C++ code on each side, plus ~400 lines wrappers for BCrypt* calls (resource management, error handling and links to CNG API docs). I've also written a small cryptography-based Python program that tries to mimic the C++ client and I cannot get that part to run. If anyone has a few minutes to spare to give my Python code a second pair of eyeballs, I'd really appreciate it. I've saved up all of that on this Gist: https://gist.github.com/AndreLouisCaron/ab5ee411d0722a0981feceddbf5cb3d9 The gist contents are as follows: - genkeys.py: generate a public/private key pair, write to disk in Microsoft's format; - server.cpp: load secret key, compute signature, save payload & signature to disk; - client.cpp: load public key, payload & signature from disk, verify signature; - common.h: stuff shared by client.cpp & server.cpp; - client.py: same as client.cpp, but using cryptography. I also have an alternate C++ client based on OpenSSL which might be a better source of inspiration. I'll see if I can extract pars of that too as a reference since it might be easier to map to cryptography's internals. Thanks in advance, André

3 5