Urllib: disable SSL sertificates check

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

Fix for local environment for example for such exceptions:

Traceback (most recent call last):
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/http/client.py", line 1414, in connect
    server_hostname=server_hostname)
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/home/alexey/anaconda3/envs/torch_04/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

pyopenssl

Use 

pip install pyopenssl==0.12.0

when you got such exception on Centos 5.9:

$ pip install pyopenssl
Downloading/unpacking pyopenssl
  Downloading pyOpenSSL-0.13.1.tar.gz (254kB): 254kB downloaded
  Running setup.py egg_info for package pyopenssl
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Installing collected packages: pyopenssl
  Running setup.py install for pyopenssl
    building 'OpenSSL.crypto' extension

...........

gcc -pthread -shared build/temp.linux-x86_64-2.6/OpenSSL/rand/rand.o build/temp.linux-x86_64-2.6/OpenSSL/util.o -L/usr/lib64 -lssl -lcrypto -lpython2.6 -o build/lib.linux-x86_64-2.6/OpenSSL/rand.so

building 'OpenSSL.SSL' extension

creating build/temp.linux-x86_64-2.6/OpenSSL/ssl

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/usr/kerberos/include -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c OpenSSL/ssl/connection.c -o build/temp.linux-x86_64-2.6/OpenSSL/ssl/connection.o

OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_context’:

OpenSSL/ssl/connection.c:289: warning: implicit declaration of function ‘SSL_set_SSL_CTX’

OpenSSL/ssl/connection.c: In function ‘ssl_Connection_get_servername’:

OpenSSL/ssl/connection.c:313: error: ‘TLSEXT_NAMETYPE_host_name’ undeclared (first use in this function)

OpenSSL/ssl/connection.c:313: error: (Each undeclared identifier is reported only once

OpenSSL/ssl/connection.c:313: error: for each function it appears in.)

OpenSSL/ssl/connection.c:320: warning: implicit declaration of function ‘SSL_get_servername’

OpenSSL/ssl/connection.c:320: warning: assignment makes pointer from integer without a cast

OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_tlsext_host_name’:

OpenSSL/ssl/connection.c:346: warning: implicit declaration of function ‘SSL_set_tlsext_host_name’

error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /home/.../app/bin/python2.6 -c "import setuptools;__file__='/home/.../app/build/pyopenssl/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-vBzwJt-record/install-record.txt --single-version-externally-managed --install-headers /home/.../app/include/site/python2.6 failed with error code 1 in /home/.../app/build/pyopenssl
Traceback (most recent call last):
  File "/home/.../app/bin/pip", line 8, in <module>
    load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
  File "/home/.../app/lib/python2.6/site-packages/pip/__init__.py", line 148, in main
    return command.main(args[1:], options)
  File "/home/.../app/lib/python2.6/site-packages/pip/basecommand.py", line 169, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)

Do you check HTTPS certificates in your API clients?

http://unfoldthat.com/2012/07/30/does-https-to-api-make-any-sense.html

• All browsers use a ‘certificate store’ which contains the list of trusted root CAs.
• The certificate store can either be provided by the OS, or by the browser.
• On Windows, Chrome and IE use the operating-system provided certificate store. So they have the same points of trust. However, this means that the trust list is governed by the OS vendor, not the browser. I’m not sure how often this list is updated for Windows XP, which is still used by 50% of the world’s internet users.
• On Mac, Chrome and Safari use the operating system provided store.
• On Linux, there is no operating system provided certificate store  (see /etc/ssl/certs), so each browser maintains its own certificate store, with its own set of roots. 
• Firefox, on all platforms (I believe, I might be wrong on this) uses its own certificate store, independent of the operating system store.
• Finally, on mobile devices, everyone has their own certificate store. I’d hate to guess at how many there are or how often they are updated.

From comments:

"Python2.7's standard urllib2 module does not validate server certificates. Instead, we recommend using the "requests" or "urllib3" modules in python2, or the standard http.client.HTTPSConnection class in python3, and giving them a reasonable list of CA roots" 

Настройка HTTPS для чайников

http://klen.github.com/https-setup-ru.html

Generating a self-signed SSL certificate

import os
from socket import gethostname
from OpenSSL import crypto
 
 
def generate_self_signed_cert(cert_dir, is_valid=True):
    """Generate a SSL certificate.
 
    If the cert_path and the key_path are present they will be overwritten.
    """
    if not os.path.exists(cert_dir):
        os.makedirs(cert_dir)
    cert_path = os.path.join(cert_dir, 'squid.crt')
    key_path = os.path.join(cert_dir, 'squid.key')
 
    if os.path.exists(cert_path):
        os.unlink(cert_path)
    if os.path.exists(key_path):
        os.unlink(key_path)
 
    # create a key pair
    key = crypto.PKey()
    key.generate_key(crypto.TYPE_RSA, 1024)
 
    # create a self-signed cert
    cert = crypto.X509()
    cert.get_subject().C = 'UK'
    cert.get_subject().ST = 'London'
    cert.get_subject().L = 'London'
    cert.get_subject().O = 'Canonical'
    cert.get_subject().OU = 'Ubuntu One'
    cert.get_subject().CN = gethostname() if is_valid else gethostname()[::-1]
    cert.set_serial_number(1000)
    cert.gmtime_adj_notBefore(0)
    cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60) 
    cert.set_issuer(cert.get_subject())
    cert.set_pubkey(key)
    cert.sign(key, 'sha1')
 
    with open(cert_path, 'wt') as fd: 
        fd.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
 
    with open(key_path, 'wt') as fd: 
        fd.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, key))
 
    return cert_path, key_path

From:  http://www.themacaque.com/?p=1057

Django runserver and stunnel for testing HTTPS

To get a simple tunnel setup, we typically want to follow this route:

browser ---> https://localhost:8443 ---> http://localhost:8000 ---> runserver

Create ssl sertificate

That is, the routing of all requests on localhost port 8443 to localhost port 8000, which is where our Django runserver instance is serving up our web application and static content (if any). To setup this routing, I’ve created a simple stunnel configuration file, which also provides a few other configuration niceties, like outputting all messages to stdout rather than running silently in the background. The configuration file (fake_https) is represented below:

pid=

cert = path/to/your/stunnel.pem
sslVersion = SSLv3
foreground=yes
#debug = 7
#output = /path/to/stunnel.log

[https]
accept=8443
connect=8000
TIMEOUTclose=1

Be sure to note the use of the TIMEOUTclose option. Without this set to a low timeout value, you will notice a severe lag before your browser receives a close message. To run stunnel with this configuration, simply execute the following from the command line:

stunnel fake_https

Finally, you must tell Django’s runserver to modify all incoming HTTP requests to behave as if they were over HTTPS. This tells Django to set all request objects to return True for calls to request.is_secure(). This may be accomplished by simply setting the HTTPS environment variable to a non-zero value (i.e. True) prior to executing runserver. For example:

HTTPS=1 python manage.py runserver

You may now visit https://localhost:8443 in your web browser, and you should see activity in your stunnel terminal window and in your Django runserver terminal window, indicating a successful tunneling of all local SSL traffic to your basic Django runserver.

This simple method is a great way to test your web apps locally to ensure they behave correctly under secure and unsecure scenarios, including server-side handling of secure cookies.

http://stackoverflow.com/questions/8023126/how-can-i-test-https-connections-with-django-as-easily-as-i-can-non-https-connec

How to generate a self-signed ssl certificate

openssl genrsa 1024 > stunnel.key
openssl req -new -x509 -nodes -sha1 -days 365 -key stunnel.key > stunnel.cert
cat stunnel.key stunnel.cert > stunnel.pem
dd if=/dev/urandom count=2 | openssl dhparam -rand - 512 > stunnel.pem

This generates a self-signed certificate which should be sufficient for testing purposes.
http://code.google.com/p/django-weave/wiki/HTTPSDevelopment