Feb 29, 2012

Generating a self-signed SSL certificate

import os
from socket import gethostname
from OpenSSL import crypto
 
 
def generate_self_signed_cert(cert_dir, is_valid=True):
    """Generate a SSL certificate.
 
    If the cert_path and the key_path are present they will be overwritten.
    """
    if not os.path.exists(cert_dir):
        os.makedirs(cert_dir)
    cert_path = os.path.join(cert_dir, 'squid.crt')
    key_path = os.path.join(cert_dir, 'squid.key')
 
    if os.path.exists(cert_path):
        os.unlink(cert_path)
    if os.path.exists(key_path):
        os.unlink(key_path)
 
    # create a key pair
    key = crypto.PKey()
    key.generate_key(crypto.TYPE_RSA, 1024)
 
    # create a self-signed cert
    cert = crypto.X509()
    cert.get_subject().C = 'UK'
    cert.get_subject().ST = 'London'
    cert.get_subject().L = 'London'
    cert.get_subject().O = 'Canonical'
    cert.get_subject().OU = 'Ubuntu One'
    cert.get_subject().CN = gethostname() if is_valid else gethostname()[::-1]
    cert.set_serial_number(1000)
    cert.gmtime_adj_notBefore(0)
    cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60) 
    cert.set_issuer(cert.get_subject())
    cert.set_pubkey(key)
    cert.sign(key, 'sha1')
 
    with open(cert_path, 'wt') as fd: 
        fd.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
 
    with open(key_path, 'wt') as fd: 
        fd.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, key))
 
    return cert_path, key_path

From:  http://www.themacaque.com/?p=1057

Monitoring with collectd

http://habrahabr.ru/blogs/python/139053/

Python Debuggers

pudb
http://adw0rd.com/2012/3/24/python-django-pudb/
http://habrahabr.ru/post/158139/

pydbgr

pdb / ipdb
http://adw0rd.com/2012/10/7/python-pdb/
# set break point in Company.get_location()
import pdb; pdb.set_trace()

# in python shell
import pdb
from accounts.models import Company
Company.objects.all()[0].get_location()
# (Pdb) self
# (Pdb) next
# (Pdb) ...
# (Pdb) next
# (Pdb) step
# (Pdb) return
# (Pdb) continue
django-extensions
./manage.py runserver_plus

Feb 23, 2012

Wireshark

You can use the specific HTTP header display filters to show either just the request headers, just the response headers or both.
tshark tcp port 80 or tcp port 443 -V -2 -R "http.request || http.response"

Run Wireshark as a Normal User on Ubuntu 11.10

Install everything you need:
sudo apt-get install wireshark tshark libcap2-bin
According to /usr/share/doc/wireshark/README.Debian you have to add your user to the group wireshark
# check group
cat /etc/group | grep wireshark

# create it if not exists
sudo groupadd wireshark

# add user to this new group
sudo usermod -a -G wireshark `whoami`
# check user groups
groups `whoami`
# yourusername : yourusergroup ..... wireshark
# or
cat /etc/group | grep wireshark
# wireshark:x:1002:yourusername

# log in to a new group
newgrp wireshark
Then lets add capture privileges http://wiki.wireshark.org/CaptureSetup/CapturePrivileges :
sudo chgrp wireshark /usr/bin/dumpcap
sudo chmod 754 /usr/bin/dumpcap
sudo setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/dumpcap

# ensure that everything ok:
getcap /usr/bin/dumpcap
# /usr/bin/dumpcap = cap_net_admin,cap_net_raw+eip
You shoud be able run tshark without sudo:
tshark -R 'ip.addr==178.159.244.20'

# amqp
tshark -i lo tcp port 5672

Feb 21, 2012

Release It! Design and Deploy Production-Ready Software - Nygard M.T.

Stability Antipatterns

Run longevity tests. It’s the only way to catch longevity bugs.

Integration Point

  • Beware this necessary evil (Every integration point will eventually fail in some way, and you need to be prepared for that failure.)
  • Prepare for the many forms of failure
  • Know when to open up abstractions (Debugging integration point failures usually requires peeling back a layer of abstraction. Failures are often difficult to debug at the application layer, because most of them violate the high-level protocols. Packet sniffers and other network diagnostics can help.)
  • Failures propagate quickly
  • Apply patterns to avert Integration Points problems (Defensive programming via Circuit Breaker, Timeouts, Decoupling Middleware, and Handshaking will all help you avoid the dangers of Integration Points.)

Chain Reactions

  • One server down jeopardizes the rest
  • Hunt for resource leaks
  • Hunt for obscure timing bugs
  • Defend with Bulkheads (Partitioning servers, with Bulkheads, can prevent Chain Reactions from taking out the entire service—though they won’t help the callers of whichever partition does go down. Use Circuit Breaker on the calling side for that.)

Cascading Failures

A cascading failure occurs when problems in one layer cause problems in callers.
  • Stop cracks from jumping the gap (A cascading failure occurs when cracks jump from one system or layer to another, usually because of insufficiently paranoid integration points. A cascading failure can also happen after a chain reaction in a lower layer. Your system surely calls out to other enterprise systems; make sure you can stay up when they go down.)
  • Scrutinize resource pools (A cascading failure often results from a resource pool, such as a connection pool, that gets exhausted when none of its calls return. The threads that get the connections block forever; all other threads get blocked waiting for connections. Safe resource pools always limit the time a thread can wait to check out a resource.)
  • Defend with Timeouts and Circuit Breaker (A cascading failure happens after something else has already gone wrong. Circuit Breaker protects your system by avoiding calls out to the troubled integration point. Using Timeouts ensures that you can come back from a call out to the troubled one.)

Users


  • Users consume memory (Each user’s session requires some memory. Minimize that memory to improve your capacity. Use a session only for caching so you can purge the session’s contents if memory gets tight.)
  • Users do weird, random things (Users in the real world do things that you won’t predict (or sometimes understand). If there’s a weak spot in your application, they’ll find it through sheer numbers. Test scripts are useful for functional testing but too predictable for stability testing. Hire a bunch of chimpanzees to hammer on keyboards for more realistic testing.)
  • Malicious users are out there
  • Users will gang up on you

Blocked Threads

  • The Blocked Threads antipattern is the proximate cause of most failures (Application failures nearly always relate to Blocked Threads in one way or another, including the ever-popular “gradual slow-down” and “hung server.” The Blocked Threads antipattern leads to Chain Reactions and Cascading Failures.)
  • Scrutinize resource pools
  • Use proven primitives (Any library of concurrency utilities has more testing than your newborn queue.)
  • Defend with Timeouts
  • Beware the code you cannot see (All manner of problems can lurk in the shadows of third-party code. Be very wary. Test it yourself.)

Attacks of Self-Denial

Good marketing can kill you at any time.
  • Keep the lines of communication open
  • Protect shared resources
  • Expect rapid redistribution of any cool or valuable offer 

Scaling Effects



  • Examine production versus QA environments to spot Scaling Effects (You get bitten by Scaling Effects when you move from small one-to-one development and test environments to full-sized production environments. Patterns that work fine in small environments or one-to-one environments might slow down or fail completely when you move to production sizes.)
  • Watch out for point-to-point communication
  • Watch out for shared resources

Unbalanced Capacities

  • Examine server and thread counts (In development and QA, your system probably looks like one or two servers, and so do all the QA versions of the other systems you call. In production, the ratio might be more like ten to one instead of one to one. Check the ratio of front-end to back-end servers, along with the number of threads each side can handle, in production compared to QA.)
  • Observe near scaling effects and users (Unbalanced Capacities is a special case of Scaling Effects: one side of a relationship scales up much more than the other side. A change in traffic patterns—seasonal, market-driven, or publicity-driven—can cause a usually benign front-end system to suddenly flood a back-end system, in much the same way as a Slashdot or Digg post causes traffic to suddenly flood websites.)
  • Stress both sides of the interface (If you provide the back-end system, see what happens if it suddenly gets ten times the highest ever demand, hitting the most expensive transaction. Does it fail completely? Does it slow down and recover? If you provide the front-end system, see what happens if calls to the back end stop responding or get very slow.)

Slow Responses

  • Slow Responses triggers Cascading Failures
  • For websites, Slow Responses causes more traffic (Users waiting for pages frequently hit the Reload button, generating even more traffic to your already overloaded system.)
  • Consider Fail Fast
  • Hunt for memory leaks or resource contention

SLA Inversion

  • Don’t make empty promises
  • Examine every dependency
  • Decouple your SLAs

Unbounded Result Sets

  • Use realistic data volumes
  • Don’t rely on the data producers
  • Put limits into other application-level protocols

Stability Patterns


Use Timeouts

  • Apply to Integration Points, Blocked Threads, and Slow Responses
  • Apply to recover from unexpected failures
  • Consider delayed retries

Circuit Breaker



  • Don’t do it if it hurts
  • Use together with Timeouts
  • Expose, track, and report state changes

Bulkheads (Переборки)


  • Save part of the ship
  • Decide whether to accept less efficient use of resources
  • Pick a useful granularity
  • Very important with shared services models

Steady State (Установившееся состояние)

  • Avoid fiddling (Human intervention leads to problems. Eliminate the need for recurring human intervention. Your system should run at least for a typical deployment cycle without manual disk cleanups or nightly restarts.)
  • Purge data with application logic
  • Limit caching
  • Roll the logs
Fail Fast
  • Avoid Slow Responses and Fail Fast
  • Reserve resources, verify Integration Points early
  • Use for input validation

Handshaking (Рукопожатие)

  • Create cooperative demand control
  • Consider health checks
  • Build Handshaking into your own low-level protocols

Test Harness

  • Emulate out-of-spec failures
  • Stress the caller
  • Leverage shared harnesses for common failures
  • Supplement, don’t replace, other testing methods

Decoupling Middleware

  • Decide at the last responsible moment
  • Avoid many failure modes through total decoupling
  • Learn many architectures, and choose among them

Capacity Antipatterns

Horizontal scaling

Vertical scaling


Resource Pool Contention

  • Eliminate contention under normal loads
  • If possible, size resource pools to the request thread pool
  • Prevent vicious cycles
  • Watch for the Blocked Threads pattern

Excessive JSP Fragments

  • Don’t use code for content (Loading JSP classes into memory is a kind of caching.)

AJAX Overkill

  • Avoid needless requests
  • Respect your session architecture
  • Minimize the size of replies
  • Increase the size of your web tier

Overstaying Sessions

  • Curtail session retention
  • Remember that users don’t understand sessions
  • Keep keys, not whole objects

Wasted Space in HTML

The Reload Button

Fast sites don’t provoke the user into hitting the Reload button.

Handcrafted SQL

Minimize handcrafted SQL

Database Eutrophication

  • Create indexes
  • Purge sludge
  • Keep reports out of production

Integration Point Latency

Integration point latency is like the house advantage in blackjack. The more often you play, the more often it works against you. Avoid chatty remote protocols. They take longer to execute, and they tie up those precious request-handling threads.

Cookie Monsters

Use cookies for identifiers, not entire objects.

Capacity Patterns

  • Pool Connections 
  • Use Caching Carefully 
  • Precompute Content 
  • Tune the Garbage Collector

Transparency

Adaptation

Versioning API

Feb 20, 2012

Online Python Tutor, IDE & Debugging Tool

http://people.csail.mit.edu/pgbovine/python/ - learn Python by writing code and visualizing execution
ideone.com - online
repl.it - online, with source code

Feb 18, 2012

Linux Filesystem Hierarchy Standard


Linux distributions use the FHS: http://www.pathname.com/fhs/pub/fhs-2.3.html http://ru.wikipedia.org/wiki/FHS

You can also try man hier.

I'll try to sum up answers your questions off the top of my head, but I strongly suggest that you read through the FHS:
  • /bin is for non-superuser system binaries
  • /sbin is for superuser (root) system binaries
  • /usr/bin & /usr/sbin are for non-critical shared non-superuser or superuser binaries, respectively
  • /mnt is for temporarily mounting a partition
  • /media is for mounting many removable media at once
  • /dev contains your system device files; it's a long story :)
  • The /usr folder, and its subfolders, can be shared with other systems, so that they will have access to the same programs/files installed in one place. Since /usr is typically on a separate filesystem, it doesn't contain binaries that are necessary to bring the system online.
  • /root is separate because it may be necessary to bring the system online without mounting other directories which may be on separate partitions/hard drives/servers
  • Yes, /etc stands for "et cetera". Configuration files for the local system are stored there.
  • /opt is a place where you can install programs that you download/compile. That way you can keep them separate from the rest of the system, with all of the files in one place.
  • /proc contains information about the kernel and running processes
  • /var contains variable size files like logs, mail, webpages, etc.
To access a system, you generally don't need /var, /opt, /usr, /home; some of potentially largest directories on a system.
One of my favorites, which some people don't use, is /srv. It's for data that is being hosted via services like http/ftp/samba. I've see /var used for this a lot, which isn't really its purpose.

Feb 17, 2012

Test Django transaction using mocking and TransactionTestCase

It's pretty simple using mock side_effect:
class MyTransactionTestCase(TransactionTestCase):
    fixtures = ['some_fixture.json',]

    @patch.object(SomeObjectInsideTransaction, 'some_method')
    def test_disable_transaction(self, some_method):
        some_method.side_effect = IOError("Unexpected exception")

        try:
            self.client.get('/your/view/url/')
        except IOError:
            pass

        # assertions here
You have to use TransactionTestCase:
Django TestCase classes make use of database transaction facilities, if available, to speed up the process of resetting the database to a known state at the beginning of each test. A consequence of this, however, is that the effects of transaction commit and rollback cannot be tested by a Django TestCase class. If your test requires testing of such transactional behavior, you should use a Django TransactionTestCase.

Feb 14, 2012

Python видео и слайды с конференций

Python:
http://pyvideo.org/  http://pyvideo.ru/
EuroPython 2012 http://lanyrd.com/2012/europython/coverage/
ekbPy: http://video.yandex.by/#search?text=ekbpy http://ekbpy.ru/materialy/ (ru)
Kyiv.py http://habrahabr.ru/post/141208/KharkivPy: http://kharkivpy.org.ua/
PyCon Ukraine: http://ua.pycon.org/
PyCon, DjangoCon: http://python.mirocommunity.org/ http://pyvideo.org/ http://klewel.com/conferences/djangocon-2012/ (eng)
Pycon USA 2012: http://www.youtube.com/playlist?list=PL2814D3290BAA8837&feature=plcp http://pyvideo.org/category/17/pycon-us-2012 http://blip.tv/pycon-us-videos-2009-2010-2011
France DjangoCong: http://lanyrd.com/2012/rencontres-django/video/ (fr)
Pycon apac http://www.youtube.com/pyconapac
Pycon AU http://www.youtube.com/pyconau

Screencasts: http://godjango.com/

Summary, Notes

Selenium + python + jenkins

http://blog.shiningpanda.com/2012/02/selenium-python-and-jenkins-on-debian_14.html

BDD (Lattuce и Splinter)

BDD (Behaviour Driven Development) - тесты пишутся на native english т.е. в итоге их смогут писать не программисты (тестировщики, кодеры или менеджеры)

Видео: http://python.mirocommunity.org/video/5169/djangocon-2011-testing-with-le
Используется Lattuce и надстройка над Selenium - Splinter


Видео: http://www.youtube.com/watch?v=OMLDHNaUMB8
Freshen + Behave

Feb 9, 2012

M2Crypto implicit declaration of function ‘SSLv2_method’ fix

pip install -e bzr+http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/m2crypto/precise/#egg=M2Crypto

Feb 3, 2012

PIL with JPEG, ZLIB, FREETYPE2, LITTLECMS support in virtualenv on Ubuntu 11.10

To enable JPEG and PNG support install:
sudo apt-get install libjpeg8 libjpeg8-dev
sudo apt-get install zlib1g-dev
sudo apt-get install libfreetype6 libfreetype6-dev
sudo apt-get install liblcms1 liblcms1-dev
On x64 system symlink libraries:
sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/liblcms.so /usr/lib
cd /usr/include
ln -s freetype2 freetype
And finally in your virtualenv:
pip install PIL
You should see somethings like this during installation:
    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      linux2 2.6.7 (r267:88850, Aug 11 2011, 12:18:09)
                  [GCC 4.6.1]
    --------------------------------------------------------------------
    *** TKINTER support not available
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    --- FREETYPE2 support available
    --- LITTLECMS support available
    --------------------------------------------------------------------

To check, for example jpeg support, execute:
python -vvv -c "import _imaging" 2>&1 | grep jpeg
Should output:
#   clear[2] jpeg_encoder
#   clear[2] jpeg_decoder
#   clear[2] jpeglib_version

Feb 2, 2012

Django / Python memory profiling

Exapmple:
from accounts.models import UserProfile
p = UserProfile.objects.get(pk=1)
sys
import sys
sys.getsizeof(p)
# 64
sys.getrefcount(p)
# 2
gc
import gc
gc.get_referrers(p)
# [{'Branding': accounts.models.Branding ....
gc.get_objects()
# ...... many objects here
meliae
Unfortunately doesn't work in IPython
./manage.py shell --plain
from meliae import scanner
scanner.get_recursive_size(p)
# (223346, 160192339)
scanner.dump_all_objects('dump.json')
from meliae import loader
om = loader.load('dump.json')
# loaded line 188222, 188223 objs,  22.0 /  22.0 MiB read in 17.7s        
# checked   188222 /   188223 collapsed     5331    
# set parents   182895 /   182896            
# collapsed in 1.7s
om.summarize()
#Total 182896 objects, 810 types, Total size = 31.2MiB (32687019 bytes)
# Index   Count   %      Size   % Cum     Max Kind
#     0    7986   4  11340336  34  34 3146008 dict
#     1   50557  27   4265768  13  47    4648 tuple
#     2   51518  28   4236069  12  60   15347 str
#     3    2358   1   2131632   6  67     904 type
#     4     784   0   1782256   5  72   12624 module
#     5   12163   6   1459560   4  77     120 function
#     6   11453   6   1374360   4  81     120 code
#     7    5356   2   1026560   3  84    4856 list
#     8    2990   1    505996   1  86   23616 unicode
#     9     118   0    403088   1  87    3416 Options
#    10    7548   4    362304   1  88      48 datetime.datetime
#    11   11133   6    267192   0  89      24 int
#    12    2732   1    240416   0  89      88 weakref
#    13     696   0    239424   0  90     344 __proxy__
#    14     161   0    179032   0  91    1112 RelatedObject
#    15     140   0    149536   0  91    1112 ManyToOneRel
#    16     234   0    138376   0  92    3368 SortedDict
#    17    1461   0    116880   0  92      80 wrapper_descriptor
#    18     102   0    113424   0  92    1112 Distribution
#    19     119   0    107576   0  93     904 ModelBase
pip install SquareMap RunSnakeRun
runsnakemem dump.json  # !!! really long operation (wait 2-5min)

heapy
from guppy import hpy
hp = hpy()
h = hp.heap()
h
#Partition of a set of 234126 objects. Total size = 34667856 bytes.
# Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
#     0  90538  39  8491768  24   8491768  24 str
#     1  55160  24  4627144  13  13118912  38 tuple
#     2   3399   1  3018792   9  16137704  47 dict (no owner)
#     3   2267   1  2044536   6  18182240  52 type
#     4    807   0  1779624   5  19961864  58 dict of module
#     5  13148   6  1577760   5  21539624  62 function
#     6   2267   1  1512968   4  23052592  66 dict of type
#     7  12500   5  1500000   4  24552592  71 types.CodeType
#     8   5043   2  1061080   3  25613672  74 list
#     9    240   0   776832   2  26390504  76 dict of 0x18b9fd0
#<1478 more rows. Type e.g. '_.more' to view.>
h.get_rp(40)
#Reference Pattern by <[dict of] class>.
# 0: _ --- [-] 234126 (0x15262c0 | 0x1738820 | 0x1739860 | 0x173b230 | 0x173d1...
# 1: a      [-] 55170 tuple: 0x1024970*45, 0x104a480*28, 0x104a7d0*27...
# 2: aa ---- [S] 2267 type: class..., types.InstanceType, types.MethodType
# 3: ab      [S] 12500 types.CodeType: functional.py:187:__promise__...
# 4: ac ---- [S] 142 dict of module: ..management..., django, sys, types
# 5: ad      [-] 3676 function: urllib.quote..., urllib.splitnport
# 6: ada ---- [S] 324 dict of module: __main__, copy_reg, linecache, os...
# 7: adb      [S] 453 dict of type: ..WarningMessage, ..catch_warnings...
# 8: adc ---- [S] 201 dict of class: ..DictMixin, ..UserDict..., .._Environ
# 9: add      [-] 67 dict of django.db.models.base.ModelBase: 0x19c6370...
#
Pympler
Identifying memory leaks

The refbrowser module


objgraph
apt-get install graphviz graphviz-dev
pip install objgraph
import objgraph
objgraph.show_refs([p], filename='sample-graph.png')
objgraph.show_most_common_types()
objgraph.show_growth()
objgraph.show_backref([x], filename="/tmp/backrefs.png")
Linux
http://www.linuxatemyram.com/
free -m
ps aux --sort -rss
cat /proc/PID/status
pmap -x PID
Django
geordi - A Django middleware for interactive profiling

From:
http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-dude-where-s-my-ram-a-deep-dive-into-how-python-uses-memory-4896725
http://python.mirocommunity.org/video/1759/pyconau-2010-whats-eating-my-m 
http://jam-bazaar.blogspot.com/2009/11/memory-debugging-with-meliae.html

See also
https://fedorahosted.org/pulp/wiki/Debugging/MemoryLeaks 

http://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended

Chaos Python

Add this into your functional tests and smoke it.
import sys, random

def chaos_trace(frame, event, arg): 
   if event == 'line' and random.random() < 0.000001: 
       raise MemoryError()
   return chaos_trace

sys.settrace(chaos_trace)
You will get some lovely random failures injected into your code. A great way to find bugs, and make sure your reasoning is sound in the face of CHAOS!

A website returning all possible HTTP status codes and code descriptions. For test purposes.

https://github.com/IlianIliev/Status-Codes-Site/blob/master/httpstatuscodes.py