Linux Filesystem Hierarchy Standard

Linux distributions use the FHS: http://www.pathname.com/fhs/pub/fhs-2.3.html http://ru.wikipedia.org/wiki/FHS

You can also try man hier.

I'll try to sum up answers your questions off the top of my head, but I strongly suggest that you read through the FHS:

• /bin is for non-superuser system binaries
• /sbin is for superuser (root) system binaries
• /usr/bin & /usr/sbin are for non-critical shared non-superuser or superuser binaries, respectively
• /mnt is for temporarily mounting a partition
• /media is for mounting many removable media at once
• /dev contains your system device files; it's a long story :)
• The /usr folder, and its subfolders, can be shared with other systems, so that they will have access to the same programs/files installed in one place. Since /usr is typically on a separate filesystem, it doesn't contain binaries that are necessary to bring the system online.
• /root is separate because it may be necessary to bring the system online without mounting other directories which may be on separate partitions/hard drives/servers
• Yes, /etc stands for "et cetera". Configuration files for the local system are stored there.
• /opt is a place where you can install programs that you download/compile. That way you can keep them separate from the rest of the system, with all of the files in one place.
• /proc contains information about the kernel and running processes
• /var contains variable size files like logs, mail, webpages, etc.

To access a system, you generally don't need /var, /opt, /usr, /home; some of potentially largest directories on a system.

One of my favorites, which some people don't use, is /srv. It's for data that is being hosted via services like http/ftp/samba. I've see /var used for this a lot, which isn't really its purpose.

Test Django transaction using mocking and TransactionTestCase

It's pretty simple using mock side_effect:

class MyTransactionTestCase(TransactionTestCase):
    fixtures = ['some_fixture.json',]

    @patch.object(SomeObjectInsideTransaction, 'some_method')
    def test_disable_transaction(self, some_method):
        some_method.side_effect = IOError("Unexpected exception")

        try:
            self.client.get('/your/view/url/')
        except IOError:
            pass

        # assertions here

You have to use TransactionTestCase:

Django TestCase classes make use of database transaction facilities, if available, to speed up the process of resetting the database to a known state at the beginning of each test. A consequence of this, however, is that the effects of transaction commit and rollback cannot be tested by a Django TestCase class. If your test requires testing of such transactional behavior, you should use a Django TransactionTestCase.

Python видео и слайды с конференций

Python:
http://pyvideo.org/  http://pyvideo.ru/
EuroPython 2012 http://lanyrd.com/2012/europython/coverage/
ekbPy: http://video.yandex.by/#search?text=ekbpy http://ekbpy.ru/materialy/ (ru)
Kyiv.py http://habrahabr.ru/post/141208/KharkivPy: http://kharkivpy.org.ua/
PyCon Ukraine: http://ua.pycon.org/
PyCon, DjangoCon: http://python.mirocommunity.org/ http://pyvideo.org/ http://klewel.com/conferences/djangocon-2012/ (eng)
Pycon USA 2012: http://www.youtube.com/playlist?list=PL2814D3290BAA8837&feature=plcp http://pyvideo.org/category/17/pycon-us-2012 http://blip.tv/pycon-us-videos-2009-2010-2011
France DjangoCong: http://lanyrd.com/2012/rencontres-django/video/ (fr)
Pycon apac http://www.youtube.com/pyconapac
Pycon AU http://www.youtube.com/pyconau

Screencasts: http://godjango.com/

Summary, Notes

http://readthedocs.org/docs/pycon-2012-notes/en/latest/
http://habrahabr.ru/post/127464/
http://pydanny-event-notes.readthedocs.org/en/latest/index.html

Linux:
Linux Vacation / Eastern Europe: http://lvee.org (ru)

Misc
Я.Субботник: как Yandex использует Python и Django http://www.youtube.com/watch?v=_VyPNATTC4w

Selenium + python + jenkins

http://blog.shiningpanda.com/2012/02/selenium-python-and-jenkins-on-debian_14.html

BDD (Lattuce и Splinter)

BDD (Behaviour Driven Development) - тесты пишутся на native english т.е. в итоге их смогут писать не программисты (тестировщики, кодеры или менеджеры)

Видео: http://python.mirocommunity.org/video/5169/djangocon-2011-testing-with-le
Используется Lattuce и надстройка над Selenium - Splinter


Видео: http://www.youtube.com/watch?v=OMLDHNaUMB8
+ Freshen + Behave

M2Crypto implicit declaration of function ‘SSLv2_method’ fix

pip install -e bzr+http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/m2crypto/precise/#egg=M2Crypto

PIL with JPEG, ZLIB, FREETYPE2, LITTLECMS support in virtualenv on Ubuntu 11.10

To enable JPEG and PNG support install:

sudo apt-get install libjpeg8 libjpeg8-dev
sudo apt-get install zlib1g-dev
sudo apt-get install libfreetype6 libfreetype6-dev
sudo apt-get install liblcms1 liblcms1-dev

On x64 system symlink libraries:

sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/liblcms.so /usr/lib

cd /usr/include
ln -s freetype2 freetype

And finally in your virtualenv:

pip install PIL

You should see somethings like this during installation:

    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      linux2 2.6.7 (r267:88850, Aug 11 2011, 12:18:09)
                  [GCC 4.6.1]
    --------------------------------------------------------------------
    *** TKINTER support not available
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    --- FREETYPE2 support available
    --- LITTLECMS support available
    --------------------------------------------------------------------

To check, for example jpeg support, execute:

python -vvv -c "import _imaging" 2>&1 | grep jpeg

Should output:

#   clear[2] jpeg_encoder
#   clear[2] jpeg_decoder
#   clear[2] jpeglib_version

Django / Python memory profiling

Exapmple:

from accounts.models import UserProfile
p = UserProfile.objects.get(pk=1)

sys

import sys
sys.getsizeof(p)
# 64
sys.getrefcount(p)
# 2

gc

import gc
gc.get_referrers(p)
# [{'Branding': accounts.models.Branding ....
gc.get_objects()
# ...... many objects here

meliae
Unfortunately doesn't work in IPython

./manage.py shell --plain

from meliae import scanner
scanner.get_recursive_size(p)
# (223346, 160192339)
scanner.dump_all_objects('dump.json')
from meliae import loader
om = loader.load('dump.json')
# loaded line 188222, 188223 objs,  22.0 /  22.0 MiB read in 17.7s        
# checked   188222 /   188223 collapsed     5331    
# set parents   182895 /   182896            
# collapsed in 1.7s
om.summarize()
#Total 182896 objects, 810 types, Total size = 31.2MiB (32687019 bytes)
# Index   Count   %      Size   % Cum     Max Kind
#     0    7986   4  11340336  34  34 3146008 dict
#     1   50557  27   4265768  13  47    4648 tuple
#     2   51518  28   4236069  12  60   15347 str
#     3    2358   1   2131632   6  67     904 type
#     4     784   0   1782256   5  72   12624 module
#     5   12163   6   1459560   4  77     120 function
#     6   11453   6   1374360   4  81     120 code
#     7    5356   2   1026560   3  84    4856 list
#     8    2990   1    505996   1  86   23616 unicode
#     9     118   0    403088   1  87    3416 Options
#    10    7548   4    362304   1  88      48 datetime.datetime
#    11   11133   6    267192   0  89      24 int
#    12    2732   1    240416   0  89      88 weakref
#    13     696   0    239424   0  90     344 __proxy__
#    14     161   0    179032   0  91    1112 RelatedObject
#    15     140   0    149536   0  91    1112 ManyToOneRel
#    16     234   0    138376   0  92    3368 SortedDict
#    17    1461   0    116880   0  92      80 wrapper_descriptor
#    18     102   0    113424   0  92    1112 Distribution
#    19     119   0    107576   0  93     904 ModelBase

pip install SquareMap RunSnakeRun
runsnakemem dump.json  # !!! really long operation (wait 2-5min)

heapy

from guppy import hpy
hp = hpy()
h = hp.heap()
h
#Partition of a set of 234126 objects. Total size = 34667856 bytes.
# Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
#     0  90538  39  8491768  24   8491768  24 str
#     1  55160  24  4627144  13  13118912  38 tuple
#     2   3399   1  3018792   9  16137704  47 dict (no owner)
#     3   2267   1  2044536   6  18182240  52 type
#     4    807   0  1779624   5  19961864  58 dict of module
#     5  13148   6  1577760   5  21539624  62 function
#     6   2267   1  1512968   4  23052592  66 dict of type
#     7  12500   5  1500000   4  24552592  71 types.CodeType
#     8   5043   2  1061080   3  25613672  74 list
#     9    240   0   776832   2  26390504  76 dict of 0x18b9fd0
#<1478 more rows. Type e.g. '_.more' to view.>
h.get_rp(40)
#Reference Pattern by <[dict of] class>.
# 0: _ --- [-] 234126 (0x15262c0 | 0x1738820 | 0x1739860 | 0x173b230 | 0x173d1...
# 1: a      [-] 55170 tuple: 0x1024970*45, 0x104a480*28, 0x104a7d0*27...
# 2: aa ---- [S] 2267 type: class..., types.InstanceType, types.MethodType
# 3: ab      [S] 12500 types.CodeType: functional.py:187:__promise__...
# 4: ac ---- [S] 142 dict of module: ..management..., django, sys, types
# 5: ad      [-] 3676 function: urllib.quote..., urllib.splitnport
# 6: ada ---- [S] 324 dict of module: __main__, copy_reg, linecache, os...
# 7: adb      [S] 453 dict of type: ..WarningMessage, ..catch_warnings...
# 8: adc ---- [S] 201 dict of class: ..DictMixin, ..UserDict..., .._Environ
# 9: add      [-] 67 dict of django.db.models.base.ModelBase: 0x19c6370...
#

Pympler
Identifying memory leaks

The refbrowser module

objgraph

apt-get install graphviz graphviz-dev
pip install objgraph

import objgraph
objgraph.show_refs([p], filename='sample-graph.png')

objgraph.show_most_common_types()

objgraph.show_growth()

objgraph.show_backref([x], filename="/tmp/backrefs.png")

Linux
http://www.linuxatemyram.com/

free -m

ps aux --sort -rss

cat /proc/PID/status

pmap -x PID

Django

geordi - A Django middleware for interactive profiling

From:
http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-dude-where-s-my-ram-a-deep-dive-into-how-python-uses-memory-4896725
http://python.mirocommunity.org/video/1759/pyconau-2010-whats-eating-my-m 
http://jam-bazaar.blogspot.com/2009/11/memory-debugging-with-meliae.html

See also
https://fedorahosted.org/pulp/wiki/Debugging/MemoryLeaks 

http://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended

Chaos Python

Add this into your functional tests and smoke it.

import sys, random

def chaos_trace(frame, event, arg): 
   if event == 'line' and random.random() < 0.000001: 
       raise MemoryError()
   return chaos_trace

sys.settrace(chaos_trace)

You will get some lovely random failures injected into your code. A great way to find bugs, and make sure your reasoning is sound in the face of CHAOS!

A website returning all possible HTTP status codes and code descriptions. For test purposes.

https://github.com/IlianIliev/Status-Codes-Site/blob/master/httpstatuscodes.py