User mode Linux

User mode Linux (UML, Линукс пользовательского режима) — вариант ядра Linux, который позволяет запустить несколько виртуализованных Линукс-систем (guests) в качестве обычных приложений в основной Linux‐системе (host).

http://uml.devloop.org.uk/howto.html
http://uml.devloop.org.uk/faq.html
http://user-mode-linux.sourceforge.net/index.html

Запуск

Скачать ядро, скомпилировать:

make defconfig ARCH=um
make ARCH=um

Скачать root_fs, например Debian-Wheezy

Запустить:

host% mkdir /tmp/uml
host% TMPDIR=/tmp/uml ./linux ubda=Debian-Wheezy-AMD64-root_fs mem=1024m

Остановить:

guest% halt

Настройка сети

http://user-mode-linux.sourceforge.net/network.html

host% sudo aptitude install uml-utilities

Добавить пользователя в группу uml-net:

host% sudo vi /etc/group
newgrp uml-net  # login to group

hot-plug it into a running UML:
pCtikQ - umlid, можно найти в папке ~/.uml
uml_console
tuntap - TUN and TAP are virtual-network kernel devices. Being network devices supported entirely in software, they differ from ordinary network devices which are backed up by hardware network adapters

host% uml_mconsole pCtikQ config eth0=tuntap,,,192.168.0.254
# or do it via bash command
host% uml_mconsole `ls -1t ~/.uml | head -n1` config eth0=tuntap,,,192.168.0.254

Enable the device within UML:

guest% ifconfig eth0 192.168.0.253 up
guest% route add default gw 192.168.0.254

Set up NAT on the host in order to communicate with the outside Internet:

host% sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Запуск в gdb

http://web2.clarkson.edu/class/cs644/kernel/setup/uml/gdb_uml.html

host% TMPDIR=/tmp/uml gdb ./linux
(gdb) handle SIGSEGV pass nostop noprint
(gdb) handle SIGUSR1 pass nostop noprint
(gdb) break netlink_create
(gdb) run ubda=Debian-Wheezy-AMD64-root_fs mem=512m

handle SIGSEGV pass nostop noprint
handle SIGUSR1 pass nostop noprint
r ubda=Debian-Wheezy-AMD64-root_fs mem=512m

Tips on debugging optimized code

The Linux kernel and all kernel modules are always built with optimization on. This is not just for performance reasons: the kernel code requires and assumes that it is optimized and does not function otherwise.

Tips on debugging optimized code

Как работают рекомендательные системы. Лекция в Яндексе

http://habrahabr.ru/company/yandex/blog/241455/

mWatch

Docker

http://habrahabr.ru/company/infobox/blog/240623/
https://blog.docker.com/2014/10/docker-1-3-signed-images-process-injection-security-options-mac-shared-directories/
http://habrahabr.ru/post/247547/

OpenStack

http://www.opennet.ru/opennews/art.shtml?num=40851

Physical Web

Physical Web is a Google's Internet of Things (IoT) via simple web-based API. Every object broadcasts its URL periodically via Bluetooth 4.0 Low Energy (BLE). There is both an Android and an iOS app available for download (both of which are open source) that will monitor for broadcast URL advertisement packets and present them to the user. Hardware hackers can evidently build their own smart objects using the RFduino, a diminutive microcontroller that runs on battery power and is Arduino-compatible.

http://lwn.net/SubscriberLink/615050/0edeaf985ad1289c/

TLP утилита для управления питанием

linrunner.de/en/tlp/docs/tlp-linux-advanced-power-management.html

So sad

Python Is Slow Make It Faster With C

http://bbit.co.nz/blog/2014/09/12/python-slow-make-it-faster-c/
https://github.com/beneboy/py-c-integration-example

Таблица математических символов

$\forall$	для всех
$\exists$	существует
$\{x: P(x)\}$	множество таких $x$ для которых $P(x)$ верно
$\{P\} A \{Q\}$	пред- и постусловия алгоритма $A$
$a \in S$	$a$ элемент множества $S$
$a \notin S$
$\emptyset$	пустое множество
$\mathbb{N}$	множество натуральных чисел
$\mathbb{Z}$	множество целых чисел
$\mathbb{Q}$	множество рациональных чисел
$\mathbb{R}$	множество вещественных чисел
$\equiv$
$\oplus$
$\cap$
$\cup$
$\supset$
$\subset$

Хаггарти Р. - Дискретная математика для программистов

Википедия
Wikipedia

systemd

Python Arsenal for Reverse Engineering

http://pythonarsenal.erpscan.com/

Share Ubuntu desktop to Windows

http://www.mikogo.com

Ты стал таким замкнутым...

Interective Python

OnlinePythonTutor

https://github.com/pgbovine/OnlinePythonTutor/blob/master/v3/docs/embedding-HOWTO.md

Skulpt - Python JavaScript implementation

Clewn

Clewn provides Gdb support within Vim - Clewn is a standalone program controlling Vim through a NetBeans interface.

List of default key mappings:

        CTRL-Z  send an interrupt to GDB and the program it is running
        B       info breakpoints
        L       info locals
        A       info args
        S       step
        I       stepi
        CTRL-N  next: next source line, skipping all function calls
        X       nexti
        F       finish
        R       run
        Q       quit
        C       continue
        W       where
        CTRL-U  up: go up one frame
        CTRL-D  down: go down one frame

cursor position: ~
        CTRL-B  set a breakpoint on the line where the cursor is located
        CTRL-E  clear all breakpoints on the line where the cursor is located

mouse pointer position: ~
        CTRL-P  print the value of the variable defined by the mouse pointer
                position
        CTRL-X  print the value that is referenced by the address whose
                value is that of the variable defined by the mouse pointer
                position
        CTRL-K  set a breakpoint at assembly address shown by mouse position
        CTRL-H  clear a breakpoint at assembly address shown by mouse position
        CTRL-J  add the selected variable at mouse position to the watched
                variables window

Note: You can tune Vim behavior when using netBeans keys with the mouse
pointer position, by setting Vim balloondelay to a smaller value than its
default. The following setting has been reported to give better results:

    :set balloondelay=100 ~

Note: After browsing your code, you can always return rapidly to the line of
code where GDB is currently stopped, by using CTRL-U and CTRL-D.

изобретатель шахмат запросил у правителя в качестве награды одно зерно на первое поле шахматной доски, два на второе и т.д., т.е. вдвое больше на каждое следующее. когда правитель понял что ему придется раскошелиться на 2 в 64й степени зерен пшеницы, он приказал отрубить голову изобретателю и таким образом стал первым, кто применил метод отсечения в качестве меры контроля комбинаторного взрыва : )

Data structures

Python internals

Improving Python's Memory Allocator
Stepping Through CPython
Why Python is Slow: Looking Under the Hood
Python’s Innards

Eli Bendersky's - Python internals
Питон изнутри. Введение (Python’s Innards)


http://pythoninternal.wordpress.com/

Видео с Minsk Python Meetup

Read auxv

LD_SHOW_AUXV=1 /bin/true

or

Elfutils

https://access.redhat.com/site/documentation/en-US/Red_Hat_Developer_Toolset/1/html/User_Guide/chap-elfutils.html

sudo apt-get install elfutils
eu-readelf --all /lib/x86_64-linux-gnu/libc.so.6

gcc dumpspecs

gcc -dumpspecs
# ...
# *startfile:
# %{!mandroid|tno-android-ld:%{!shared: %{pg|p|profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s}}    crti.o%s # %{static:crtbeginT.o%s;shared|pie:crtbeginS.o%s;:crtbegin.o%s};:%{shared: crtbegin_so%O%s;:  %{static: crtbegin_static%O%s;: crtbegin_dynamic%O%s}}}
# ...

http://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/Spec-Files.html#Spec-Files

ld --verbose
#...
# OUTPUT_ARCH(i386:x86-64)
# ENTRY(_start)
# SEARCH_DIR("/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib/x86_64-linux-gnu"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); # SEARCH_DIR("=/usr/lib");
SECTIONS
# ...

https://sourceware.org/binutils/docs/ld/

Core dump

Allow core dump
Permanent:

sudo vi /etc/security/limits.conf
#add line
#YOUR_USERNAME     soft    core            1000000

Temporary:

ulimit -c unlimited

Core dump

sleep 30
# Type Control-\
Quit (core dumped)

kill -ABRT <PID>
# 11602 abort (core dumped)

In C

abort()

Core dump misc

cat /proc/sys/kernel/core_pattern
man 5 core
cat /proc/PID/coredump_filter

Software architecture design principles

GRASP - General Responsibility Assignment Software Patterns (or Principles) consists of guidelines for assigning responsibility to classes and objects in object-oriented design. The different patterns and principles used in GRASP are: Controller, Creator, Indirection, Information Expert, High Cohesion, Low Coupling, Polymorphism, Protected Variations, and Pure Fabrication.
SOLID (ru)
KISS Keep it simple, stupid
DRY Don't repeat yourself
YAGNI You aren't gonna need it
Worse is better
Law of Demeter, LoD Each unit should have only limited knowledge about other units: only units "closely" related to the current unit. Each unit should only talk to its friends; don't talk to strangers. Only talk to your immediate friends.

localStorageWrapper.js

Kgdb, Kdb

Using kgdb, kdb and the kernel debugger internals

Kgdb is a debug stub that allows another machine to connect a source-level debugger over a serial link.

kdb reuses kgdb's breakpoint and polled I/O infrastructure in order to implement a machine-level kernel debugger that runs entirely on the machine being debugged.

Java installed

Programmer

Kernel. Misc

# Options
sudo sysctl -a
less /usr/include/linux/sysctl.h

ulimit -a # Sets or reports user resource limits.

cat /proc/sys/kernel/ostype
cat /proc/sys/kernel/osrelease

ls /lib/modules/$(uname -r)/kernel/fs
List open files
lsof -p

File
stat text.txt
file arch.tar.xz
fuser -v server.sock

Inode
ls -i
df -i

Port
fuser -v -n tcp 11100

Interrupts

Timing

Page and buffer cache fadvise, madvise, readahead

pdflush

TCP: an approximation to the real state diagram

https://drive.google.com/file/d/0B5DpX7DPuSN1TklhLVVsMnVCTXM/edit?usp=sharing

Resources that have influenced me as a C programmer

From http://www.reddit.com/r/C_Programming/comments/249cmt/resources_that_have_influenced_me_as_a_c/

http://stackoverflow.com/questions/tagged/c

The GNU C Library Reference. Also commonly available as 'info libc'. It should be read like a book. Really.

The Kernel CodingStyle, has some pretty good advice.

What every programmer should know about memory is very relevant for high-speed processing. This is the LWN article.

Kernel design patterns. Another great LWN link. I always smile a little whenever I use the container_of() macro.

Recursive Make Considered Harmful. Makes a fine beverage when mixed with Autotools.

Gnulib - the GNU Portability Library. Add in small quantities.

GNU C Language Extensions. I try to stay pretty compiler-agnostic, but there are some nifty things here. I would have never known about zero-length arrays if not for this.

Linux Device Drivers, 3rd edition.. Definitely worth reading.

The Linux Programming Interface. Likewise.

K&R Second Edition. Almost forgot this one!

GNU Screen. Nothing specific to C programming, but this utility deserves mention.

More than anything, man pages, 'info libc', reading header files, etc. Ways to look things up without first turning to Google, which often links to low signal-to-noise resources.

click is a Python package for creating beautiful command line interfaces

click is a Python package for creating beautiful command line interfaces in a composable way with as little amount of code as necessary. It’s the “Command Line Interface Creation Kit”.

import click

@click.command()
@click.option('--count', default=1, help='number of greetings')
@click.option('--name', prompt='Your name',
              help='the person to greet', required=True)
def hello(count, name):
    for x in range(count):
        print('Hello %s!' % name)

if __name__ == '__main__':
    hello()

python click_test.py --help
Usage: click_test.py [OPTIONS] NAME

Options:
  --count=COUNT  number of greetings
  --help         Show this message and exit.

The Linux Graphics Stack

The Direct Rendering Manager (DRM) is a device-independent kernel-level device driver that provides support for the Direct Rendering Infrastructure.

Direct Rendering Infrastructure (DRI) is an interface and a free software implementation inside the kernel used by the X Window System/Wayland to securely allow user applications to access the video hardware without requiring data to be passed through the display server. Its primary application is to provide hardware acceleration for the Mesa implementation of OpenGL, which is the core of the DRI OpenGL drivers. Without DRI, programs have to use the CPU while rendering (indirect rendering), which degrades overall performance. DRI has also been adapted to provide OpenGL acceleration on a framebuffer console without a display server running.

Mode setting (KMS) is a software operation that activates a display mode (screen resolution, color depth, and refresh rate) for a computer's display adapter.

Wayland is a protocol that specifies the communication between the display server (called Wayland compositor) and its clients. It was initially designed as a replacement for the X Window System. In stark contrast to the latter, Wayland clients will render without detour directly into their own buffer located in the graphics memory, through the use of EGL with some additional Wayland-specific extensions to EGL.
The display manager is to do the compositing, hence it will incorporate a big chunk of the functionality of current-day compositing window managers. It will composite those buffers to form the on-screen display of application windows. The Wayland protocol is essentially only about input handling and buffer management. The handling of the input hardware relies on evdev in Linux, and similar components in other operating systems.

EGL is an interface between Khronos rendering APIs (such as OpenGL, OpenGL ES or OpenVG) and the underlying native platform windowing system. EGL handles graphics context management, surface/buffer binding, rendering synchronization, and enables "high-performance, accelerated, mixed-mode 2D and 3D rendering using other Khronos APIs." EGL is managed by the non-profit technology consortium Khronos Group.

evdev (for event device) is a component of the Linux kernel for handling input (from keyboards, mice, joysticks, etc.) and a closely related input driver for both the X.Org Server and Wayland compositors. The kernel component is glue code which translates input events from peripheral-specific drivers into a generic structure which the input driver can easily translate into X11 events. Thus every input device with a Linux driver is compatible with the X.Org input driver, making X.Org much easier to configure.

Mesa is a collection of free and open-source libraries that implement OpenGL and several other APIs related to hardware-accelerated 3D rendering, 3D computer graphics and GPGPU.

Elbrus-4S

Russia is ready to produce new processor Elbrus-4S based on own microprocessors architecture with operations dependency analysis and order optimization via compiler. Also they have Linux based OS with modified 2.6.33 kernel that supports hard real-time scheduling

News

MCST News

Architecture (ru)
Elbrus OS (ru)

Full stack developer

Full-stack разработчики
The Full Stack Part I 
What is a Full Stack developer?

Shared libraries and dynamic linker

cc -g -c prog.c mod1.c mod2.c mod3.c
cc -g -o prog_nolib prog.o mod1.o mod2.o mod3.o

strip(1) - discard symbols from object files.

Static lib:

ar r libdemo.a mod1.o mod2.o mod3.o

ar tv libdemo.a
# rw-rw-r-- 1000/1000 1003352 Apr 14 11:24 2014 mod1.o
# rw-rw-r-- 1000/1000 403352 Apr 14 11:24 2014 mod2.o
# rw-rw-r-- 1000/1000  43352 Apr 14 11:24 2014 mod3.o

cc -g -o prog prog.o libdemo.a
# or
cc -g -o prog prog.o -Lmylibdir -ldemo

./prog

Dinamic lib:

gcc -g -c -fPIC -Wall mod1.c mod2.c mod3.c
gcc -g -shared -o libfoo.so mod1.o mod2.o mod3.o
# or 
gcc -g -fPIC -Wall mod1.c mod2.c mod3.c -shared -o libfoo.so

gcc -g -Wall -o prog prog.c libfoo.so
LD_LIBRARY_PATH=. ./prog

nm(1) - list symbols from object files
readelf(1) - displays information about ELF files
objdump(1) - display information from object files
ldd(1) - print shared library dependencies
ldconfig(1) - configure dynamic linker run-time bindings

nm mod1.o
readelf -s mod1.o

objdump --all-headers libfoo.so
readelf -d libfoo.so

Process shared libs:

cat /proc/PID/maps

Shared libs that prog required:

ldd /usr/sbin/tcpdump
# linux-vdso.so.1 =>  (0x00007fffe17ff000)
# libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007ff285828000)
# libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007ff2855ed000)
# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff285224000)
# libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff285020000)
# libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ff284e07000)
# /lib64/ld-linux-x86-64.so.2 (0x00007ff28606f000)

Where function defined:

nm -A /usr/lib/lib*.so 2> /dev/null | grep ' crypt$'

All system shared libs:

ldconfig –p
#1768 libs found in cache `/etc/ld.so.cache'
# libzvbi.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzvbi.so.0
# libzvbi-chains.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzvbi-chains.so.0
# libzltext.so.0.13 (libc6,x86-64) => /usr/lib/libzltext.so.0.13
# libzlcore.so.0.13 (libc6,x86-64) => /usr/lib/libzlcore.so.0.13
# libzip.so.2 (libc6,x86-64) => /usr/lib/libzip.so.2
#...

Path: LD_LIBRARY_PATH, DT_RPATH, DT_RUNPATH, $ORIGIN

gcc -Wl,-rpath,'$ORIGIN'/lib ...

Run-time symbol resolution:

gcc -g -c -fPIC -Wall -c foo.c
gcc -g -shared -o libfoo.so foo.o
gcc -g -o prog prog.c libfoo.so
LD_LIBRARY_PATH=. ./prog
# main-xyz

Monitoring dynamic linker:

LD_DEBUG=help ./prog
LD_DEBUG=all ./prog

Two types of people

Unix as IDE. Memmory profiling

cat /proc/meminfo
#MemTotal:       16350248 kB
#MemFree:        14502016 kB
#Buffers:          133516 kB
#Cached:           738584 kB
#..
#Mapped:           106640 kB
#Shmem:             78664 kB
#Slab:             101588 kB
#SReclaimable:      81208 kB
#SUnreclaim:        20380 kB
#KernelStack:        3000 kB
#PageTables:        18616 kB
#..

cat /proc/iomem 
#00000000-00000fff : reserved
#00001000-0009d7ff : System RAM
#0009d800-0009ffff : reserved
#000a0000-000bffff : PCI Bus 0000:00
#000c0000-000c7fff : Video ROM
#...

cat /proc/pagetypeinfo
#Page block order: 9
#Pages per block:  512
#
# ...
#
#Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
#Node 0, zone      DMA            1            0            6            1            0 
#Node 0, zone    DMA32            8           32         1742            2            0 
#Node 0, zone   Normal          222           94         6069            2            0

cat /proc/buddyinfo 
#Node 0, zone      DMA      0      0      0      0      0      0      0      0      1      1      3 
#Node 0, zone    DMA32   1734   1323    616    162     59     32     10     13      7      3    749 
#Node 0, zone   Normal   2712   3081   1760    654    320    144     51     27     27     12   2728

sudo cat /proc/slabinfo
sudo slabtop

Shared Memory Segments

ipcs -m
#------ Shared Memory Segments --------
#key        shmid      owner      perms      bytes      nattch     status      
#0x00000000 0          alex       600        393216     2          dest         
#0x00000000 360449     alex       600        4194304    2          dest         
#0x00000000 196610     alex       600        2097152    2          dest         
#0x00000000 229379     alex       600        1048576    2          dest

ipcs -m -p
#
#------ Shared Memory Creator/Last-op PIDs --------
#shmid      owner      cpid       lpid      
#0          alex       3613       4418      
#360449     alex       3993       4418      
#196610     alex       3317       4418      
#229379     alex       3613       4418

Qemu. Mips cpu virtualization

About MIPS architecture

Download initrd.gz and vmlinux-3.2.0-4-4kc-malta from http://ftp.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/

# install
qemu-system-mips -M malta -kernel vmlinux-3.2.0-4-4kc-malta -initrd initrd.gz \
 -hda debian74_mips.img  -append "root=/dev/sda1 console=ttyS0" -nographic

# boot
qemu-system-mips -M malta -kernel vmlinux-3.2.0-4-4kc-malta \
 -hda debian74_mips.img -append "root=/dev/sda1 console=ttyS0" -nographic

On guest host:

uname -a
# Linux debianmips 3.2.0-4-4kc-malta #1 Debian 3.2.54-2 mips GNU/Linux

cat /proc/cpuinfo 
# system type  : MIPS Malta
# processor  : 0
# cpu model  : MIPS 24Kc V0.0  FPU V0.0

mips hello world (http://en.wikipedia.org/wiki/MIPS_architecture#MIPS_assembly_language):

00400640 <main>:
#include <stdio.h>

int main() {
  400640: 27bdffe0  addiu sp,sp,-32
  400644: afbf001c  sw ra,28(sp)
  400648: afbe0018  sw s8,24(sp)
  40064c: 03a0f021  move s8,sp
    printf("Hello world!\n");
  400650: 3c020040  lui v0,0x40
  400654: 24440810  addiu a0,v0,2064
  400658: 0c100140  jal 400500 <puts@plt>
  40065c: 00200825  move at,at
    return 0;
  400660: 00001021  move v0,zero
}
  400664: 03c0e821  move sp,s8
  400668: 8fbf001c  lw ra,28(sp)
  40066c: 8fbe0018  lw s8,24(sp)
  400670: 27bd0020  addiu sp,sp,32
  400674: 03e00008  jr ra
  400678: 00200825  move at,at
  40067c: 00200825  move at,at

i386 hello world:

0804840c <main>:
#include <stdio.h>

int main() {
 804840c: 55                    push   ebp
 804840d: 89 e5                 mov    ebp,esp
 804840f: 83 e4 f0              and    esp,0xfffffff0
 8048412: 83 ec 10              sub    esp,0x10
    printf("Hello world!\n");
 8048415: c7 04 24 c0 84 04 08  mov    DWORD PTR [esp],0x80484c0
 804841c: e8 cf fe ff ff        call   80482f0 <puts@plt>
    return 0;
 8048421: b8 00 00 00 00        mov    eax,0x0
}
 8048426: c9                    leave  
 8048427: c3                    ret    
 8048428: 90                    nop
 8048429: 90                    nop
 804842a: 90                    nop
 804842b: 90                    nop
 804842c: 90                    nop
 804842d: 90                    nop
 804842e: 90                    nop
 804842f: 90                    nop

x64 hello world:

00000000004004fd <main>:
#include <stdio.h>

int main() {
  4004fd: 55                    push   rbp
  4004fe: 48 89 e5              mov    rbp,rsp
    printf("Hello world!\n");
  400501: bf a4 05 40 00        mov    edi,0x4005a4
  400506: e8 d5 fe ff ff        call   4003e0 <puts@plt>
    return 0;
  40050b: b8 00 00 00 00        mov    eax,0x0
}
  400510: 5d                    pop    rbp
  400511: c3                    ret    
  400512: 66 2e 0f 1f 84 00 00  nop    WORD PTR cs:[rax+rax*1+0x0]
  400519: 00 00 00 
  40051c: 0f 1f 40 00           nop    DWORD PTR [rax+0x0]

Instructions for other archs with qemu:
https://gmplib.org/~tege/qemu.html

Kernel. Power management

cat /sys/module/cpuidle/parameters/off

cd /sys/devices/system/cpu/cpu0/cpuidle
cat */power
cat */usage

C-State

powertop

The cpuidle subsystem

Морфологический анализ языка

pymorphy http://www.youtube.com/watch?v=qPNaWSUtD6E&list=PL7sYAFudE7X1wb54NOiMaRkE5RUbJZrG5

Programming Erlang: Software for a Concurrent World by Joe Armstrong

Terminal

Oblivion
text color #D3D7CF
background color #2E3436

Inconsolata Medium 13

Tango

gconftool --set --type string /apps/gnome-terminal/profiles/Default/palette \ 
'#3b3b3b:#cf6a4c:#99ad6a:#d8ad4c:#597bc5:#a037b0:#71b9f8:#adadad:#636363:#f79274:#c1d592:#ffd574:#81a3ed:#c85fd8:#99e1ff:#d5d5d5'

CSS is awesome ))

Kernel. Networking

struct sk_buff

struct net_device
devices: eth ppp asdn atm
rx recieve
tz transmit

Explicit Congestion Notification (ECN) — (Явное Уведомление о Перегруженности)

ls -l /sys/class/net

cat /proc/net/sockstat
cat /proc/net/dev
cat /proc/net/tcp  # ports
cat /proc/net/udp
cat /proc/net/unix

Nearly all cards support DMA mode

IPv4
IPv6

UDP
TCP

mtu maximum transmission unit
ARP address resolution protocol
NAT network address translation; masquerading
transparent proxy
netfilter; iptables
finite state machine конечный автомат
tcpdump witeshark

tshark -V -2 -R 'ip.addr==208.118.235.148'

# http://stackoverflow.com/questions/5907527/application-control-of-tcp-retransmission-on-linux
/proc/sys/net/ipv4/tcp_retries2

man 5 socketckall

man 3 netlink
man 5 netlink
/proc/net/netlink

Unix as IDE. Man

man -k someterm
apropos someterm

info -k printf
info coreutils printf

Dia

sudo aptitude install autodia
autodia -l python -D -H -m -K -k -A -i test.py
autodia -l DBI -i "mysql:databasename:localhost" -U root -P pass

Extended file attributes, ACL

Does file system support?

sudo tune2fs -l /dev/sda1 | grep "Default mount options:"
# Default mount options:    user_xattr acl

Extended file attributes

Extended attributes: the good, the not so good, the bad.
On Linux and FreeBSD, cp will only preserve xattrs if given the option --preserve=xattr, also implied by the -a option. cp will not duplicate attributes by default, which is a reasonable default choice, even if it will be wrong in some situations (where you will have to use the options).

Extended Attributes – lsattr and chattr

• a : append only 
• c : compressed 
• d : no dump 
• e : extent format 
• i : immutable 
• j : data journalling 
• s : secure deletion 
• t : no tail-merging 
• u : undeletable 
• A : no atime updates 
• C : no copy on write 
• D : synchronous directory updates 
• S : synchronous updates 
• T : top of directory hierarchy

lsattr test.txt
chattr -a test.txt

SELinux (англ. Security-Enhanced Linux — Linux с улучшенной безопасностью) — реализация системы принудительного контроля доступа, которая может работать параллельно с классической дискреционной системой контроля доступа. Входит в стандартное ядро Linux. Также для функционирования SELinux требуются модифицированные версии некоторых утилит (ps, ls, и тд), которые обеспечивают поддержку новых функций ядра, и поддержка со стороны файловой системы.

setfattr -n user.comment -v "this is a comment" testfile
getfattr testfile
# user.comment
getfattr -n user.comment testfile
# user.comment="this is a comment"

POSIX Access Control Lists on Linux
DAC model

man 5 acl
getfacl dir
setfacl -m user:joe:rwx dir
man chacl

GCC, DDD

gcc

# -S to output assembly
gcc -O0 -S hello.c
# -masm=intel or -masm=att

ddd

gcc -g -c sample sample.c
ddd sample

ddd --disassemble sample

X86 Registers

Kernel. Modules, program exec, syscalls, elf

man 2 syscalls

system_call()

strace -e trace=open date
strace -c date

ltrace -T -S rmdir t
ltrace -c -S rmdir t

ldd `which python`

Elf

# view headers, section...
readelf -a a.out
readelf --sections=vmlinux a.out
readelf --dynamic a.out

# disassembly
objdump -M intel -S a.out

# list symbols from object files
nm a.out

# dependencies resolved from this list
cat /proc/kallsyms | grep printk

cat /boot/System.map-3.13.0 | grep printk

stat a.out
file a.out

Modules

# list modules
lsmod

# Modules dependencies
cat /lib/modules/3.12.7/modules.dep | grep vboxdrv
man depmod

# info
modinfo vboxdrv

# view system device information
systool -v -m vboxdrv

struct module
sys_init_module() -> load_module()
sys_delete_module()
request_module()
hello world module

man 2 init_module
man 2 delete_module

# which module loader
cat /proc/sys/kernel/modprobe
sudo sysctl -a | grep modprobe

man modprobe
man insmod
man rmmod

ll /sbin/udevd

udevadm monitor

cat /proc/sys/kernel/taintined
dmesg | grep taint

Linux kernel. Devices and drivers

cat /proc/partitions
cat /proc/devices

tmpfs

df -h # tmpfs and subfolders v

udev

ps aux | grep [u]dev  # demons
udevadm monitor  # new devices

struct gendisk
add_disk()

struct block_device
struct block_device_operations
struct request_queue
struct request

sumit_bio()
generic_make_request

Linux I/O Scheduler
struct elevator_ops
vfs_ioctl() - call filesystem specific ioctl methods

man 2 ioctl # control device

I/O ports and I/O memmory vs DMA
IOMMU

cat /proc/ioports
cat /proc/mem

struct device
struct bus_type
bus_register()

sysfs vs Kernfs introduced in 3.14

ll /sys/block
ll /sys/bus/pci/

ISA, PCI, PCI Express

struct pci_bus

struct pci_dev
struct pci_driver

Vfs
struct inode
struct inode_operations

dmesg|grep Inode
#[    0.001283] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)

struct file_struct - opened by process

struct file
struct file_operations

RCU

struct dentry
struct dentry_operations

NFS, AFS, SMB (CIFS)

struct super_block

mount
ll /dev/disk/by-*

ramfs, tmpfs, sysfs, procfs, debugfs CONFIG_DEBUG_FS

cat /proc/filesystems
ll /proc/buddyinfo
cat /proc/buddyinfo
cat /proc/slabinfo
sudo cat /proc/slabinfo
cat /proc/vmstat
cat /proc/kallsyms
ll /proc/kcore  # + gdb, ddd
cat /proc/interrupts

sys_open() - adds fd to current->fs
ACL
Ext2 reservation window

Ext3
writeback, ordered, journal

Arch

OpenRISC — открытый микропроцессор архитектуры RISC с открытым исходным кодом на языке описания аппаратного обеспечения Verilog. Проект создан сообществом OpenCores и распространяется по лицензии GNU LGPL. OpenRISC воплощён аппаратно и успешно производится в виде интегральных микросхем и ПЛИС (PLD).

SPARC - SPARC (Scalable Processor ARChitecture — масштабируемая архитектура процессора) — архитектура RISC-микропроцессоров, первоначально разработанная в 1985 году компанией Sun Microsystems.
МЦСТ R1000 (NUMA)

ARM (Advanced RISC Machine, Acorn RISC Machine, усовершенствованная RISC-машина) — семейство лицензируемых 32-битных и 64-битных микропроцессорных ядер разработки компании ARM Limited.

MIPS (англ. Microprocessor without Interlocked Pipeline Stages) — микропроцессор, разработанный компанией MIPS Computer Systems (в настоящее время MIPS Technologies) в соответствии с концепцией проектирования процессоров RISC (то есть для процессоров с сокращенным набором команд).

DEC Alpha (также известный как Alpha AXP) — 64-разрядный RISC микропроцессор, первоначально разработанный и произведённый компанией DEC, которая использовала его в собственной линейке рабочих станций и серверов.

PowerPC (или сокращённо PPC) — микропроцессорная RISC-архитектура, созданная в 1991 году альянсом компаний Apple, IBM и Motorola, известным как AIM.

IA-64 (also called Intel Itanium architecture) is the architecture of the Itanium family of 64-bit of Intel microprocessors. The architecture originated at Hewlett-Packard (HP), and was later jointly developed by HP and Intel.
The Itanium architecture is based on explicit instruction-level parallelism, in which the compiler decides which instructions to execute in parallel. This contrasts with other superscalar architectures, which depend on the processor to manage instruction dependencies at runtime. In all Itanium models, up to and including Tukwila, cores execute up to six instructions per clock cycle. The first Itanium processor, codenamed Merced, was released in 2001.

IA-32 (i386) (Intel Architecture, 32-bit, sometimes called i386 through metonymy) is the title of the third generation of x86 architecture, first implemented in the Intel 80386 microprocessors in 1985. It was the first incarnation of x86 to support 32-bit computing. As such, "IA-32" may be used as a metonym to refer to all x86 versions that supported 32-bit computing.

x86-64 (x64) — 64-битная аппаратная платформа (чипсет и архитектура набора команд), разработанная компанией AMD для выполнения 64-разрядных приложений.
Это расширение архитектуры x86 с почти полной обратной совместимостью.

List of Linux supported architectures
Modern Microprocessors. A 90 Minute Guide!

Reviewing Kali Linux – the distro for security geeks

http://www.binarytides.com/kali-linux-security-distro/

Kernel and Hardware Abbriviations

ASMP - Asymmetric multiprocessing
SMP - Symmetric Multiprocessing
Регистр процессора
OOO - Внеочередное исполнение (англ. Out-of-order execution) likely, unlikely
Кэш процессора
Алгоритмы кэширования: LRU - Least Recently Used; MRU - Most Recently Used
Выравнивание данных


UMA - Uniform Memory Access
NUMA - Non-Uniform Memory Access

BIOS  SLIC - Software Licensing Description Table
UEFI, EFI Extensible Firmware Interface (англ. Расширяемый интерфейс прошивки) — интерфейс между операционной системой и микропрограммами, управляющими низкоуровневыми функциями оборудования, его основное предназначение: корректно инициализировать оборудование при включении системы и передать управление загрузчику операционной системы. EFI предназначен для замены BIOS — интерфейса, который традиционно используется всеми IBM PC-совместимыми персональными компьютерами.
ACPI - Advanced Configuration and Power Interface — усовершенствованный интерфейс управления конфигурацией и питанием. Suspend, Hibernate
e820 - Функция прерывания 15H BIOS персонального компьютера, использующаяся для получения информации о распределении адресного пространства, имеющейся оперативной памяти и зарезервированных адресах.
DMI - (Desktop Management Interface) — интерфейс программирования приложений (API), позволяющий программному обеспечению собирать данные о характеристиках компьютера. dmidecode
RAM
ROM - Постоянное запоминающее устройство (ПЗУ)

MMU - Блок управления памятью или устройство управления памятью (англ. memory management unit)
PAE - Physical Address Extension — режим работы встроенного блока управления памятью x86-совместимых процессоров, в котором используются 64-битные элементы таблиц страниц, c помощью которых процессор может адресовать 64 ГБ физической памяти (вместо 4 ГБ, адресуемых при использовании 32-разрядных таблиц), хотя каждая задача (программа) всё равно может адресовать максимум до 4 ГБ виртуальной памяти. Также, в новых моделях процессоров в PAE-режиме старший бит элемента таблицы страниц отвечает за запрет исполнения кода в странице, что затрудняет атаку по методу переполнения буфера.
NX bit - No eXecute Bit
PSE - Page Size Extension (PSE) — режим работы встроенного блока управления памятью x86-совместимых процессоров, в котором используются страницы размером в 4 (32-битные таблицы страниц) или 2 мегабайта (PAE/AMD64, 64-битные таблицы), в дополнение к обычным страницам x86 архитектуры в 4 килобайта.
DMA - Прямой доступ к памяти
TLB - Буфер ассоциативной трансляции (англ. Translation lookaside buffer, TLB) — это специализированный кэш центрального процессора, используемый для ускорения трансляции адреса виртуальной памяти в адрес физической памяти.
PGD - Global
PUD - Upper
PMD - Middle
PTE- Page Table entry

Slab Slub Slob
ASLR Address space layout randomization — технология, применяемая в операционных системах, при использовании которой случайным образом изменяется расположение в адресном пространстве процесса важных структур, а именно: образа исполняемого файла, подгружаемых библиотек, кучи и стека.


Completely Fair Scheduler
Round robin scheduling
FIFO - First In, First Out /usr/include/x86_64-linux-gnu/sys/queue.h

MALLOC_CHECK_ and MALLOC_PERTURB_

http://l3net.wordpress.com/2013/06/19/malloc_check_-and-malloc_perturb_/

Recent versions of Linux glibc (2.x) include a a malloc() implementation tunable using environment variables. This allows the user to diagnose allocation problems such as heap corruption, double free etc.

When MALLOC_CHECK_ is set to 3, a diagnostic message is printed on stderr and the program is aborted. A value of 0 disables the diagnostic – see man malloc for more details.

Setting the MALLOC_PERTURB_ environment variable causes the malloc functions in libc to return memory which has been wiped and initialized with the byte value of the environment variable. Setting MALLOC_PERTURB_ to zero disables the feature.

No special arguments need to be passed during program compilation. These diagnostics work on any precompiled programs. Unlike valgrind, the speed of execution is not affected.

export MALLOC_CHECK_=3
export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))

Valgrind overview: runtime memory checker and a bit more aka использование #valgrind на селе

valgrind --tool=callgrind ls 

http://www.youtube.com/watch?v=dKngHt9Cvok

Alleyoop is a Valgrind front-end for the GNOME environment, made by Jeffrey Stedfast.

Filesystem monitoring in the Linux kernel

http://www.lanedo.com/2013/filesystem-monitoring-linux-kernel/

How a mail server works

http://xmodulo.com/2014/01/how-mail-server-works.html

Linux kernel. Scheduler

priority to load weight
load balancing, kmigrate

run queues, sched entities:

cat /proc/sched_debug
# Sched Debug Version: v0.11, 3.13.6 #4
# ktime                                   : 11620320.261793
# sched_clk                               : 11629180.728702
# cpu_clk                                 : 11629180.728834
# jiffies                                 : 4297797377
# sched_clock_stable                      : 1
# 
# cpu#0, 3077.541 MHz
#   .nr_running                    : 0
#   .load                          : 0
#   .nr_switches                   : 10802526
#   .nr_load_updates               : 1998334
#   .nr_uninterruptible            : -2228
#   .next_balance                  : 4297.797394
# .....
# rt_rq[0]:/
#   .rt_nr_running                 : 0
#   .rt_throttled                  : 0
#   .rt_time                       : 0.011064
#   .rt_runtime                    : 950.000000
# 
# runnable tasks:
#             task   PID         tree-key  switches  prio     exec-runtime ...
# ------------------------------------------------------------------------ ...
#      ksoftirqd/0     3    875652.689495     83449   120    875652.689495 ...
#     kworker/0:0H     5      3837.050667         6   100      3837.050667 ...
#        rcu_sched     7    875652.806577    388480   120    875652.806577 ...

sudo cat /proc/schedstat
cat /proc/PID/sched
cat /proc/PID/limits
ls /proc/sys/kernel | grep sched
sudo sysctl -A | grep sched

Real time policy (priority 90):

sudo chrt -r -p 90 22336

SCHED_RR implements a round robin method
SCHED_FIFO uses a first in, first out mechanism

schedtool -h

struct thread_info
struct task_struct
struct nsproxy

do_fork()
do_execve()
kernel_thread()

struct sched_class
struct rq
struct sched_entity
scheduler_tick()
context_switch()

struct sched_class fair_sched_class
check_preempt_tick()

struct sched_class rt_sched_class
task_tick_rt()

New in 3.14
Earliest deadline first scheduling
struct sched_class dl_sched_class
task_tick_dl()

find_busiest_queue() compares rqs load weights