SlideShare a Scribd company logo
Advanced Python
Programming (Part II)
       Happy Day #5
         2011.10
Topics

• Performance Tuning
• Garbage Collection
• Extending Python
Performance Tuning
Python's Speed
                                                  Among Most Popular Languages




             C          C++           Java        Lisp         C#         Pascal       Python   Ruby   PHP   Perl
data source (on Oct.17, 2011):
• http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html
• http://shootout.alioth.debian.org/u64q/which-programming-languages-are-fastest.php
7 Steps to Gain Speed
1) Find performance bottlenecks
2) Use better algorithms
3) Use faster tools
4) Write optimized code
5) Hire optimizers
6) Write your own extension modules
7) Parallelize the computation
Step 0
Is It Fast Enough Already?
Step 1
Find Performance Bottlenecks
Find Performance
           Bottlenecks
•   Profile, no guess
    -   profile
        •   a pure Python module
    -   cProfile
        •   written in C, new in Python 2.5
        •   same interface with profile, but lower overhead
    -   hotshot
        •   written in C, new in Python 2.2
        •   not maintained and might be removed
cProfile Usage
•   cProfile.run('foo()')
•   cProfile.run('foo()', 'profile.result')
•   python -m cProfile -o profile.result myscript.py
•   p = pstats.Stats('profile.result')
•   p.sort_stats('cumulative').print_stats()
    •   sort by 'cumulative' to find what algorithms are taking time
    •   sort by 'time' to find what functions are taking time
•   RunSnakeRun for GUI guys
•   RTFM, please
•   for IPython, type %prun?
Line Profile
 • line_profile and kernprof
@profile
def slow_function():
    ...

$ kernprof.py -l -v script_to_profile.py
...
Line #      Hits         Time Per Hit    % Time Line Contents
==============================================================
     1                                           @profile
     2                                           def slow_function():
     3         1            3      3.0      0.2      s = 0
     4      1001          934      0.9     48.6      for i in xrange(1000):
     5      1000          984      1.0     51.2           s += i
     6         1            1      1.0      0.1      return s
Step 2
Use Better Algorithms
How To Know Which is
       Better?
 • timeit!
 • python -m timeit -s "setup" "statement"
 • e.g. which is faster, "d.has_key(k)" or "k in
     d"?
$ python -m timeit -s "d=dict(zip(range(1000), range(1000)))"
"d.has_key(500)"
1000000 loops, best of 3: 0.223 usec per loop

$ python -m timeit -s "d=dict(zip(range(1000), range(1000)))"
"500 in d"
10000000 loops, best of 3: 0.115 usec per loop
Use Bettern Algorithms
• How to calculate sum([1, 2, ..., 100])?
Use Bettern Algorithms
 • How to calculate sum([1, 2, ..., 100])?
s = 0
for i in range(101):                8.3usec
    s += i
Use Bettern Algorithms
 • How to calculate sum([1, 2, ..., 100])?
s = 0
for i in range(101):                8.3usec
    s += i

s = sum(range(101))                 2.8usec
Use Bettern Algorithms
 • How to calculate sum([1, 2, ..., 100])?
s = 0
for i in range(101):                8.3usec
    s += i

s = sum(range(101))                 2.8usec
s = sum(xrange(101))                2.03usec
Use Bettern Algorithms
 • How to calculate sum([1, 2, ..., 100])?
s = 0
for i in range(101):                8.3usec
    s += i

s = sum(range(101))                 2.8usec
s = sum(xrange(101))                2.03usec
s = (1 + 100) * 100 / 2             0.109usec
Advanced Data Types
• membership testing:
 • set & dict: O(1) vs. tuple & list: O(n)
• return iterator instead of a large list
• array, collections.deque, heapq, bisect
Examples
lst = []
for i in xrange(10000):
    lst.insert(0, i)

lst = collections.deque()
for i in xrange(10000):          25317% faster
    lst.appendleft(i)




sorted(lst, reverse=True)[:10]

heapq.nlargest(10, lst)
                                 613% faster
Do Less Computation

• Pre-computation
• Lazy computation
• Cache
• Approximation Algorithms
Example
def fib(n):
    if n <= 1:
        return 1                 fib(25): 59.8ms
    return fib(n-2) + fib(n-1)
Example
def cache(func):
    c = {}
    def _(n):
        r = c.get(n)              fib(25): 59.8ms
        if r is None:
             r = c[n] = func(n)
        return r
    return _                      with @cache:
@cache                            fib(25): 0.524us
def fib(n):
    if n <= 1:
        return 1                  112000 times faster!
    return fib(n-2) + fib(n-1)
Step 3
Use Faster Tools
Use Faster Tools
•   use iterator form
    •   range() -> xrange()
    •   map() -> itertools.imap()
    •   list comprehension -> generator expression
    •   dict.items() -> dict.iteritems()
    •   for i in range(len(seq)): ->
        •   for item in seq:
        •   for i, item in enumerate(seq):
Use Faster Tools
•   SAX is faster and memory efficient than DOM
•   use C version of modules
    •   profile -> cProfile
    •   StringIO -> cStringIO
    •   pickle -> cPickle
    •   elementTree -> cElementTree / lxml
•   select has lower overhead than poll (and epoll at low
    number of connections)
•   numpy is essential for high volume numeric work
numpy Example
from itertools import izip
a=range(1000)
b=range(1000)
c = [ai+bi for ai, bi in izip(a, b)]

import numpy
a=numpy.arange(1000)
b=numpy.arange(1000)
c = a + b
Step 4
Write Optimized Code
Write Optimized Code

• Less temporary objects
 • e.g. accumulator vs. sum
 • however, string concatenation has been
    optimized after Python 2.5
Write Optimized Code
 • use key= instead of cmp= when sorting
lst = open('/Users/hongqn/projects/shire/luzong/
group.py').read().split()

lst.sort(cmp=lambda x, y: cmp(x.lower(), y.lower()))


lst.sort(key=str.lower)     377% faster
Write Optimized Code
• local variables are faster than global
   variables
def f():
    for i in xrange(10000):
         r = abs(i)

def f():
    _abs = abs
    for i in xrange(10000):
         r = _abs(i)
                              28% faster

• you can eliminate dots, too
Write Optimized Code
• inline function inside time-critical loops
def f(x):
    return x + 1
for i in xrange(10000):
    r = f(i)


for i in xrange(10000):
    r = i + 1                  187% faster
Write Optimized Code
• do not import modules in loops
for i in xrange(10000):
    import string
    r = string.lower('Python')


import string
for i in xrange(10000):
    r = string.lower('Python')
                                 178% faster
Write Optimized Code
• list comprehensions are faster than for-
  loops
lst = []
for i in xrange(10000):
    lst.append(i)



lst = [i for i in xrange(10000)]     213% faster
Write Optimized Code
• use "while 1" for time-critical loops
   (readability lost!)
a = 0
while True:
    a += 1
    if a > 10000:
        break

a = 0
while 1:
    a += 1
    if a > 10000:        78% faster
         break
Write Optimized Code
• "not not x" is faster than "bool(x)" (not
  recommended!)
bool([])



not not []                  196% faster
Step 5
Hire Optmizers
Hire Optimizers

• sys.setcheckinterval()
 • Python checks for thread switch and
    signal handling periodly (default 100
    python virtual instructions)
 • set it to a larger value for better
    performance in cost of responsiveness
Hire Optimizers

• gc.disable()
 • disable automatic garbage collection
• gc.set_threshold()
 • collect less frequently
Hire Optimizers
•   Psyco
    •   JIT for 32bit only. <=Python 2.6
•   PyPy
    •   Alternative implementation of Python
•   Shed Skin
    •   Python-to-C++ compiler
•   numexpr
    •   numpy expression evaluator
Step 6
Write Your Own Extension Modules
Write Your Own
        Extension Modules
•   Python/C API
    •   Official API
•   ctypes
    •   Call dynamic link library in Python
•   SWIG
    •   Automatically generate interface code
•   Pyrex / Cython
    •   write extension using Python-like language
•   Boost.Python
    •   C++ API
•   Weave
    •   Inline C code
Write Your Own
   Extension Modules


• C-level Optimization
Step 7
Parallelize the Computation
Over CPUs
•   multi-threading
    •   threading (be careful with GIL!)
•   multi-processing
    •   fork
    •   subprocess
    •   multiprocessing
•   async
    •   asyncore
    •   twisted
    •   greenlet/gevent
•   PyOpenCL / PyCUDA
multiprocessing
                Example
sum(xrange(1, 10000001))                172ms

from multiprocessing import Pool
pool = Pool()
sum(pool.map(sum, (xrange(i, i+1000000)
                   for i in xrange(1, 10000000, 1000000))))

                                        104ms on dual-core
                                        49ms on 8-core
Over Cluster
• XML-RPC / Json-RPC / Thrift / Protocol
  Buffer
• Pyro
• Parallel Python
• dumbo on Hadoop
• dpark
Gold Rule


Premature optimization is the root of all evil.
                              -- Donald Knuth
Garbage Collection
What Is Garbage


• An object which will not be used in any
  future
• i.e. no other object refers to it
Why Collecting
        Garbage?

• re-sell it (recycle)
• make programs simpler
Garbage Collection in
    Everywhere

• by lilinghui, Sep.20, 2011
• http://svn.douban.com/projects/shire/wiki/
  HallOfFire/Platform
Basic Garbage
Collection Algorithms
• Reference couting
• Mark-and-sweep
• Mark-and-compact
• Copy
Garbage Collection in
     CPython

• Reference couting
• Mark-and-sweep
Reference Counting
def f():
  a = A()
  a.b = B()
  a.b.c = C()
  a.c = b.c
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a           b
  a.c = b.c
                1           1


                      c
                      2
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a           b
  a.c = b.c
                1           1
  del a.c

                      c
                      2
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a           b
  a.c = b.c
                1           1
  del a.c

                      c
                      1
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a           b
  a.c = b.c
                1           1
  del a.c
  del a.b
                      c
                      1
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a           b
  a.c = b.c
                1           0
  del a.c
  del a.b
                      c
                      1
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a
  a.c = b.c
                1
  del a.c
  del a.b
                      c
                      0
Reference Counting
def f():            frame
  a = A()
  a.b = B()
  a.b.c = C()   a
  a.c = b.c
                1
  del a.c
  del a.b
Reference Counting
def f():
  a = A()
  a.b = B()
  a.b.c = C()   a
  a.c = b.c
                0
  del a.c
  del a.b
Reference Counting
def f():
  a = A()
  a.b = B()
  a.b.c = C()
  a.c = b.c
  del a.c
  del a.b
Pros & Cons of
  Reference Counting
• Pros
 • collect early
 • predictable run-time behavior
• Cons
 • slow
 • circle reference
Circle Reference
def f():              frame
  a = A()
  a.b = B()
  a.b.c = C()     a           b
  a.b.c.b = a.b
                  1           2


                        c
                        1
Circle Reference
def f():              frame
  a = A()
  a.b = B()
  a.b.c = C()     a           b
  a.b.c.b = a.b
                  1           2
 del a.b

                        c
                        1
Circle Reference
def f():              frame
  a = A()
  a.b = B()
  a.b.c = C()     a           b
  a.b.c.b = a.b
                  1           1
 del a.b

                        c
                        1
Circle Reference
def f():
  a = A()
  a.b = B()
  a.b.c = C()            b
  a.b.c.b = a.b
                         1
 del a.b

                  c
                  1
Weak Reference
a reference not strong enough to keep an object alive

>>> import weakref       import weakref
>>> class Object:
...     pass             _id2obj_dict = weakref.WeakValueDictionary()
...
>>> o = Object()         def remember(obj):
>>> r = weakref.ref(o)       oid = id(obj)
>>> o2 = r()                 _id2obj_dict[oid] = obj
>>> o is o2                  return oid
True
>>> del o, o2            def id2obj(oid):
>>> print r()                return _id2obj_dict[oid]
None
Mark-and-Sweep
                              frame
1. Mark root objects
and all their referents
reachable                 a           b
2. Sweep unreachable      1           1
objects
                                c
                                1
Generations
• generations and thresholds
 • generation 0 (youngest): 700
 • generation 1 (middle):10
 • generation 2 (oldest): 10
   • and long_lived_pending /
      long_lived_total > 25% (Python 2.7+)
Python's Optimization


• Track only container objects
• Use referent count to find root objects
__del__


• circle referenced objects with __del__()
  methods will be put in gc.garbage
• do not use __del__
GC Module
• gc.enable() / gc.disable()
• gc.collect()
• gc.get_threshold() / gc.set_threshold()
• gc.set_debug()
• gc.get_referers() / gc.get_referents()
• gc.get_objects()
Extending Python
Python/C API
#include <Python.h>

static PyObject *SpamError;

static PyObject *
spam_system(PyObject *self, PyObject *args)
{
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = system(command);
    if (sts < 0) {
        PyErr_SetString(SpamError, "System command failed");
        return NULL;
    }
    return PyLong_FromLong(sts);
}

static PyMethodDef SpamMethods[] = {
    {"system", spam_system, METH_VARARGS,
     "Execute a shell command."},
    {NULL, NULL, 0, NULL}         /* Sentinel */
};

PyMODINIT_FUNC
initspam(void)
{
    PyObject *m;

    m = Py_InitModule("spam", SpamMethods);
    if (m == NULL)
        return;

    SpamError = PyErr_NewException("spam.error", NULL, NULL);
    Py_INCREF(SpamError);
    PyModule_AddObject(m, "error", SpamError);
}
Building with
                 setuptools
from setuptools import setup, Extension

module1 = Extension('spam',
                    sources = ['spam.c'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])


python setup.py install
python setup.py build_ext --inplace
Parse and Build Values
• int PyArg_ParseTuple(PyObject *args,
  const char *format, ...)

• int PyArg_ParseTupleAndKeywords(PyObject
  *args, PyObject *kw, const char *format,
  char *keywords[], ...)

• int PyArg_BuildValue(const char
  *format, ...)
Manage Reference
      Counting
• Py_INCREF(PyObject *o) /
  Py_XINCREF(PyObject *o)
• Py_DECREF(PyObject *o) /
  Py_XINCREF(PyObject *o) /
  Py_CLEAR(PyObject *o)
• New/Borrowed/Stealing references
Exception Handling
• errno-like mechanism
• PyErr_SetString() / PyErr_SetObject() to
  set error
• PyErr_Occurred() to check error
• Most functions return an error indicator,
  e.g. NULL, -1
Global Interpreter Lock
• Release GIL before running blocking C code
   Py_BEGIN_ALLOW_THREADS
   ...Do some blocking I/O operation...
   Py_END_ALLOW_THREADS


• Reacquire GIL before calling into Python functions
   PyGILState_STATE gstate;
   gstate = PyGILState_Ensure();

   /* Perform Python actions here.   */
   result = CallSomeFunction();
   /* evaluate result */

   /* Release the thread. No Python API allowed beyond this point. */
   PyGILState_Release(gstate);
Cython / Pyrex
Python Style Code
from libc.stdlib cimport system as c_system

class error(Exception):
    pass

def system(char* cmd):
    cdef int retval = c_system(cmd)
    if retval < 0:
        raise error("System command failed")
    return retval
setup.py
from setuptools import setup, Extension
from Cython.Distutils import build_ext

extmod = Extension("spam", sources=['spam.pyx'])

setup(
    name="Spam",
    cmdclass = {'build_ext': build_ext},
    ext_modules = [extmod])
#include <Python.h>
#include "structmember.h"
                                                                             static PyGetSetDef Noddy_getseters[] = {




                                                                                                                                                   Extension Types
typedef struct {                                                                 {"first",
    PyObject_HEAD                                                                 (getter)Noddy_getfirst, (setter)Noddy_setfirst,
    PyObject *first;                                                              "first name",
    PyObject *last;                                                               NULL},
    int number;                                                                  {"last",
} Noddy;                                                                          (getter)Noddy_getlast, (setter)Noddy_setlast,
                                                                                  "last name",
static void                                                                       NULL},
Noddy_dealloc(Noddy* self)                                                       {NULL} /* Sentinel */
{                                                                            };
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);                                                  static PyObject *
    self->ob_type->tp_free((PyObject*)self);                                 Noddy_name(Noddy* self)
}                                                                            {
                                                                                 static PyObject *format = NULL;
static PyObject *                                                                PyObject *args, *result;
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{                                                                                if (format == NULL) {
    Noddy *self;                                                                     format = PyString_FromString("%s %s");
                                                                                     if (format == NULL)
      self = (Noddy *)type->tp_alloc(type, 0);                                           return NULL;
      if (self != NULL) {                                                        }
          self->first = PyString_FromString("");
          if (self->first == NULL)                                               args = Py_BuildValue("OO", self->first, self->last);
            {                                                                    if (args == NULL)
              Py_DECREF(self);                                                       return NULL;
              return NULL;
            }                                                                    result = PyString_Format(format, args);
                                                                                 Py_DECREF(args);
          self->last = PyString_FromString("");
          if (self->last == NULL)                                                return result;
            {                                                                }
              Py_DECREF(self);
              return NULL;                                                   static PyMethodDef Noddy_methods[] = {
            }                                                                    {"name", (PyCFunction)Noddy_name, METH_NOARGS,
                                                                                  "Return the name, combining the first and last name"
          self->number = 0;                                                      },
      }                                                                          {NULL} /* Sentinel */
                                                                             };
      return (PyObject *)self;
}                                                                            static PyTypeObject NoddyType = {
                                                                                 PyObject_HEAD_INIT(NULL)
static int                                                                       0,                         /*ob_size*/
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)                          "noddy.Noddy",             /*tp_name*/
{                                                                                sizeof(Noddy),             /*tp_basicsize*/
    PyObject *first=NULL, *last=NULL, *tmp;                                      0,                         /*tp_itemsize*/
                                                                                 (destructor)Noddy_dealloc, /*tp_dealloc*/
      static char *kwlist[] = {"first", "last", "number", NULL};                 0,                         /*tp_print*/
                                                                                 0,                         /*tp_getattr*/
      if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist,              0,                         /*tp_setattr*/
                                        &first, &last,                           0,                         /*tp_compare*/
                                        &self->number))                          0,                         /*tp_repr*/
          return -1;                                                             0,                         /*tp_as_number*/
                                                                                 0,                         /*tp_as_sequence*/
      if (first) {                                                               0,                         /*tp_as_mapping*/
          tmp = self->first;                                                     0,                         /*tp_hash */
          Py_INCREF(first);                                                      0,                         /*tp_call*/
          self->first = first;                                                   0,                         /*tp_str*/
          Py_DECREF(tmp);                                                        0,                         /*tp_getattro*/
      }                                                                          0,                         /*tp_setattro*/
                                                                                 0,                         /*tp_as_buffer*/
      if (last) {                                                                Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
          tmp = self->last;                                                      "Noddy objects",           /* tp_doc */
          Py_INCREF(last);                                                       0,!      !                            /* tp_traverse */
          self->last = last;                                                     0,!      !                            /* tp_clear */
          Py_DECREF(tmp);                                                        0,!      !                            /* tp_richcompare */
      }                                                                          0,!      !                            /* tp_weaklistoffset */
                                                                                 0,!      !                            /* tp_iter */
      return 0;                                                                  0,!      !                            /* tp_iternext */
}                                                                                Noddy_methods,             /* tp_methods */
                                                                                 Noddy_members,             /* tp_members */
static PyMemberDef Noddy_members[] = {                                           Noddy_getseters,           /* tp_getset */
    {"number", T_INT, offsetof(Noddy, number), 0,                                0,                         /* tp_base */
     "noddy number"},                                                            0,                         /* tp_dict */
    {NULL} /* Sentinel */                                                        0,                         /* tp_descr_get */
};                                                                               0,                         /* tp_descr_set */
                                                                                 0,                         /* tp_dictoffset */
static PyObject *                                                                (initproc)Noddy_init,      /* tp_init */
Noddy_getfirst(Noddy *self, void *closure)                                       0,                         /* tp_alloc */
{                                                                                Noddy_new,                 /* tp_new */
    Py_INCREF(self->first);                                                  };
    return self->first;
}                                                                            static PyMethodDef module_methods[] = {
                                                                                 {NULL} /* Sentinel */
static int                                                                   };
Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
{                                                                            #ifndef PyMODINIT_FUNC!    /* declarations for DLL import/export */
  if (value == NULL) {                                                       #define PyMODINIT_FUNC void
    PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");   #endif
    return -1;                                                               PyMODINIT_FUNC
  }                                                                          initnoddy3(void)
                                                                             {
    if (! PyString_Check(value)) {                                               PyObject* m;
      PyErr_SetString(PyExc_TypeError,
                      "The first attribute value must be a string");             if (PyType_Ready(&NoddyType) < 0)
      return -1;                                                                     return;
    }
                                                                                 m = Py_InitModule3("noddy3", module_methods,
    Py_DECREF(self->first);                                                                         "Example module that creates an extension type.");
    Py_INCREF(value);
    self->first = value;                                                         if (m == NULL)
                                                                                   return;
    return 0;
}                                                                                Py_INCREF(&NoddyType);
                                                                                 PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
static PyObject *                                                            }
Noddy_getlast(Noddy *self, void *closure)
{
    Py_INCREF(self->last);
    return self->last;
}

static int
Noddy_setlast(Noddy *self, PyObject *value, void *closure)
{
  if (value == NULL) {
    PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
    return -1;
  }

    if (! PyString_Check(value)) {
      PyErr_SetString(PyExc_TypeError,
                      "The last attribute value must be a string");
      return -1;
    }

    Py_DECREF(self->last);
    Py_INCREF(value);
    self->last = value;

    return 0;
}
#include <Python.h>
#include "structmember.h"
                                                                             static PyGetSetDef Noddy_getseters[] = {




                                                                                                                                                   Extension Types
typedef struct {                                                                 {"first",
    PyObject_HEAD                                                                 (getter)Noddy_getfirst, (setter)Noddy_setfirst,
    PyObject *first;                                                              "first name",
    PyObject *last;                                                               NULL},
    int number;                                                                  {"last",
} Noddy;                                                                          (getter)Noddy_getlast, (setter)Noddy_setlast,
                                                                                  "last name",
static void                                                                       NULL},
Noddy_dealloc(Noddy* self)                                                       {NULL} /* Sentinel */
{                                                                            };
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);                                                  static PyObject *
    self->ob_type->tp_free((PyObject*)self);                                 Noddy_name(Noddy* self)
}                                                                            {
                                                                                 static PyObject *format = NULL;
static PyObject *                                                                PyObject *args, *result;
Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{                                                                                if (format == NULL) {
    Noddy *self;                                                                     format = PyString_FromString("%s %s");
                                                                                     if (format == NULL)
      self = (Noddy *)type->tp_alloc(type, 0);                                           return NULL;
      if (self != NULL) {                                                        }
          self->first = PyString_FromString("");
          if (self->first == NULL)                                               args = Py_BuildValue("OO", self->first, self->last);
            {                                                                    if (args == NULL)
              Py_DECREF(self);                                                       return NULL;
              return NULL;
            }                                                                    result = PyString_Format(format, args);
                                                                                 Py_DECREF(args);
          self->last = PyString_FromString("");
          if (self->last == NULL)                                                return result;
            {
              Py_DECREF(self);
                                                                             }
                                                                                                                                                         cdef class Noddy:
              return NULL;                                                   static PyMethodDef Noddy_methods[] = {
            }                                                                    {"name", (PyCFunction)Noddy_name, METH_NOARGS,
                                                                                  "Return the name, combining the first and last name"
                                                                                 },
                                                                                                                                                             """Noddy objects"""
          self->number = 0;
      }                                                                          {NULL} /* Sentinel */
                                                                             };
      return (PyObject *)self;
}

static int
                                                                             static PyTypeObject NoddyType = {
                                                                                 PyObject_HEAD_INIT(NULL)
                                                                                 0,                         /*ob_size*/
                                                                                                                                                             cdef object first, last
                                                                                 "noddy.Noddy",             /*tp_name*/
Noddy_init(Noddy *self, PyObject *args, PyObject *kwds)
{
    PyObject *first=NULL, *last=NULL, *tmp;
                                                                                 sizeof(Noddy),
                                                                                 0,
                                                                                                            /*tp_basicsize*/
                                                                                                            /*tp_itemsize*/
                                                                                                                                                             cdef public int number
                                                                                 (destructor)Noddy_dealloc, /*tp_dealloc*/
      static char *kwlist[] = {"first", "last", "number", NULL};                 0,                         /*tp_print*/
                                                                                 0,                         /*tp_getattr*/
      if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist,              0,                         /*tp_setattr*/
                                        &first, &last,
                                        &self->number))
                                                                                 0,
                                                                                 0,
                                                                                 0,
                                                                                                            /*tp_compare*/
                                                                                                            /*tp_repr*/
                                                                                                            /*tp_as_number*/
                                                                                                                                                             def __cinit__(self, str first="", str last="", int number=0):
          return -1;

      if (first) {
                                                                                 0,
                                                                                 0,
                                                                                 0,
                                                                                                            /*tp_as_sequence*/
                                                                                                            /*tp_as_mapping*/
                                                                                                            /*tp_hash */
                                                                                                                                                                 self.first = first
          tmp = self->first;
          Py_INCREF(first);
          self->first = first;
          Py_DECREF(tmp);
                                                                                 0,
                                                                                 0,
                                                                                 0,
                                                                                                            /*tp_call*/
                                                                                                            /*tp_str*/
                                                                                                            /*tp_getattro*/
                                                                                                                                                                 self.last = last
                                                                                 0,                         /*tp_setattro*/
      }

      if (last) {
                                                                                 0,                         /*tp_as_buffer*/
                                                                                 Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
                                                                                                                                                                 self.number = number
          tmp = self->last;                                                      "Noddy objects",           /* tp_doc */
          Py_INCREF(last);                                                       0,!      !                            /* tp_traverse */
          self->last = last;                                                     0,!      !                            /* tp_clear */
          Py_DECREF(tmp);                                                        0,!      !                            /* tp_richcompare */
      }                                                                          0,!
                                                                                 0,!
                                                                                 0,!
                                                                                          !
                                                                                          !
                                                                                          !
                                                                                                                       /* tp_weaklistoffset */
                                                                                                                       /* tp_iter */
                                                                                                                       /* tp_iternext */
                                                                                                                                                             property first:
      return 0;
}                                                                                Noddy_methods,
                                                                                 Noddy_members,
                                                                                 Noddy_getseters,
                                                                                                            /* tp_methods */
                                                                                                            /* tp_members */
                                                                                                            /* tp_getset */
                                                                                                                                                                 """first name"""
static PyMemberDef Noddy_members[] = {
    {"number", T_INT, offsetof(Noddy, number), 0,
     "noddy number"},
    {NULL} /* Sentinel */
                                                                                 0,
                                                                                 0,
                                                                                 0,
                                                                                                            /* tp_base */
                                                                                                            /* tp_dict */
                                                                                                            /* tp_descr_get */
                                                                                                                                                                 def __get__(self):
                                                                                 0,                         /* tp_descr_set */
};

static PyObject *
                                                                                 0,
                                                                                 (initproc)Noddy_init,
                                                                                                            /* tp_dictoffset */
                                                                                                            /* tp_init */
                                                                                                                                                                     return self.first
Noddy_getfirst(Noddy *self, void *closure)                                       0,                         /* tp_alloc */
{
    Py_INCREF(self->first);                                                  };
                                                                                 Noddy_new,                 /* tp_new */
                                                                                                                                                                 def __set__(self, str value):
    return self->first;
}                                                                            static PyMethodDef module_methods[] = {

                                                                             };
                                                                                 {NULL} /* Sentinel */                                                               self.first = value
static int
Noddy_setfirst(Noddy *self, PyObject *value, void *closure)
{                                                                            #ifndef PyMODINIT_FUNC!    /* declarations for DLL import/export */
  if (value == NULL) {                                                       #define PyMODINIT_FUNC void


  }
    PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
    return -1;
                                                                             #endif
                                                                             PyMODINIT_FUNC
                                                                             initnoddy3(void)
                                                                                                                                                             property last:
                                                                             {
    if (! PyString_Check(value)) {
      PyErr_SetString(PyExc_TypeError,
                                                                                 PyObject* m;                                                                    """last name"""
                      "The first attribute value must be a string");             if (PyType_Ready(&NoddyType) < 0)

    }
      return -1;                                                                     return;
                                                                                                                                                                 def __get__(self):
                                                                                 m = Py_InitModule3("noddy3", module_methods,
    Py_DECREF(self->first);
    Py_INCREF(value);
                                                                                 if (m == NULL)
                                                                                                    "Example module that creates an extension type.");
                                                                                                                                                                     return self.last
    self->first = value;

    return 0;
                                                                                   return;

                                                                                 Py_INCREF(&NoddyType);
                                                                                                                                                                 def __set__(self, str value):
}

static PyObject *
Noddy_getlast(Noddy *self, void *closure)
                                                                             }
                                                                                 PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType);
                                                                                                                                                                     self.last = value
{
    Py_INCREF(self->last);
    return self->last;
}

static int
                                                                                                                                                             def name(self):
Noddy_setlast(Noddy *self, PyObject *value, void *closure)
{
  if (value == NULL) {
                                                                                                                                                                 """Return the name, combining the first and last name"""
    PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");

  }
    return -1;
                                                                                                                                                                 return "%s %s" % (self.first, self.last)
    if (! PyString_Check(value)) {
      PyErr_SetString(PyExc_TypeError,
                      "The last attribute value must be a string");
      return -1;
    }

    Py_DECREF(self->last);
    Py_INCREF(value);
    self->last = value;

    return 0;
}
GIL
with nogil:
    ...


cdef void my_callback(void *data) with gil:
    ...


cdef void my_gil_free_func(int spam) nogil:
    ...
ctypes
Call C Function in
          Python
from ctypes import cdll
libc = cdll.LoadLibrary('libc.dylib')

class error(Exception):
    pass

def system(cmd):
    retval = libc.system(cmd)
    if retval < 0:
        raise error("System command failed.")
    return retval
GIL


• ctypes.cdll: Release GIL
• ctypes.pydll: With GIL
Q &A
Thanks!

More Related Content

KEY
Python在豆瓣的应用
Qiangning Hong
 
PDF
服务框架: Thrift & PasteScript
Qiangning Hong
 
PDF
OneRing @ OSCamp 2010
Qiangning Hong
 
PDF
Mastering Kotlin Standard Library
Nelson Glauber Leal
 
PDF
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
Fernando Hamasaki de Amorim
 
PPTX
Kotlin coroutines and spring framework
Sunghyouk Bae
 
PDF
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Masahiro Nagano
 
PDF
Kotlin @ Coupang Backend 2017
Sunghyouk Bae
 
Python在豆瓣的应用
Qiangning Hong
 
服务框架: Thrift & PasteScript
Qiangning Hong
 
OneRing @ OSCamp 2010
Qiangning Hong
 
Mastering Kotlin Standard Library
Nelson Glauber Leal
 
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
Fernando Hamasaki de Amorim
 
Kotlin coroutines and spring framework
Sunghyouk Bae
 
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Masahiro Nagano
 
Kotlin @ Coupang Backend 2017
Sunghyouk Bae
 

What's hot (20)

PDF
Новый InterSystems: open-source, митапы, хакатоны
Timur Safin
 
PDF
Kotlin @ Coupang Backed - JetBrains Day seoul 2018
Sunghyouk Bae
 
PDF
あなたのScalaを爆速にする7つの方法
x1 ichi
 
PDF
Practical Testing of Ruby Core
Hiroshi SHIBATA
 
PDF
High Performance tDiary
Hiroshi SHIBATA
 
ZIP
AnyMQ, Hippie, and the real-time web
clkao
 
PDF
JUnit5 and TestContainers
Sunghyouk Bae
 
PPTX
05 pig user defined functions (udfs)
Subhas Kumar Ghosh
 
PPTX
A Functional Guide to Cat Herding with PHP Generators
Mark Baker
 
PDF
Building a High-Performance Distributed Task Queue on MongoDB
MongoDB
 
PDF
RxSwift to Combine
Bo-Young Park
 
PDF
RxSwift to Combine
Bo-Young Park
 
PDF
RubyKaigi2015 making robots-with-mruby
yamanekko
 
PDF
Internship final report@Treasure Data Inc.
Ryuichi ITO
 
PDF
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
PPTX
APIs and Synthetic Biology
Uri Laserson
 
PDF
Using Python3 to Build a Cloud Computing Service for my Superboard II
David Beazley (Dabeaz LLC)
 
PDF
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
PPTX
Running Ruby on Solaris (RubyKaigi 2015, 12/Dec/2015)
ngotogenome
 
Новый InterSystems: open-source, митапы, хакатоны
Timur Safin
 
Kotlin @ Coupang Backed - JetBrains Day seoul 2018
Sunghyouk Bae
 
あなたのScalaを爆速にする7つの方法
x1 ichi
 
Practical Testing of Ruby Core
Hiroshi SHIBATA
 
High Performance tDiary
Hiroshi SHIBATA
 
AnyMQ, Hippie, and the real-time web
clkao
 
JUnit5 and TestContainers
Sunghyouk Bae
 
05 pig user defined functions (udfs)
Subhas Kumar Ghosh
 
A Functional Guide to Cat Herding with PHP Generators
Mark Baker
 
Building a High-Performance Distributed Task Queue on MongoDB
MongoDB
 
RxSwift to Combine
Bo-Young Park
 
RxSwift to Combine
Bo-Young Park
 
RubyKaigi2015 making robots-with-mruby
yamanekko
 
Internship final report@Treasure Data Inc.
Ryuichi ITO
 
Wprowadzenie do technologi Big Data i Apache Hadoop
Sages
 
APIs and Synthetic Biology
Uri Laserson
 
Using Python3 to Build a Cloud Computing Service for my Superboard II
David Beazley (Dabeaz LLC)
 
RestMQ - HTTP/Redis based Message Queue
Gleicon Moraes
 
Running Ruby on Solaris (RubyKaigi 2015, 12/Dec/2015)
ngotogenome
 
Ad

Viewers also liked (20)

PDF
Python于Web 2.0网站的应用 - QCon Beijing 2010
Qiangning Hong
 
KEY
New Design of OneRing
Qiangning Hong
 
PDF
合久必分,分久必合
Qiangning Hong
 
PDF
豆瓣技术架构的发展历程 @ QCon Beijing 2009
Qiangning Hong
 
PDF
Don Bailey - A Million Little Tracking Devices
Source Conference
 
PPT
How To Automate Part 1
Sean Durocher
 
PDF
29c3 OpenBTS workshop - Mini-Workshop
Alexander Chemeris
 
PPTX
DAB+ for local and community radio
radioradioradio
 
PDF
Creating Excel files with Python and XlsxWriter
jmncnamara
 
ODP
20 cool things python
Pippi Labradoodle
 
PPTX
基于Python构建可扩展的自动化运维平台
liuts
 
PDF
Sed Unix Utility Explained
Peter Krumins
 
PPT
Mixing Python and Java
Andreas Schreiber
 
PDF
Python Advanced – Building on the foundation
Kevlin Henney
 
PDF
Java Tutorial:Learn Java in 06:00:00
Justin Lin
 
PPTX
MySQL压力测试经验
Jinrong Ye
 
PDF
高效Linux SA
Jinrong Ye
 
PDF
MySQL数据库设计、优化
Jinrong Ye
 
PDF
MySQL技术分享:一步到位实现mysql优化
Jinrong Ye
 
Python于Web 2.0网站的应用 - QCon Beijing 2010
Qiangning Hong
 
New Design of OneRing
Qiangning Hong
 
合久必分,分久必合
Qiangning Hong
 
豆瓣技术架构的发展历程 @ QCon Beijing 2009
Qiangning Hong
 
Don Bailey - A Million Little Tracking Devices
Source Conference
 
How To Automate Part 1
Sean Durocher
 
29c3 OpenBTS workshop - Mini-Workshop
Alexander Chemeris
 
DAB+ for local and community radio
radioradioradio
 
Creating Excel files with Python and XlsxWriter
jmncnamara
 
20 cool things python
Pippi Labradoodle
 
基于Python构建可扩展的自动化运维平台
liuts
 
Sed Unix Utility Explained
Peter Krumins
 
Mixing Python and Java
Andreas Schreiber
 
Python Advanced – Building on the foundation
Kevlin Henney
 
Java Tutorial:Learn Java in 06:00:00
Justin Lin
 
MySQL压力测试经验
Jinrong Ye
 
高效Linux SA
Jinrong Ye
 
MySQL数据库设计、优化
Jinrong Ye
 
MySQL技术分享:一步到位实现mysql优化
Jinrong Ye
 
Ad

Similar to Python高级编程(二) (20)

PPT
Profiling and optimization
g3_nittala
 
PDF
What’s eating python performance
Piotr Przymus
 
PPTX
Pygrunn 2012 down the rabbit - profiling in python
Remco Wendt
 
PDF
High-Performance Python
Work-Bench
 
PPTX
Down the rabbit hole, profiling in Django
Remco Wendt
 
PDF
(Ebook) High Performance Python by Micha Gorelick, Ian Ozsvald
gamvdbw117
 
PDF
High Performance Python 2nd Edition Micha Gorelick
danuzakhiem
 
PDF
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
PyData
 
PDF
Pythran: Static compiler for high performance by Mehdi Amini PyData SV 2014
PyData
 
PDF
Python faster for loop
💾 Radek Fabisiak
 
PDF
High Performance Python 2nd Edition Micha Gorelick Ian Ozsvald
hunelibuzhan
 
PDF
Optimizing Python
AdimianBE
 
PDF
High Performance Python Practical Performant Programming for Humans 2nd Editi...
cemernania2a
 
PDF
Cluj.py Meetup: Extending Python in C
Steffen Wenz
 
PPT
Euro python2011 High Performance Python
Ian Ozsvald
 
PDF
Functional python
Jesué Junior
 
PDF
Thinking in Functions: Functional Programming in Python
Anoop Thomas Mathew
 
PDF
Python Performance 101
Ankur Gupta
 
PDF
Migrating from matlab to python
ActiveState
 
PDF
PyPy London Demo Evening 2013
Carl Friedrich Bolz
 
Profiling and optimization
g3_nittala
 
What’s eating python performance
Piotr Przymus
 
Pygrunn 2012 down the rabbit - profiling in python
Remco Wendt
 
High-Performance Python
Work-Bench
 
Down the rabbit hole, profiling in Django
Remco Wendt
 
(Ebook) High Performance Python by Micha Gorelick, Ian Ozsvald
gamvdbw117
 
High Performance Python 2nd Edition Micha Gorelick
danuzakhiem
 
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
PyData
 
Pythran: Static compiler for high performance by Mehdi Amini PyData SV 2014
PyData
 
Python faster for loop
💾 Radek Fabisiak
 
High Performance Python 2nd Edition Micha Gorelick Ian Ozsvald
hunelibuzhan
 
Optimizing Python
AdimianBE
 
High Performance Python Practical Performant Programming for Humans 2nd Editi...
cemernania2a
 
Cluj.py Meetup: Extending Python in C
Steffen Wenz
 
Euro python2011 High Performance Python
Ian Ozsvald
 
Functional python
Jesué Junior
 
Thinking in Functions: Functional Programming in Python
Anoop Thomas Mathew
 
Python Performance 101
Ankur Gupta
 
Migrating from matlab to python
ActiveState
 
PyPy London Demo Evening 2013
Carl Friedrich Bolz
 

Recently uploaded (20)

PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PDF
REPORT: Heating appliances market in Poland 2024
SPIUG
 
PDF
Beyond Automation: The Role of IoT Sensor Integration in Next-Gen Industries
Rejig Digital
 
PDF
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
A Day in the Life of Location Data - Turning Where into How.pdf
Precisely
 
PDF
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
PDF
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
Architecture of the Future (09152021)
EdwardMeyman
 
PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
PDF
Event Presentation Google Cloud Next Extended 2025
minhtrietgect
 
PPTX
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
PDF
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
PDF
Advances in Ultra High Voltage (UHV) Transmission and Distribution Systems.pdf
Nabajyoti Banik
 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
PPTX
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PDF
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
REPORT: Heating appliances market in Poland 2024
SPIUG
 
Beyond Automation: The Role of IoT Sensor Integration in Next-Gen Industries
Rejig Digital
 
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
A Day in the Life of Location Data - Turning Where into How.pdf
Precisely
 
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Architecture of the Future (09152021)
EdwardMeyman
 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
Event Presentation Google Cloud Next Extended 2025
minhtrietgect
 
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
Advances in Ultra High Voltage (UHV) Transmission and Distribution Systems.pdf
Nabajyoti Banik
 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
NewMind AI Weekly Chronicles - July'25 - Week IV
NewMind AI
 

Python高级编程(二)

  • 1. Advanced Python Programming (Part II) Happy Day #5 2011.10
  • 2. Topics • Performance Tuning • Garbage Collection • Extending Python
  • 4. Python's Speed Among Most Popular Languages C C++ Java Lisp C# Pascal Python Ruby PHP Perl data source (on Oct.17, 2011): • http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html • http://shootout.alioth.debian.org/u64q/which-programming-languages-are-fastest.php
  • 5. 7 Steps to Gain Speed 1) Find performance bottlenecks 2) Use better algorithms 3) Use faster tools 4) Write optimized code 5) Hire optimizers 6) Write your own extension modules 7) Parallelize the computation
  • 6. Step 0 Is It Fast Enough Already?
  • 8. Find Performance Bottlenecks • Profile, no guess - profile • a pure Python module - cProfile • written in C, new in Python 2.5 • same interface with profile, but lower overhead - hotshot • written in C, new in Python 2.2 • not maintained and might be removed
  • 9. cProfile Usage • cProfile.run('foo()') • cProfile.run('foo()', 'profile.result') • python -m cProfile -o profile.result myscript.py • p = pstats.Stats('profile.result') • p.sort_stats('cumulative').print_stats() • sort by 'cumulative' to find what algorithms are taking time • sort by 'time' to find what functions are taking time • RunSnakeRun for GUI guys • RTFM, please • for IPython, type %prun?
  • 10. Line Profile • line_profile and kernprof @profile def slow_function(): ... $ kernprof.py -l -v script_to_profile.py ... Line # Hits Time Per Hit % Time Line Contents ============================================================== 1 @profile 2 def slow_function(): 3 1 3 3.0 0.2 s = 0 4 1001 934 0.9 48.6 for i in xrange(1000): 5 1000 984 1.0 51.2 s += i 6 1 1 1.0 0.1 return s
  • 11. Step 2 Use Better Algorithms
  • 12. How To Know Which is Better? • timeit! • python -m timeit -s "setup" "statement" • e.g. which is faster, "d.has_key(k)" or "k in d"? $ python -m timeit -s "d=dict(zip(range(1000), range(1000)))" "d.has_key(500)" 1000000 loops, best of 3: 0.223 usec per loop $ python -m timeit -s "d=dict(zip(range(1000), range(1000)))" "500 in d" 10000000 loops, best of 3: 0.115 usec per loop
  • 13. Use Bettern Algorithms • How to calculate sum([1, 2, ..., 100])?
  • 14. Use Bettern Algorithms • How to calculate sum([1, 2, ..., 100])? s = 0 for i in range(101): 8.3usec s += i
  • 15. Use Bettern Algorithms • How to calculate sum([1, 2, ..., 100])? s = 0 for i in range(101): 8.3usec s += i s = sum(range(101)) 2.8usec
  • 16. Use Bettern Algorithms • How to calculate sum([1, 2, ..., 100])? s = 0 for i in range(101): 8.3usec s += i s = sum(range(101)) 2.8usec s = sum(xrange(101)) 2.03usec
  • 17. Use Bettern Algorithms • How to calculate sum([1, 2, ..., 100])? s = 0 for i in range(101): 8.3usec s += i s = sum(range(101)) 2.8usec s = sum(xrange(101)) 2.03usec s = (1 + 100) * 100 / 2 0.109usec
  • 18. Advanced Data Types • membership testing: • set & dict: O(1) vs. tuple & list: O(n) • return iterator instead of a large list • array, collections.deque, heapq, bisect
  • 19. Examples lst = [] for i in xrange(10000): lst.insert(0, i) lst = collections.deque() for i in xrange(10000): 25317% faster lst.appendleft(i) sorted(lst, reverse=True)[:10] heapq.nlargest(10, lst) 613% faster
  • 20. Do Less Computation • Pre-computation • Lazy computation • Cache • Approximation Algorithms
  • 21. Example def fib(n): if n <= 1: return 1 fib(25): 59.8ms return fib(n-2) + fib(n-1)
  • 22. Example def cache(func): c = {} def _(n): r = c.get(n) fib(25): 59.8ms if r is None: r = c[n] = func(n) return r return _ with @cache: @cache fib(25): 0.524us def fib(n): if n <= 1: return 1 112000 times faster! return fib(n-2) + fib(n-1)
  • 24. Use Faster Tools • use iterator form • range() -> xrange() • map() -> itertools.imap() • list comprehension -> generator expression • dict.items() -> dict.iteritems() • for i in range(len(seq)): -> • for item in seq: • for i, item in enumerate(seq):
  • 25. Use Faster Tools • SAX is faster and memory efficient than DOM • use C version of modules • profile -> cProfile • StringIO -> cStringIO • pickle -> cPickle • elementTree -> cElementTree / lxml • select has lower overhead than poll (and epoll at low number of connections) • numpy is essential for high volume numeric work
  • 26. numpy Example from itertools import izip a=range(1000) b=range(1000) c = [ai+bi for ai, bi in izip(a, b)] import numpy a=numpy.arange(1000) b=numpy.arange(1000) c = a + b
  • 28. Write Optimized Code • Less temporary objects • e.g. accumulator vs. sum • however, string concatenation has been optimized after Python 2.5
  • 29. Write Optimized Code • use key= instead of cmp= when sorting lst = open('/Users/hongqn/projects/shire/luzong/ group.py').read().split() lst.sort(cmp=lambda x, y: cmp(x.lower(), y.lower())) lst.sort(key=str.lower) 377% faster
  • 30. Write Optimized Code • local variables are faster than global variables def f(): for i in xrange(10000): r = abs(i) def f(): _abs = abs for i in xrange(10000): r = _abs(i) 28% faster • you can eliminate dots, too
  • 31. Write Optimized Code • inline function inside time-critical loops def f(x): return x + 1 for i in xrange(10000): r = f(i) for i in xrange(10000): r = i + 1 187% faster
  • 32. Write Optimized Code • do not import modules in loops for i in xrange(10000): import string r = string.lower('Python') import string for i in xrange(10000): r = string.lower('Python') 178% faster
  • 33. Write Optimized Code • list comprehensions are faster than for- loops lst = [] for i in xrange(10000): lst.append(i) lst = [i for i in xrange(10000)] 213% faster
  • 34. Write Optimized Code • use "while 1" for time-critical loops (readability lost!) a = 0 while True: a += 1 if a > 10000: break a = 0 while 1: a += 1 if a > 10000: 78% faster break
  • 35. Write Optimized Code • "not not x" is faster than "bool(x)" (not recommended!) bool([]) not not [] 196% faster
  • 37. Hire Optimizers • sys.setcheckinterval() • Python checks for thread switch and signal handling periodly (default 100 python virtual instructions) • set it to a larger value for better performance in cost of responsiveness
  • 38. Hire Optimizers • gc.disable() • disable automatic garbage collection • gc.set_threshold() • collect less frequently
  • 39. Hire Optimizers • Psyco • JIT for 32bit only. <=Python 2.6 • PyPy • Alternative implementation of Python • Shed Skin • Python-to-C++ compiler • numexpr • numpy expression evaluator
  • 40. Step 6 Write Your Own Extension Modules
  • 41. Write Your Own Extension Modules • Python/C API • Official API • ctypes • Call dynamic link library in Python • SWIG • Automatically generate interface code • Pyrex / Cython • write extension using Python-like language • Boost.Python • C++ API • Weave • Inline C code
  • 42. Write Your Own Extension Modules • C-level Optimization
  • 43. Step 7 Parallelize the Computation
  • 44. Over CPUs • multi-threading • threading (be careful with GIL!) • multi-processing • fork • subprocess • multiprocessing • async • asyncore • twisted • greenlet/gevent • PyOpenCL / PyCUDA
  • 45. multiprocessing Example sum(xrange(1, 10000001)) 172ms from multiprocessing import Pool pool = Pool() sum(pool.map(sum, (xrange(i, i+1000000) for i in xrange(1, 10000000, 1000000)))) 104ms on dual-core 49ms on 8-core
  • 46. Over Cluster • XML-RPC / Json-RPC / Thrift / Protocol Buffer • Pyro • Parallel Python • dumbo on Hadoop • dpark
  • 47. Gold Rule Premature optimization is the root of all evil. -- Donald Knuth
  • 49. What Is Garbage • An object which will not be used in any future • i.e. no other object refers to it
  • 50. Why Collecting Garbage? • re-sell it (recycle) • make programs simpler
  • 51. Garbage Collection in Everywhere • by lilinghui, Sep.20, 2011 • http://svn.douban.com/projects/shire/wiki/ HallOfFire/Platform
  • 52. Basic Garbage Collection Algorithms • Reference couting • Mark-and-sweep • Mark-and-compact • Copy
  • 53. Garbage Collection in CPython • Reference couting • Mark-and-sweep
  • 54. Reference Counting def f(): a = A() a.b = B() a.b.c = C() a.c = b.c
  • 55. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a b a.c = b.c 1 1 c 2
  • 56. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a b a.c = b.c 1 1 del a.c c 2
  • 57. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a b a.c = b.c 1 1 del a.c c 1
  • 58. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a b a.c = b.c 1 1 del a.c del a.b c 1
  • 59. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a b a.c = b.c 1 0 del a.c del a.b c 1
  • 60. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a a.c = b.c 1 del a.c del a.b c 0
  • 61. Reference Counting def f(): frame a = A() a.b = B() a.b.c = C() a a.c = b.c 1 del a.c del a.b
  • 62. Reference Counting def f(): a = A() a.b = B() a.b.c = C() a a.c = b.c 0 del a.c del a.b
  • 63. Reference Counting def f(): a = A() a.b = B() a.b.c = C() a.c = b.c del a.c del a.b
  • 64. Pros & Cons of Reference Counting • Pros • collect early • predictable run-time behavior • Cons • slow • circle reference
  • 65. Circle Reference def f(): frame a = A() a.b = B() a.b.c = C() a b a.b.c.b = a.b 1 2 c 1
  • 66. Circle Reference def f(): frame a = A() a.b = B() a.b.c = C() a b a.b.c.b = a.b 1 2 del a.b c 1
  • 67. Circle Reference def f(): frame a = A() a.b = B() a.b.c = C() a b a.b.c.b = a.b 1 1 del a.b c 1
  • 68. Circle Reference def f(): a = A() a.b = B() a.b.c = C() b a.b.c.b = a.b 1 del a.b c 1
  • 69. Weak Reference a reference not strong enough to keep an object alive >>> import weakref import weakref >>> class Object: ... pass _id2obj_dict = weakref.WeakValueDictionary() ... >>> o = Object() def remember(obj): >>> r = weakref.ref(o) oid = id(obj) >>> o2 = r() _id2obj_dict[oid] = obj >>> o is o2 return oid True >>> del o, o2 def id2obj(oid): >>> print r() return _id2obj_dict[oid] None
  • 70. Mark-and-Sweep frame 1. Mark root objects and all their referents reachable a b 2. Sweep unreachable 1 1 objects c 1
  • 71. Generations • generations and thresholds • generation 0 (youngest): 700 • generation 1 (middle):10 • generation 2 (oldest): 10 • and long_lived_pending / long_lived_total > 25% (Python 2.7+)
  • 72. Python's Optimization • Track only container objects • Use referent count to find root objects
  • 73. __del__ • circle referenced objects with __del__() methods will be put in gc.garbage • do not use __del__
  • 74. GC Module • gc.enable() / gc.disable() • gc.collect() • gc.get_threshold() / gc.set_threshold() • gc.set_debug() • gc.get_referers() / gc.get_referents() • gc.get_objects()
  • 77. #include <Python.h> static PyObject *SpamError; static PyObject * spam_system(PyObject *self, PyObject *args) { const char *command; int sts; if (!PyArg_ParseTuple(args, "s", &command)) return NULL; sts = system(command); if (sts < 0) { PyErr_SetString(SpamError, "System command failed"); return NULL; } return PyLong_FromLong(sts); } static PyMethodDef SpamMethods[] = { {"system", spam_system, METH_VARARGS, "Execute a shell command."}, {NULL, NULL, 0, NULL} /* Sentinel */ }; PyMODINIT_FUNC initspam(void) { PyObject *m; m = Py_InitModule("spam", SpamMethods); if (m == NULL) return; SpamError = PyErr_NewException("spam.error", NULL, NULL); Py_INCREF(SpamError); PyModule_AddObject(m, "error", SpamError); }
  • 78. Building with setuptools from setuptools import setup, Extension module1 = Extension('spam', sources = ['spam.c']) setup (name = 'PackageName', version = '1.0', description = 'This is a demo package', ext_modules = [module1]) python setup.py install python setup.py build_ext --inplace
  • 79. Parse and Build Values • int PyArg_ParseTuple(PyObject *args, const char *format, ...) • int PyArg_ParseTupleAndKeywords(PyObject *args, PyObject *kw, const char *format, char *keywords[], ...) • int PyArg_BuildValue(const char *format, ...)
  • 80. Manage Reference Counting • Py_INCREF(PyObject *o) / Py_XINCREF(PyObject *o) • Py_DECREF(PyObject *o) / Py_XINCREF(PyObject *o) / Py_CLEAR(PyObject *o) • New/Borrowed/Stealing references
  • 81. Exception Handling • errno-like mechanism • PyErr_SetString() / PyErr_SetObject() to set error • PyErr_Occurred() to check error • Most functions return an error indicator, e.g. NULL, -1
  • 82. Global Interpreter Lock • Release GIL before running blocking C code Py_BEGIN_ALLOW_THREADS ...Do some blocking I/O operation... Py_END_ALLOW_THREADS • Reacquire GIL before calling into Python functions PyGILState_STATE gstate; gstate = PyGILState_Ensure(); /* Perform Python actions here. */ result = CallSomeFunction(); /* evaluate result */ /* Release the thread. No Python API allowed beyond this point. */ PyGILState_Release(gstate);
  • 84. Python Style Code from libc.stdlib cimport system as c_system class error(Exception): pass def system(char* cmd): cdef int retval = c_system(cmd) if retval < 0: raise error("System command failed") return retval
  • 85. setup.py from setuptools import setup, Extension from Cython.Distutils import build_ext extmod = Extension("spam", sources=['spam.pyx']) setup( name="Spam", cmdclass = {'build_ext': build_ext}, ext_modules = [extmod])
  • 86. #include <Python.h> #include "structmember.h" static PyGetSetDef Noddy_getseters[] = { Extension Types typedef struct { {"first", PyObject_HEAD (getter)Noddy_getfirst, (setter)Noddy_setfirst, PyObject *first; "first name", PyObject *last; NULL}, int number; {"last", } Noddy; (getter)Noddy_getlast, (setter)Noddy_setlast, "last name", static void NULL}, Noddy_dealloc(Noddy* self) {NULL} /* Sentinel */ { }; Py_XDECREF(self->first); Py_XDECREF(self->last); static PyObject * self->ob_type->tp_free((PyObject*)self); Noddy_name(Noddy* self) } { static PyObject *format = NULL; static PyObject * PyObject *args, *result; Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { if (format == NULL) { Noddy *self; format = PyString_FromString("%s %s"); if (format == NULL) self = (Noddy *)type->tp_alloc(type, 0); return NULL; if (self != NULL) { } self->first = PyString_FromString(""); if (self->first == NULL) args = Py_BuildValue("OO", self->first, self->last); { if (args == NULL) Py_DECREF(self); return NULL; return NULL; } result = PyString_Format(format, args); Py_DECREF(args); self->last = PyString_FromString(""); if (self->last == NULL) return result; { } Py_DECREF(self); return NULL; static PyMethodDef Noddy_methods[] = { } {"name", (PyCFunction)Noddy_name, METH_NOARGS, "Return the name, combining the first and last name" self->number = 0; }, } {NULL} /* Sentinel */ }; return (PyObject *)self; } static PyTypeObject NoddyType = { PyObject_HEAD_INIT(NULL) static int 0, /*ob_size*/ Noddy_init(Noddy *self, PyObject *args, PyObject *kwds) "noddy.Noddy", /*tp_name*/ { sizeof(Noddy), /*tp_basicsize*/ PyObject *first=NULL, *last=NULL, *tmp; 0, /*tp_itemsize*/ (destructor)Noddy_dealloc, /*tp_dealloc*/ static char *kwlist[] = {"first", "last", "number", NULL}; 0, /*tp_print*/ 0, /*tp_getattr*/ if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist, 0, /*tp_setattr*/ &first, &last, 0, /*tp_compare*/ &self->number)) 0, /*tp_repr*/ return -1; 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ if (first) { 0, /*tp_as_mapping*/ tmp = self->first; 0, /*tp_hash */ Py_INCREF(first); 0, /*tp_call*/ self->first = first; 0, /*tp_str*/ Py_DECREF(tmp); 0, /*tp_getattro*/ } 0, /*tp_setattro*/ 0, /*tp_as_buffer*/ if (last) { Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ tmp = self->last; "Noddy objects", /* tp_doc */ Py_INCREF(last); 0,! ! /* tp_traverse */ self->last = last; 0,! ! /* tp_clear */ Py_DECREF(tmp); 0,! ! /* tp_richcompare */ } 0,! ! /* tp_weaklistoffset */ 0,! ! /* tp_iter */ return 0; 0,! ! /* tp_iternext */ } Noddy_methods, /* tp_methods */ Noddy_members, /* tp_members */ static PyMemberDef Noddy_members[] = { Noddy_getseters, /* tp_getset */ {"number", T_INT, offsetof(Noddy, number), 0, 0, /* tp_base */ "noddy number"}, 0, /* tp_dict */ {NULL} /* Sentinel */ 0, /* tp_descr_get */ }; 0, /* tp_descr_set */ 0, /* tp_dictoffset */ static PyObject * (initproc)Noddy_init, /* tp_init */ Noddy_getfirst(Noddy *self, void *closure) 0, /* tp_alloc */ { Noddy_new, /* tp_new */ Py_INCREF(self->first); }; return self->first; } static PyMethodDef module_methods[] = { {NULL} /* Sentinel */ static int }; Noddy_setfirst(Noddy *self, PyObject *value, void *closure) { #ifndef PyMODINIT_FUNC! /* declarations for DLL import/export */ if (value == NULL) { #define PyMODINIT_FUNC void PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute"); #endif return -1; PyMODINIT_FUNC } initnoddy3(void) { if (! PyString_Check(value)) { PyObject* m; PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string"); if (PyType_Ready(&NoddyType) < 0) return -1; return; } m = Py_InitModule3("noddy3", module_methods, Py_DECREF(self->first); "Example module that creates an extension type."); Py_INCREF(value); self->first = value; if (m == NULL) return; return 0; } Py_INCREF(&NoddyType); PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType); static PyObject * } Noddy_getlast(Noddy *self, void *closure) { Py_INCREF(self->last); return self->last; } static int Noddy_setlast(Noddy *self, PyObject *value, void *closure) { if (value == NULL) { PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute"); return -1; } if (! PyString_Check(value)) { PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string"); return -1; } Py_DECREF(self->last); Py_INCREF(value); self->last = value; return 0; }
  • 87. #include <Python.h> #include "structmember.h" static PyGetSetDef Noddy_getseters[] = { Extension Types typedef struct { {"first", PyObject_HEAD (getter)Noddy_getfirst, (setter)Noddy_setfirst, PyObject *first; "first name", PyObject *last; NULL}, int number; {"last", } Noddy; (getter)Noddy_getlast, (setter)Noddy_setlast, "last name", static void NULL}, Noddy_dealloc(Noddy* self) {NULL} /* Sentinel */ { }; Py_XDECREF(self->first); Py_XDECREF(self->last); static PyObject * self->ob_type->tp_free((PyObject*)self); Noddy_name(Noddy* self) } { static PyObject *format = NULL; static PyObject * PyObject *args, *result; Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { if (format == NULL) { Noddy *self; format = PyString_FromString("%s %s"); if (format == NULL) self = (Noddy *)type->tp_alloc(type, 0); return NULL; if (self != NULL) { } self->first = PyString_FromString(""); if (self->first == NULL) args = Py_BuildValue("OO", self->first, self->last); { if (args == NULL) Py_DECREF(self); return NULL; return NULL; } result = PyString_Format(format, args); Py_DECREF(args); self->last = PyString_FromString(""); if (self->last == NULL) return result; { Py_DECREF(self); } cdef class Noddy: return NULL; static PyMethodDef Noddy_methods[] = { } {"name", (PyCFunction)Noddy_name, METH_NOARGS, "Return the name, combining the first and last name" }, """Noddy objects""" self->number = 0; } {NULL} /* Sentinel */ }; return (PyObject *)self; } static int static PyTypeObject NoddyType = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ cdef object first, last "noddy.Noddy", /*tp_name*/ Noddy_init(Noddy *self, PyObject *args, PyObject *kwds) { PyObject *first=NULL, *last=NULL, *tmp; sizeof(Noddy), 0, /*tp_basicsize*/ /*tp_itemsize*/ cdef public int number (destructor)Noddy_dealloc, /*tp_dealloc*/ static char *kwlist[] = {"first", "last", "number", NULL}; 0, /*tp_print*/ 0, /*tp_getattr*/ if (! PyArg_ParseTupleAndKeywords(args, kwds, "|SSi", kwlist, 0, /*tp_setattr*/ &first, &last, &self->number)) 0, 0, 0, /*tp_compare*/ /*tp_repr*/ /*tp_as_number*/ def __cinit__(self, str first="", str last="", int number=0): return -1; if (first) { 0, 0, 0, /*tp_as_sequence*/ /*tp_as_mapping*/ /*tp_hash */ self.first = first tmp = self->first; Py_INCREF(first); self->first = first; Py_DECREF(tmp); 0, 0, 0, /*tp_call*/ /*tp_str*/ /*tp_getattro*/ self.last = last 0, /*tp_setattro*/ } if (last) { 0, /*tp_as_buffer*/ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ self.number = number tmp = self->last; "Noddy objects", /* tp_doc */ Py_INCREF(last); 0,! ! /* tp_traverse */ self->last = last; 0,! ! /* tp_clear */ Py_DECREF(tmp); 0,! ! /* tp_richcompare */ } 0,! 0,! 0,! ! ! ! /* tp_weaklistoffset */ /* tp_iter */ /* tp_iternext */ property first: return 0; } Noddy_methods, Noddy_members, Noddy_getseters, /* tp_methods */ /* tp_members */ /* tp_getset */ """first name""" static PyMemberDef Noddy_members[] = { {"number", T_INT, offsetof(Noddy, number), 0, "noddy number"}, {NULL} /* Sentinel */ 0, 0, 0, /* tp_base */ /* tp_dict */ /* tp_descr_get */ def __get__(self): 0, /* tp_descr_set */ }; static PyObject * 0, (initproc)Noddy_init, /* tp_dictoffset */ /* tp_init */ return self.first Noddy_getfirst(Noddy *self, void *closure) 0, /* tp_alloc */ { Py_INCREF(self->first); }; Noddy_new, /* tp_new */ def __set__(self, str value): return self->first; } static PyMethodDef module_methods[] = { }; {NULL} /* Sentinel */ self.first = value static int Noddy_setfirst(Noddy *self, PyObject *value, void *closure) { #ifndef PyMODINIT_FUNC! /* declarations for DLL import/export */ if (value == NULL) { #define PyMODINIT_FUNC void } PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute"); return -1; #endif PyMODINIT_FUNC initnoddy3(void) property last: { if (! PyString_Check(value)) { PyErr_SetString(PyExc_TypeError, PyObject* m; """last name""" "The first attribute value must be a string"); if (PyType_Ready(&NoddyType) < 0) } return -1; return; def __get__(self): m = Py_InitModule3("noddy3", module_methods, Py_DECREF(self->first); Py_INCREF(value); if (m == NULL) "Example module that creates an extension type."); return self.last self->first = value; return 0; return; Py_INCREF(&NoddyType); def __set__(self, str value): } static PyObject * Noddy_getlast(Noddy *self, void *closure) } PyModule_AddObject(m, "Noddy", (PyObject *)&NoddyType); self.last = value { Py_INCREF(self->last); return self->last; } static int def name(self): Noddy_setlast(Noddy *self, PyObject *value, void *closure) { if (value == NULL) { """Return the name, combining the first and last name""" PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute"); } return -1; return "%s %s" % (self.first, self.last) if (! PyString_Check(value)) { PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string"); return -1; } Py_DECREF(self->last); Py_INCREF(value); self->last = value; return 0; }
  • 88. GIL with nogil: ... cdef void my_callback(void *data) with gil: ... cdef void my_gil_free_func(int spam) nogil: ...
  • 90. Call C Function in Python from ctypes import cdll libc = cdll.LoadLibrary('libc.dylib') class error(Exception): pass def system(cmd): retval = libc.system(cmd) if retval < 0: raise error("System command failed.") return retval
  • 91. GIL • ctypes.cdll: Release GIL • ctypes.pydll: With GIL
  • 92. Q &A