Exploring Python
================

Simon Willison
Stack Overflow DevDays
Amsterdam 2 Nov 2009

------------------------------------

Python fundamentals

- Dynamic language
  (not scripting, not weakly typed)
- General purpose
- Compiles to bytecode
- Significant whitespace
  (get over it already!)
- 18 years old
- Multiple implementations
  CPython, Jython, IronPython, PyPy
- Open source

In an interactive prompt:

$ python
>>> print "hello world"
'hello world'
>>> import urllib
>>> u = urllib.urlopen("http://stackoverflow.carsonified.com/")
>>> u.read()
...
>>> u.geturl()
...
>>> u.info()
...
>>> import this # the Zen of Python

The rest of the demos used iPython: http://ipython.scipy.org/

$ ipython # some of the below will throw errors, that's OK
>>> d = {'dog': 1, 'cat': 123, ('foo', 'bar'): {'moo': 'foo'}}
>>> t = (1, 2, 45, 12, 'dog')
>>> l = ['dog', 'cat', 12, ('a', 'b')]
>>> s = set(['a', 'b', 'c'])
>>> l
>>> t
>>> del l[0]
>>> l
>>> del t[0]
>>> l[0]
>>> l[1]
>>> len(l)
>>> for item in l:
...     print item
>>> d['cat']
>>> d[('foo', 'bar')]
>>> d[[234, 234]] = 12
>>> s
>>> 'a' in s
>>> 'oenuth' in s

------------------------------------

Next I talked about the BNP mapping project:

www.guardian.co.uk/politics/2009/oct/20/bnp-membership-list-wikileaks
www.guardian.co.uk/politics/interactive/2009/oct/19/bnp-membership-uk-map

It turns out these are called http://en.wikipedia.org/wiki/Choropleth_map

Then I showed how to make a world-wide Choropleth using the Guardian's data 
on numbers of endangered species (in particular mammals).

http://en.wikipedia.org/wiki/File:BlankMap-World6.svg was the source SVG map

www.guardian.co.uk/environment/datablog/2009/oct/23/endangered-species-red-list-data-review
http://spreadsheets.google.com/ccc?key=tYrEXu8zzrPw_b6C775X-iA

Or grab the CSV directly with this link:

http://spreadsheets.google.com/pub?key=tYrEXu8zzrPw_b6C775X-iA&output=csv

>>> import csv
>>> reader = csv.reader('endangered.csv')
>>> headers = reader.next()
>>> row = reader.next()
>>> zip(headers, row)
>>> dict(_)
>>> reader = csv.reader('endangered.csv')
>>> headers = reader.next()
>>> headers[1] = 'ISO'
>>> headers[2] = 'Mammals'
>>> rows = []
>>> for row in reader:
...     rows.append(dict(zip(headers, row)))
>>> mammals = [
...    (r['ISO'], int(r['Mammals']))
...    for r in rows
... ]
>>> mammals.sort(key = lambda p: p[1])

In colourer.py:

class Colourer(object):
    def __init__(self, min_col, max_col):
        self.min_col = min_col
        self.max_col = max_col
    
    def colour(self, ratio):
        return (
            self.min_col[0] + ratio * (
                self.max_col[0] - self.min_col[0]
            ),
            self.min_col[1] + ratio * (
                self.max_col[1] - self.min_col[1]
            ),
            self.min_col[2] + ratio * (
                self.max_col[2] - self.min_col[2]
            ),
        )

>>> c = colourer.Colourer((255, 255, 255), (255, 0, 0))
>>> from xml.etree import ElementTree
>>> et = ElementTree.parse('BlankMap-World6.svg')
>>> et.getroot()
>>> et.getroot().find('./{http://www.w3.org/2000/svg}style')
>>> style = _
>>> print style.text
>>> def tohex(rgb):
...     return '%02x%02x%02x' % rgb
>>> template = '.%s { fill: #%s; }'
>>> css = []
>>> for iso, value in mammals:
...     if not iso:
...         continue
...     rgb = c.colour(value / 183.0)
...     css.append(template % (iso.lower(), tohex(rgb)))
>>> rules = '\n'.join(css)
>>> style.text += rules
>>> et.write('output.svg')
>>> !open -a Firefox output.svg

The USA and Alaska showed up in grey due to an issue in the SVG file - I fixed 
this by knocking out the style rule using "Inspect Element" in Firebug.

Half way through the above I digressed to talk about list comprehensions:

    numbers = [1, 3 ,4]
    doubles = [i * 2 for i in numbers]
    odds = [i for i in numbers if i % 2 == 1]
    [transform for item in iterable]
    [transform for item in iterable if condition]

------------------------------------

Next I talked about PyPI, easy_install and virtualenv:

http://pypi.python.org/
http://peak.telecommunity.com/DevCenter/setuptools
http://pypi.python.org/pypi/virtualenv

I didn't mention it in the talk, but I would strongly recommend looking in to
pip as an alternative to the easy_install command:

http://pypi.python.org/pypi/pip

------------------------------------

The next example was a screen scraper for producing iCal files from the 
conference website:

http://stackoverflow.carsonified.com/events/amsterdam/

The demo failed because I installed the wrong version of the library. I should 
have used the latest version from http://codespeak.net/icalendar/

The code also used BeautifulSoup: 
    http://www.crummy.com/software/BeautifulSoup/

Here's the final code:

    from BeautifulSoup import BeautifulSoup as Soup
    import urllib
    from dateutil import parser
    import icalendar
    
    URL = 'http://stackoverflow.carsonified.com/events/amsterdam/'
    
    def scrape_ical(url):
        soup = Soup(urllib.urlopen(url))
        trs = soup.findAll('tr')
        day = trs[0].findAll('th')[1].contents[0] + ' '
        trs = trs[1:]
        cal = icalendar.Calendar()
        for tr in trs:
            time_td = tr.findAll('td')[0]
            summary_td = tr.findAll('td')[1]
            start, end = time_td.contents[0].split(' - ')
            summary = ''.join(summary_td.findAll(text=True))
            event = icalendar.Event()
            event.add('summary', summary)
            event.add('dtstart', parser.parse(day + start))
            event.add('dtend', parser.parse(day + end))
            cal.add_component(event)
        return cal
    
    print scrape_ical(URL).as_string()

------------------------------------

Next I introduced Django:

http://www.djangoproject.com/

I demonstrated a Django implementation of the Stack Overflow DevDays site - 
the code is available here: http://github.com/simonw/devdays/

------------------------------------

Advanced Topics

- Generators
  infinite iterators by pausing and resuming
  a function

- Metaclasses
  A kind of constructor-for-classes - modify
  how a class itself is constructed at runtime

- Coroutines
  Building microthreads on top of generators

I strongly recommend the David Beazley tutorials:

http://www.dabeaz.com/generators-uk/index.html
http://www.dabeaz.com/coroutines/index.html

------------------------------------

The Future of Python

http://www.python.org/download/releases/3.1.1/
http://mail.python.org/pipermail/python-ideas/2009-October/006305.html
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan

------------------------------------

Further reading

- Official Python Tutorial
  http://docs.python.org/tutorial/index.html
- Dive Into Python
  http://diveintopython.org/
- Dive Into Python 3
  http://diveintopython3.org/
- http://simonwillison.net/tags/python/