Now and then, I'd like to invite to the stage Simon Willison,
who very kindly has offered to fill in the space
and fill a much wider space.
He normally accommodates some scheduling problems.
And he will be talking to us
about cowboy development with Django.
(audience applauding)
- Howdy, y'all.
Howdy.
I've called this talk Cowboy Development with Django
because I'm going to talk about two projects
I've recently completed using the cowboy
software development process.
This isn't Agile, this doesn't involve
test-driven development and all of that stuff.
No, this is all about hacking things together
as fast as possible and hoping that they're
sort of crossing your fingers and hoping
that they'll hold up when faced with the world.
But more importantly, having a cowboy theme
to the talk means I get to have a cowboy theme tune.
[Music]
I can ride the pony if you like.
[Music]
Maybe that should have been our logo.
[Music]
So, I haven't seen the film, I'm largely informed that the coffin's got a machine gun in it.
Which is kind of awesome.
[Music]
There's just one problem.
We didn't have cowboys in England, and I am English.
What we had instead was the Napoleonic Wars.
Do you see what I did there?
19th century, you guys were all racing around your prairies
with your lassos and your bandits and all of that.
We don't have prairies, so we had to make do
with fighting the French, which is a longstanding
English tradition.
And if you're going to fight the Napoleonic Wars,
this is the most appropriate transition ever,
Obviously you need a Napoleonic sea fort.
So, the first project I want to talk about you
was built on this very fort here,
in an event that we called Super Evil Dev Fort.
Essentially what happened is,
we had a bar camp in London,
we often have bar camps in London,
and a friend of mine stuck up a slide
with a photograph of that fort,
and said that he'd just been on holiday there,
'cause you can rent it out.
And so naturally, a bunch of us said,
oh my God, we should totally rent a fort
and take like 12 nerds out there and do nerdy things
and make it the nerdiest FortNerd,
that would be really cool.
And then we thought, actually that would be really cool.
So my friend James Eilert rented the Fort.
Turns out you can rent the place for a week for 1,600 pounds
which between 10 people works out at like the cost
of two nights in a hotel room.
And we set out last November with 12 geeks
and as many laptops and cables
and network things as possible.
We set out to the Fort to build a website.
This fort, this is Fort Clonk in Alderney.
It's awesome, it has a gatehouse, a soldiers quarters.
There's a German casement,
that's a genuine Nazi machine gun nest
built by genuine Nazis.
They took over the island in the 1940s
'cause they thought if it's got a seafort there,
it must be important.
Turns out the seafort itself
was never actually used for anything.
So that's, this is Alderney, the island it's on.
That in the bottom, that's our seafort, Fort Clonk.
The fort has a causeway which gets flooded
twice a day, meaning you're completely cut off
from the mainland.
It goes without saying the fort does not have
an internet connection.
This meant that our productivity went up
by about a factor of 20.
You get a fully equipped kitchen.
The bedrooms, that bedroom, the lower bedroom there
is actually in the Nazi machine gun placement.
The machine gun nest is now a picturesque window
looking out over the English Channel.
(audience laughing)
And so we spent a whole bunch of time,
well we spent most of our time hacking, to be honest,
'cause the weather was terrible.
But it turns out, terrible weather and a fort
with six foot walls is actually a really good combination.
And this is the main sort of living area
where we all hacked on sofas for a week.
Down there, this is one of the conditions
of renting the fort is that you have to raise a flag
once a day and play the bugle when you do.
So that's my fiancee there playing the bugle.
She was surprisingly good at it.
And these are the sort of nice happy group photos
of us all enjoying ourselves on the fort.
But what we built, I'll show you the site we built
because we actually, in fact,
so the site we built,
it's a site called wildlifenearyou.com.
This is the site that took a week to build
and then an extra 10 months to actually get it
to a point where we could launch it.
This is the downside of cowboy development, if you like.
And so I'll show you a demo.
This is, it's the 80/20 thing.
Now 20% of the functionality takes 80% of the time.
And like I said, your productivity
when you're off the fork goes down somewhat.
So I'll switch on mirroring and show you what the site is.
So, wildlifenearyou.com is a site that solves the critically important question,
"Where is my nearest otter?"
So, we're in Portland.
I can type "otters near Portland" into the search box,
and apparently our nearest otter is 4,885 miles away
in Sheppard Wildlife Park in Boyston.
The flip side of the site is this is a classic user-generated content site.
Until someone in America enters an otter,
we're going to keep on sending you over to England to find otters.
So there are two aspects to the site. There's a search engine for, say, tigers or llamas
near Brighton's a good one, because there happen to be llamas near Brighton that we
know about. There you go. If you're in Brighton, your nearest llama is at Ashdown Forest Llama
Farm, which I haven't been to yet, but it sounds amazing.
And so the flip side of Wildlife.nu is you sign up for the site, and you please don't
tweet a link to this gist yet. I'll explain why in a moment. And you can say you've been
to London Zoo and it'll say "Do you mean this London Zoo?" and you say "Yes" and then you
tell it "I saw a panda and I saw a... give me an animal."
"Gibbon."
A gibbon.
That's a good one.
Good animal.
"I saw a gibbon."
Oh, it was a Mullers-Bornean gibbon, actually.
And then you click "Finish adding this trip" and you can say the date you took the trip,
if you thought the zoo was nice or horrible, your rating for the trip, sort of one to five
stars and so forth.
And once you've done that, it ends up on your trips page,
so you can collect all of the trips.
It's like Flickr for trips to the zoo, if you like.
And so we built most of this on the fort,
and then we added a few extra things
like the import your photos from Flickr integration
once we got home, 'cause it's quite hard
to import from Flickr when you can't talk to Flickr.
And the reason I ask you not to tweet it
is the site is live, as of a few days ago.
We don't want to start promoting it widely
until we've got some data in it.
We want otters near Portland to have something within a thousand miles maybe.
But please, if you go to alpha.worldlefneru.com, you can use the site right now, enter your
trips to the zoo.
I'm hoping to arrange a trip to Portland Zoo on Friday or Saturday or something.
It's all good.
But this did lead to a slight problem for the whole super evil dev fort concept.
Building a website about fluffy animals isn't particularly evil.
So we've since renamed super evil dev fort.
It was named, of course, after Super Happy Dev House,
which is this Californian thing where a bunch of happy people
get together and they're happy in a house where they hack together
and it's all very happy.
So we're now just called Dev Fort.
We've done another event since then.
We've got another one coming up soon.
And I could not recommend this more as a way of getting out of the creative block
or just having an enormous amount of fun.
It turns out the world, at least Europe, is littered with old castles
and old buildings that you can rent.
I imagine you've probably got stuff that's about 150 years old here, which is ancient
by your standards.
A couple of interesting things about Wildlife.nu.
The search feature is powered by Zapian.
Zapian is an open source search engine.
The reason we use Zapian is that one of the developers of Zapian was on the fort with
us, which was really convenient.
So he actually built us a solar-style web service interface to Zapian.
Our species database is from Freebase.
We've got the Flickr import thing.
The most interesting feature, which I forgot to demo you,
is you can suggest changes to our zoo listings.
You can say, actually, this zoo's phone number is this,
or the address has changed to this, or whatever.
And we have a model for representing
proposed changes to other models,
which we then stash in our database
and so we have a moderation queue and stuff like that.
So that's DevFort.
This is cowboy development on holiday,
which is fair enough, you're on holiday,
nobody can tell you how to develop.
But how can you go about getting away
the cowboy development at work?
That's obviously the really important question here.
So, for those of you who don't know,
I work for The Guardian newspaper.
The Guardian is one of the major national newspapers
in England, it has an international reach,
30 million unique visitors a month to the website.
Really fantastic engineering team.
They built from scratch a custom Java Oracle CMS.
They built it on time and on budget,
which for Enterprise Java I think
is quite frankly a miracle.
And the things, I've never seen the thing go down
and it serves enormous amounts of traffic.
So I joined the Guardian last year,
and my remit there, I do all sorts of bits and pieces,
but one of the things that I'm really focusing on there
is trying to make it so we can build things faster,
because the Enterprise Java stack works fantastically well
for building sort of robust, reliable applications,
but it's not so great if you want to turn something around
in a couple of days or a week.
So I've spent a whole bunch of time there
quietly thinking to myself,
how can I do some cowboy development in an environment
where people actually do real software engineering.
And actually the opportunity came up a few months ago,
thanks to our MPs in England.
This was the, has anyone here heard about
the UK MP expenses scandal?
Quite a few people actually, that's a surprise.
Normally in England we pay enormous attention
to US politics because, well for one thing,
you guys have actual villains.
I don't think any of our MPs have ever shot a wolf
from a helicopter or anything like that.
Or did he have a guy who had $100,000 in his freezer?
I thought that was particularly classy.
UK political scandals are a little less exciting.
The MP expenses scandal, it was found that our MPs
have been claiming on expenses for things
that they shouldn't have.
One of them claimed for some dog food.
As taxpayers, we thought that that wasn't appropriate.
So it's not quite $100,000 in the freezer.
A better example is an MP who attempted to claim
for an Oriental duck house for their duck pond.
There was another one who made a claim to help clean out the moat around his private house,
which played up to a whole bunch of fantastic Tory stereotypes.
But anyway, the MP expenses scandal originally... the story starts, but it's this woman.
This is Heather Brooke. She's a freelance journalist.
And in 2005, she filed a Freedom of Information request asking to see the paperwork
on what MPs have been claiming on their expenses.
And of course there's freedom of information requests taken very seriously.
So three years later, in February of 2008, after everyone had been stalling and all sorts of things,
it got sent to the Information Tribunal, where a lawyer for Parliament famously said,
"Transparency will damage democracy. This is why we shouldn't release our expenses."
Which I thought was awesome.
So the MPs were told they had to release their expenses.
In 2009, they quietly tried to pass a law saying that they were exempt from the Freedom of Information Act
and they announced it on the same day as the government announced proposals to extend Heathrow, to build a third runway at Heathrow
which normally would have buried the story, but thankfully we have the internet now.
My Society put out a campaign saying, "You've got six days left to kill this bill."
Blimey, it looks like the internet's won, we actually managed to kill it.
So the MPs had run out of options and they had to reveal their expenses.
They stalled again, they said, "Well, no, we've got to go through and censor them.
We've got to black out any of our addresses or things that mean that our angry constituents
will come and burn our houses down."
Then in March 2009, there was a mole.
An MP on Newsnight said, "All of the receipts of 650-odd MPs, both redacted and unredacted,
are for sale at a price of £300,000."
So evidently someone in Parliament decided to do the honourable thing and leak these expenses,
but slightly less honorable in trying to make the six figure sum out of it.
A month and a half later, the Daily Telegraph newspaper suddenly came into possession of a CD-ROM with 650 MPs' expenses on.
And as someone who works for a different newspaper, I couldn't possibly comment on how they might have got hold of those.
And that was... the next month and a half, the only thing in the UK news was revelations upon revelation about MPs' expenses
and things people have been claiming,
and dodgy mortgages, and second houses,
and some of the, just an enormous list of stuff.
If you want to go through the dirty details,
there's some excellent coverage online.
So meanwhile at the Guardian, where we,
yeah, I won't say anymore about that.
And meanwhile at the Guardian,
in April we heard that the expenses were finally coming out.
The government set a date,
and so I was in a meeting and someone said,
so they're coming out with all these documents.
Is there anything like neat and technology-like
we can do with these?
And then in June, we suddenly got word
that the expense has been bumped forward
and will be out next week.
So we now have a week period,
we have one week before all of these documents are released.
And this, like I said, I was looking for an opportunity
for cowboy development.
This was pretty much the perfect opportunity.
You've got a strong, strict deadline.
You've got a week to do something.
And so on the first day of June,
I knocked out a proof of concept,
sort of a very naughty little Django app
that showed how we could crowdsource
the analysis of these documents.
So we could show people a random page
of random documents, say, does this look dodgy?
And they could say yes or no.
And if we've got enough people looking at them,
we could work through them all.
On the Monday, I got the tentative go-ahead
to turn this into a real thing.
On Tuesday, I was given a designer
and client-side engineer to, I mean,
this is the great thing about working in Django,
is once you've got the templates,
you can throw them over the wall and such.
On Wednesday, we had an operations engineer,
so we can actually deploy this thing.
And on Thursday, we went live.
So it was a week from start to finish of this project.
This is what the website looks like.
This is the investigate your MPs expenses site.
We've got a progress bar showing you
how many of these documents we've reviewed,
saying 22,000 of you who've reviewed 192,000 of them,
only so many to go.
And this big red bar, it was critically acclaimed.
All over the internet, people were saying,
that red bar's genius,
that's the only reason people are going back.
And to be honest, the red bar was added
because Django has that convenient width ratio template tag.
So it was as easy to do it as not.
But I'll take the credit for design genius
that people really want to do that.
So if you click start reviewing,
you get a page of random MPs expenses.
And you get two questions.
You get what kind of page is this?
Is this a claim form?
Is it a receipt?
Or is it blank or other?
And you get a question, is this what page interesting?
Should our journalists go out and investigate further?
And so we've got not interesting, interesting,
interesting but known, e.g. it's an ornamental duck house,
and investigate this.
And the idea was that people hit that button
and one of our journalists springs into action
and phones up MPs and figures out what's going on.
And also people with more time,
we invited to add line items.
So if there were numbers on the page,
we'd ask people to type them in,
say what they were, potentially categorize them as well,
so that we could start adding up all of the numbers
on all of these 450,000 pages of documents
and see if they added up to anything interesting.
Each MP gets their own page.
we found we had pictures for the MPs, which is fantastic,
'cause when you're digging through someone's expenses,
it's really sort of satisfying to have their smug face
from a press job staring out of the screen at you.
And as time went on, we added up the various documents,
we added up the different line items.
This was sort of leg two of the MPs expenses scandal,
is when these documents finally came out,
half of them were blacked out.
There are some wonderful examples of pages
where everything's blacked out except the number.
And so this is a set of thumbnails from the document
showing you quite how much redaction was going on.
There's an interesting feature down the side.
You can see these filters, so you can filter this document
by pages that have already been reviewed,
pages that someone's marked as interesting,
that kind of thing.
And so here's how we built it.
The only reason this project was possible
is that a few of the MPs had decided
that this scandal wasn't gonna go away,
and the best thing they could possibly do
is release their expenses early,
stick them up on their own sort of personal websites
and say, look, I've got nothing to hide.
And there were about four or five MPs that did this,
which was brilliant because it gave us a preview
of what the data was going to look like
when it was released.
And the data, it was sort of a worst case scenario
from the point of view of actually trying
to do proper investigative reporting.
It was scanned A4 pages with handwritten scrolls
in the forms and wonky angles and bits blacked out
and the whole works.
But actually, so step one was to convert those into images.
ImageMagick turns out if you run convert pdf space
something.png, it splits pdf files
into a single image per page
and does the whole conversion and everything.
So the idea was just to run that command
on every single document and then we'd be fine.
We pulled in a list of MPs from theyworkforyou.com.
We implemented what I call frictionless registration,
the idea being we didn't want you to have to register
for an account before you could start auditing,
but at the same time, if you're an MP
and you're going through all of your expenses
and saying, "This doesn't look dodgy,
"this doesn't look dodgy."
We kind of want to be able to catch you out.
So what we did is the first time you vote,
we create you an anonymous user account
called like, "Canon 253."
And we then add a nag bar at the top of the page
asking you to set a username.
Give you a carrot and say, "Pick yourself a username
"to see all of your contributions on the page."
And the idea is that once you review a whole bunch of things
they all get added to your account,
eventually you set a username
and then we've at least got something we can show
in a high score table or whatever.
Once you've set a username, we'd like you to set a password.
I showed you those page filters earlier,
these things down here.
Those were nice and easy to build thanks to the Django ORM.
We just had a data structure that mapped
the names of the categories to the ORM query params
that you need to run to pull those out of the site.
And then a little bit of code to build up those filters.
The other problem we had to solve was matching the names.
So we had a whole bunch of MPs.
We had, so we had our idea of what an MP's name was,
like John Maxwell or something,
and then the government might call them Jonathan Maxwell
on the files that we downloaded.
So we needed a way to map those two data sets to each other.
This is a tool I built a few months previously,
which basically does exactly that.
You give it two sets of similar data.
It uses Levenstein distance to guess
which ones are the same thing,
and it highlights the ones that were a guess
so that you can go through and manually set them up.
Abby Bryant will be pleased to know
that this is written entirely in JavaScript,
so we are jumping on that bandwagon
just to train you a little bit.
On the actual day, it's Thursday,
it's six o'clock in the morning,
and we don't know how these documents
are going to be released.
We've got a hunch, but nothing more.
So there was good news and bad news.
The good news is that the website
that the government published these on
was absolutely beautiful.
And by beautiful, I mean it was really easy to scrape.
Each MP, there's a big list of MPs,
there's a link to the page about that MP,
the page about that MP links to the PDFs.
And so with just a few lines of beautiful soup
to pull out all the MPs, and a few more lines of beautiful soup
to pull out the list of PDFs.
We had a list of 10,000 URLs of PDF files
we needed to download.
Bad news was that the documents they'd released
weren't in exactly the format we expected.
This is the thumbnails of one of the documents.
And you'll notice that two of the files
there are in the landscape rather than portrait.
And also, I was expecting them all to be A4 portrait files.
Some of them weren't A4.
they were wonky sizes.
To make things even worse, the PDF file,
I suppose kind of fittingly,
considering what the MPs have been up to,
the PDF files were corrupt.
(audience laughing)
When you fire them through ImageMagick,
ImageMagick shells out to GhostScript,
and GhostScript just throws hundreds and hundreds
of error messages saying,
"Please contact the author of this PDF file
"and tell them they're violating the specification,"
which I don't know if that would have done any good.
But it meant that the CPU was pegged at sort of 99%
for five minutes for every single document
we needed to convert, which clearly wasn't going to scale.
So we've got this problem, these documents are out,
we've downloaded all these PDFs,
we have a site ready to run, which expects A4,
and we don't have any A4 images.
This is where the fact that I'm working for
a big organization as opposed to just hacking
in a bedroom comes into handy.
The technology director sent an email saying,
right, everyone drop everything today,
our only priority is getting this thing out.
This is the Drop Everything School of Project Management,
which I guess is closely related to the cowboy
development process.
And so we set up two teams.
We had a team working with Photoshop and AppleScript.
If you work for a newspaper, Photoshop and AppleScript
literally make the world go round.
And another team working with Java and IntelliJ,
'cause they're a very good PDF libraries for Java.
And we set them sort of head to head at a race
to see who could solve this problem first.
They actually both solved the problem
in about the same time.
It took them sort of an hour and 15 minutes.
And so we ended up using the Photoshop solution
to get the first 50 MPs into the system,
and sort of 50 most controversial ones.
And then we ran the Java thing overnight
on all of our workstations
to crunch down the rest of the files.
The plan was to put the images on S3.
Turns out uploading however many gigabytes of images to S3
takes bloody ages, so we asked Syncron
to run our own static hosting instead,
which worked much better.
And then we had to do QA.
The Guardian, I managed to get away
with the cowboy development process,
but the Guardian does not put things live
in front of an audience of 30 million people
without at least doing QA on them.
So we had the entire QA team,
we had four QA engineers clicking around the site
as we were uploading sort of files into it,
filing bugs which were being triaged
by someone running between their desk and my desk,
and I was fixing as fast as I possibly could.
And we managed to get it live.
The one thing we didn't have time for
was what you call it, load testing.
(audience laughing)
The lead architect for the Guardian said,
"You know what, we'll launch it, we'll load test it live."
So we launched, we went live on EC2.
I tweeted it, I said, "Just launched this."
We had a whole bunch of Guardian accounts retweeted,
and three minutes later, the site went down.
'Cause it turns out, it's a bad idea
to have more Apache children
than it is MySQL max connections.
I'm sure anyone in this room who's used those will know this
and we knew that as well, but we were load testing it live,
what could possibly go wrong.
So we fixed that one, that was an easy fix.
The next crash was entirely my fault.
So we have this progress bar on the front page,
and people have been, the progress bar
had been edging up really fast
as people went through all these files.
So then we started pushing more files in,
and I noticed that because we were adding files,
the progress bar was actually going back down again.
So on Twitter I said, hey, check this out.
If you hit refresh on our homepage,
the red bar goes down instead of up.
(audience laughing)
And then 30 seconds later,
I go, please stop refreshing our homepage.
(audience laughing)
Our database hurts.
We were now pegging the database at 135% of CPU.
And it turns out, when we ran the MySQL show process list
to see what the active queries were,
it turns out the single most expensive query in the system
was the one that was used to display
the little progress bar on the homepage.
That was me getting a bit trigger-happy with the ORM.
I was doing this, I was saying,
yeah, I can hear SQL nerds in the audience
shuddering with horror.
That innocent-looking little query there
turns into this, which is account distinct
on the left outer join against the single largest table
in our database.
(audience laughs)
And so we figured that out, and we were like,
ah, the site's falling over,
so we added sort of three lines of caching code,
which instantly dropped our database load
from 135% of CPU to 35% of CPU.
So that was good.
Of course, with hindsight, if we change that
from votes is null equals true to votes is null equals false,
it would have meant that we weren't doing
a let an outer thingamy join,
and it would have been slightly nicer.
The eventual solution, of course, was to denormalize.
We switched the query over to have a
isReviewed boolean flag on each page,
and our database load went to zero
and stayed there ever since.
So I think the lesson there is don't be an idiot, really.
We started off running on the same server.
We started running with myISAM on the same server
was the web application, which we knew was a bad idea,
and guess what, it was a really bad idea.
So at one point we had to migrate from MyISAM on one server
to InnoDB on a separate server.
Thankfully you can do this with a sed script.
If you run MySQL dump on one server,
pipe it through sed, replace ngin equals MyISAM
with ngin equals InnoDB,
and we did Latin1 to UTF-8 at the same time just to be sure,
and pipe it to the other server, that works.
This is cowboy shell scripting right there.
So, (laughs)
so that's kind of the story of the MPs Expenses app.
We had a post-mortem afterwards, and we said,
"Okay, that was cool, we got lots of good press out of it,
"we got the thing launched, let's never do that again."
And we started, and we basically had to try,
because, I mean, there are a whole bunch of downsides
to cowboy development.
One of them is, I was the only person in the organization
that knew how the thing worked,
so there was a bus factor of one,
which is never good.
We had no deployment process.
We did have backup scripts, we were good about that at least.
But we really, when you're an organization like the Guardian
you shouldn't really be deploying week-long
hacked together bits of crap on a EC2 instance
and just crossing your fingers and hoping they'll stay up.
So the last thing I wanted to talk about today
was reining in the cowboy.
What we're doing to try and make Django developments
inside the Guardian a bit less fly by your teeth.
And so the first step was to find a project
that we could do that didn't have to be done in a week
and we could actually plan for
and assign development resources to
and all of those sort of crazy things
that large organizations like to do.
So I can't talk about the eventual application for this yet
but the component we're building with Django
is a very straightforward RSS to JSON proxy service.
It's nice because it's a sort of fixed size of project.
We can use this as an excuse to get our deployment script
sorted out to figure out how we host things,
where we host things, build the correct Amazon EC2 images,
all of that sort of stuff.
And we're doing this with pair programming,
which means that the,
a pair programming with Java developers,
which means that they're learning how Django and stuff works.
We have unit tests, we use mocks,
we use continuous integration against Team City,
which means that our deployment script can actually,
the deployment scripts are written to take a build number
from the continuous integration service.
you can't deploy it unless the tests pass.
All of which, actually, setting all of that stuff up
took maybe a day or two, and we're gonna have it forever,
so that was a very worthwhile investment.
But the thing that's interesting about doing this
is I'm teaching Java developers Django,
and it means that I'm abundantly aware of points
where Django makes me feel a bit embarrassed.
Like, these guys come from a very different
sphere of development, and there are questions
that they ask, and most of the time,
we've got a good answer, but some of the time,
it's a little bit embarrassing.
So, I've got these in the wrong order actually.
The single most embarrassing thing about Django
is when the sysadmin goes,
"So, how do exceptions get logged?"
And you say, "Oh, that's easy, they get emailed to you."
And yeah, it turns out most sysadmins
don't take too kindly to that.
But to be honest, it was mostly smooth sailing.
It was slightly embarrassing that we had to configure
the database in order to get the test suite to run.
The lack of built-in logging stuff
meant that we had to do a bit of work
with the Django logging module.
Amazingly, one of the biggest complaints I had
was that with Django, when you deploy a Django application,
your tests get deployed alongside the code
because your tests are built into
the different application directories,
which I'd never considered before,
but I think in the Java world,
where you're deploying a compiled asset,
obviously you don't want to compile your tests
and ship them with your production stuff.
So in the end, our deployment script that builds a tarball
just excludes anything in the test directories
and that works fine.
Now, I have got a final thought,
which is almost entirely unrelated
to cowboy development or any of this stuff,
but I figured since this slot's now
in front of the technical design panel,
I can get a head start on everyone else.
And that's this.
When Django was released in 2005,
and in 2005, web development looked more or less like this.
You have a relational database,
your application, your templates,
and then maybe you've got a bit of caching
and some backend admin tools and XML feed stuff.
And that's the internet fluffy cloud there underneath.
I tried to draw a sheep in OmniGraffle,
but it wasn't really working.
So I've done that instead.
So this was 2005 when Django came out,
and that worked great.
Thing is, web development in 2009 looks pretty different.
You've still got those core things,
but you've also got external search indexes.
You have external web services.
I think every website I've built in the past year
has at some point made an HTTP call to something else.
You have these non-relational databases,
key value stores and so on, message queues,
offline workers, you're expected to provide an API.
These days you're expected to provide webhooks,
which are very exciting.
I mean, monitoring reporting.
I've got data structure servers on there as well.
There's a wonderful little piece of software called Redis,
which is kind of like Memcached,
only it persists to disk every 15 seconds,
which slightly terrifies me, but there you go.
And it can do set and list operations.
So you can say, atomically add this to a set,
and remove this from the set, that kind of thing.
Which means there's a whole load of things
which don't work so well against a relational database,
but that do work well against some kind of
in memory data structure running on the server somewhere.
And so really my interest in Django these days is
what can we do to make this kind of stack,
make this kind of thing more obvious
with the default Django install.
The Django community as a whole is fantastic.
We have projects that cover almost everything on this,
that's interesting, we have projects covering almost everything on this diagram
but I think really the purpose of a web framework is to
solve the common problems and the set of common problems
has expanded quite dramatically in the past few years
so thank you very much for listening
so have I got time for questions? I think I have
Yeah, I can take a few questions on any of that stuff.
- What a, sorry, could you grab the mic and get out?
What design process do you use
when you're starting a project?
Do you start with database models and ER?
- Do you mean which projects,
the Fault project or the MPs project?
The Fault project, the answer will be much more interesting.
- Well, a non-cowboy project, a more official process.
- Oh, a non-cowboy project.
Tell you what, since I'm mostly a cowboy developer,
what I'll do is I will answer that
with respect to the fort project,
because one thing I didn't have time to include in here
is we've actually done two of these forts.
We did the one in Alderney,
where we built Wildlife Near You.
Also, a few months ago, we did a fort project in Scotland,
in a sort of bijou holiday fort,
a holiday castle, which is very nice,
where we built a new application,
which again, we haven't launched,
but we're not 10 months on yet, so there's hope.
But when we did that, we tried to take into account
all the lessons we'd learned last time.
The biggest problem we had with the first thought project
is you're trying to build a site in a week,
so evidently all the developers hair off on day one
and start building things,
while the people who are doing design and IA
are still trying to figure out
what the site was actually going to do.
It meant that on day three,
you finally got the wireframes and stuff,
and they don't really fit what's being built,
you have a lot of duplicate work and a lot of stuff gets thrown away.
So for the second thought, we prioritized designing the application first, so making
sure at the very least we had consensus on what we were going to build and more or less
what it was going to do, and some kind of priority list.
It meant that for the first whole two days we didn't write any code, which for a cowboy
developer like myself was incredibly frustrating, but at the same time it meant that the remaining
five days when we were building stuff, we were actually building stuff we were going
to use.
My fiance has a wonderful presentation about the design process that we used at the Fort,
which I think she's written up somewhere, but essentially we used a thing called the KJ method,
which is a fantastic sort of group exercise for achieving consensus on what the thing is you're going to build.
And we did a lot of splitting up into wireframes.
We made sure that our database models were being designed by one team, and we had a URL master and so forth.
I might try and do a lightning talk on that
during lightning talk tomorrow, if that sounds interesting.
But anyway, yeah, it turns out achieving consensus
on what you're going to build when you're building something
with 12 people in seven days is actually a really good idea.
- This falls up with that question.
After you've achieved consensus
and when you're developing really fast,
how do you stay coordinated?
How do you stay consistent with each other?
And how do you keep up to date with that?
The primary answer is we don't really.
One of the reasons World Life NAU took a lot of time
to beat into shape after we got off the fort
is that it was lots of different development styles.
For the second fort, we basically,
the main way we did that was with pair programming.
We made sure that we were,
almost everything we were doing was paired up,
and we swapped the pairs around a lot,
and it meant that the sort of house development style
sort of spread organically, and by the last day,
everyone was writing code in more or less the same way.
I think really what that,
and we also reestablished PEP-8 standards
on the first day and that kind of thing.
What that really comes down to though is,
I think it's unrealistic to expect a team of people
who've never worked together to jump straight in
using the same coding standards.
We've found that pair programming spreads coding standards
and sort of best practices far faster
than any other method we could think of.
- In terms of like tasks though,
and keeping coordinated and getting work done
and giving up work to different people,
I mean, did you have any trouble with that?
- So coordinating tasks.
It helps if you're all in the same fort.
So we had a thing on the wall with all of our tasks,
and we had people's names on Post-it notes,
and you stuck the Post-it note with your name on
next to the task you were working on at the moment.
And that worked pretty well.
We also, I don't know if I can pull this up fast enough,
we also built a piece of software for managing this
called Bugle.
So on the second fort, the first thing we realized
when we got there is this was a much larger castle,
we were going to be working in different rooms, obviously we needed Twitter, and we didn't
have a net connection. So we did a sort of two-hour hacked Twitter clone, which we called
Bugle, because you're on a fort and you've got a bugle. And one of the features of that
[BLANK_AUDIO]