Futurechimp - Home

ruby-debug19 on Ubuntu 10.04 (with rvm)

2010-09-09T10:55:00Z

Note to self: the newest available version of ruby-debug19 will fail brutally like this on the Ubuntu 10.04:

$ ruby -Ilib:test test/controllers/test_my_account_controller.rb 
/home/dave/.rvm/gems/ruby-1.9.1-p378/gems/ruby-debug-base19-0.11.24/lib/ruby-debug-base.rb:1:in `require': no such file to load -- ruby_debug.so (LoadError)

To avoid this unpleasant fate, uninstall the latest versions of ruby-debug19 and ruby-debug-base19 (which are both at 0.11.24):

gem uninstall ruby-debug19
gem uninstall ruby-debug-base19

For some reason, that didn’t do the trick for me and I also had to manually remove the installed gem files from rvm’s gems folder:

rm -Rf ~/.rvm/gems/ruby-1.9.1-p378/gems/ruby-debug-base19-0.11.24/

After that, the gem installed just fine, one patch point back:

gem install ruby-debug-base19 -v=0.11.23
gem install ruby-debug19

Installing CouchDb 1.0.1 on Ubuntu 10.04

2010-09-03T10:23:00Z

CouchDb 1.0.1 is hard to install from source on Ubuntu 10.04, mainly because the libmozjs-dev package doesn’t exist in the apt repos. If you try following the CouchDb install from source directions, you’ll get a message like this:

configure: error: Could not find the js library.

Is the Mozilla SpiderMonkey library installed?

CouchDb relies on Mozilla’s SpiderMonkey javascript engine, but that’s not available in Ubuntu 10.04.

This should install most dependencies:

sudo apt-get build-dep couchdb

In case build-dep didn’t get this one:

sudo apt-get install xulrunner-dev

Grab the SpiderMonkey source from the Mozilla website, build it, and install it:

wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.8.0-rc1.tar.gz
tar -xvzf js-1.8.0-rc1.tar.gz
cd js/src
make BUILD_OPT=1 -f Makefile.ref
sudo make BUILD_OPT=1 JS_DIST=/usr/local -f Makefile.ref export

That’ll build and install libjs – but after a reboot, couchdb may lose track of where you’ve installed libjs. You could mess around with environment variables every time you spark up couch, but you might want to make it more permanent by informing ldconfig of the library’s location. Assuming you’re on 64-bit, you can do this:

sudo touch /etc/ld.so.conf.d/spidermonkey.conf
sudo echo "/usr/local/lib64" >/etc/ld.so.conf.d/spidermonkey.conf
sudo ldconfig

Adjust as necessary for 32-bit systems if you’re running one of those.

After that you should have no trouble grabbing the CouchDb source, untarring it, and doing the configure-make-make-install dance:

wget http://apache.mirror.anlx.net//couchdb/1.0.1/apache-couchdb-1.0.1.tar.gz
tar -xvzf apache-couchdb-1.0.1.tar.gz
cd apache-couchdb-1.0.1
./configure && make && sudo make install

Enigmamachine - first working release

2010-07-12T15:29:00Z

I’ve just put the finishing touches on the first working release of Enigmamachine. It’s a RESTful video encoder written in Ruby. It uses the thin webserver, eventmachine event framework, sinatra web framework, and datamapper ORM.

There’s a lot more info in the README, but basically it’s a standalone HTTP service which accepts a POST request triggering a video encode on the local filesystem. In short, your web app gets a video uploaded to it, and once it’s saved on your filesystem, you’d do something like:

wget http://username:password@localhost:2002/videos --post-data 'video[file]=/path/to/your/video.mp4&encoder_id=1'

This would apply whatever ffmpeg tasks you define to the uploaded video. Advantages for you? You don’t need to worry about process daemonization, threading issues, or maintaining a processing queue, you just fire off a POST request and enigmamachine does the rest.

There aren’t any callbacks yet, and I can see that this is probably the next thing to do (as well as slugging the encoder_id). That’ll wait, though. Basically it’s ready for first use, after a long long time messing with the idiosyncracies of combining ffmpeg (which logs to stderr), thin, a gem binary, and eventmachine’s multi-threaded mode. This was an interesting process all on its own, and I’ll do a write-up about Thin daemonization soon, as I think it’s a really great way to make low-ceremony services in Ruby.

Every picture tells a story

2010-03-26T14:29:00Z

I’m not much of a photographer but I couldn’t resist a snap of this situation, which we strolled past yesterday during a walk-and-talk session discussing some development issues:

UploadColumn Gem Released

2010-03-19T23:53:00Z

As part of a series of experiments with the Rails 3 beta, I’ve started gemifying a pile of plugin code we’re using. Rails Engines in Rails 3 are completely integrated into the framework (every Rails app is now a Rails::Engine!) so there are going to be some exciting times ahead as we start building modular apps which can be distributed with versioned gem dependencies. Part of the fun will be turning plugins into gems wherever possible, so that library code can be cleanly integrated into the modular apps, and everything is quick to set up at the start of a project.

The first fruit of this is the UploadColumn gem, which is just the venerable UploadColumn Plugin wrapped up and packaged as a rubygem. It works in the same way it always has, except that you can include it in your project by doing a:

sudo gem install uploadcolumn

and then a:

config.gem 'uploadcolumn', :lib => 'upload_column'

The code is forked out on Github, dive in and start hacking on it if you are an upload_column user. The tests must be for an earlier version of RSpec (they no longer run even in the canonical JNicklas repo), we’ll have to do some cleanup work when we get a chance.

MAPI is Evolving

2010-03-16T15:35:00Z

Recent changes to the email infrastructure around here have meant that those of us on Linux still have some troubles. The Exchange 2007 MAPI connector seems to work pretty well for Mac users, but Evolution’s MAPI provider plugin in Ubuntu 9.10 doesn’t work particularly well at present. It screws up calendars and inserts some strange entities (”=20”) on line breaks in emails, making us seem even geekier than we actually are.

On the upside, Evolution doesn’t freak out very often, as it did with the Exchange 2003 connector. So what happens if we try and compile the MAPI connector from source?

#!/bin/sh

#Shut down all running evolution processes
evolution --force-shutdown

# Make somewhere for the source code to live
mkdir ~/evolution
cd ~/evolution

#Download source code from gnome-evolution project page
wget http://download.gnome.org/sources/evolution-mapi/0.28/evolution-mapi-0.28.3.tar.bz2
wget http://download.gnome.org/sources/gtkhtml/3.28/gtkhtml-3.28.3.tar.bz2

#Remove evolution-MAPI package
sudo apt-get remove evolution-mapi

#Install needed dependencies for compilation
sudo apt-get install libdb-dev libnspr4-dev libnss3-dev libical-dev libsqlite3-dev
sudo apt-get install bison intltool gnome-core-devel evolution-data-server-dev 
sudo apt-get install libcanberra-gtk-dev
sudo apt-get install libgtkhtml3.8-dev network-manager-dev libunique-dev libhal-dev
sudo apt-get install libgtkimageview-dev libpst-dev libnotify-dev
sudo apt-get install libmapi-dev samba4-dev libglib2.0-dev
sudo apt-get install libebackend1.2-dev libecal1.2-dev libedataserver1.2-dev
sudo apt-get install  libedata-cal1.2-dev  libebook1.2-dev libedata-book1.2-dev
sudo apt-get install libcamel1.2-dev evolution-dev libedataserverui1.2-dev

# Compile gtkhtml
tar fxj gtkhtml-3.28.3.tar.bz2
cd gtkhtml-3.28.3
./configure && make && sudo make install
cd ..

# Compile the mapi plugin
tar fxj evolution-mapi-0.28.3.tar.bz2
cd evolution-mapi-0.28.3
./configure && make && sudo make install
cd ..

# Now you can just start evolution normally and enjoy some improvements!

Calendar invites still screw up and disappear randomly, but at least emails don’t have weird entities strewn all through them, making me feel a little better about things. Hopefully this package will get some love in the forthcoming Ubuntu 10.04! Thanks to the Ubuntu Forums people for providing partial instructions for how to compile these packages (I’ve turned that forum post into a shell script that should hopefully work for most people).

Important Note: this process only works if you’ve already got your Exchange MAPI account set up and properly configured. Installing the evolution-mapi-0.28.3 plugin breaks the UI which is used to enter your account details right at the start of the Evolution setup process, making it impossible to set up any kind of account. This somewhat sucks but I guess if you’ve already got your account set up, it won’t make much difference to you.

The shell script, ready to go, is here. If it eats your dog or provokes an existential crisis, you can back out of the changes it makes like so:

evolution --force-shutdown
cd ~/.evolution
cd evolution-mapi-0.28.3 && sudo make uninstall && cd ..  
cd gtkhtml-3.28.3  && sudo make uninstall && cd .. 
sudo apt-get install exchange-mapi

Headless Virtualbox, dm-crypt, and VRDP

2010-03-10T17:17:00Z

So we just got a new server machine into the office, and it’s got some major stonk so that we can move towards virtualised services in a big way. The machine is called gort, and it dominates:

As a result, we’ve got a few new file servers for specific purposes, and they’re running samba on Linux on top of Virtualbox rather than SMB on Windows. Some of them have sensitive data, which we want to protect in case of theft, so I’ve encrypted the crap out of everything using dm-crypt.

I was initially stumped as to how to make this whole setup work properly. A dm-crypt disk volume needs its encryption passphrase entered before it even begins to boot. However, the Virtualbox machine image needs to run using VBoxHeadless, without any kind of graphical interface, because we want to be able to boot the whole setup remotely via a console if a machine image goes down for some reason. It seemed like there was going to be a problem – either we could run the setup headless and not have to come into the office if a machine crashed, or have dm-crypt encryption as an anti-theft strategy, but not both.

The solution turned out to be remarkably simple.

Any Virtualbox VM image started using a command like

VBoxHeadless -startvm "FooBox"

automatically gets its own display server running something called “VRDP” (which I assume is maybe “Virtualbox Remote Desktop Protocol”). The first VM listens on port 3689 of the host server, and you need to explicitly tell any further VMs to listen on a different port, perhaps like so:

VBoxHeadless --startvm "Some Machine Image" -v on -vrdpport 5000 &

This will tell Virtualbox to start the VM called “Some Machine Image” and turn on the VRDP server on port 5000 (and the relinquish control of the terminal back to stdin due to the ampersand). Assuming that the host server has a hostname of “foo.bar.blah” on your network, you can then use your regular Remote Desktop Protocol (RDP) client to connect to the host server on port 5000. You’ll be shown the current state of the guest VM right from the moment it initially boots, allowing you to type your encryption passphrase in.

So the process basically looks like this:

dave@gort:_{$ VBoxHeadless --startvm "Some Machine Image" -v on -vrdpport 5000 &
[1] 8905
dave@gort:}$ Sun VirtualBox Headless Interface 3.1.4
(C) 2008-2010 Sun Microsystems, Inc.
All rights reserved.

Listening on port 5000.

dave@gort:~$

At that point, the VM image has fired up and is running, but it isn’t booting, because it needs its passphrase before it can access its system files and start to boot.

So, we can connect to the host machine (don’t bother trying to connect to the guest machine yet). One very important thing to note is that VRDP only worked for me using RDPv5:

This gets us right on to the VM image’s grub screen, and then to the dm-crypt passphrase entry.

Entering the passphrase allows boot to continue normally. I’ve got to say, I’ve been pleasantly surprised by easy Virtualbox has made everything. The admin tools are first-rate and convenient to use, networking has been easy to set up, and the learning curve has been low. Now I can take all the time I’ve saved and use it to mess with kvm.

Rsync + Samba: Don't Archive!

2010-02-17T18:37:00Z

Just a quick note to self (and whoever else it might help) – when doing an rsync backup to a mounted SMB file share, this works:

rsync -rvz --progress /path/to/source /path/to/destination/

This doesn’t:

rsync -avz --progress /path/to/source /path/to/destination/

The -a option (“archive”) doesn’t work properly with SMB, so the files are never copied. Thanks to the Linux Blog for that one!

Hubbub: A Quick Tour

2010-02-02T13:57:00Z

We recently did some e-commerce work for an online startup called Hubbub. We’re pretty proud of it here, so it seems worth it to talk a little about the challenges involved in building it.

Most e-commerce jobs are relatively simple from a business process point of view – one website has one shop inside it, and in most cases, when you buy something it instantly gets sent for fulfillment (either physically or digitally). For example, if you buy a pair of shoes from Humanic (which we worked on last year), the shoes move into the delivery system immediately, and are usually sent out the next day. Fulfillment, i.e. the delivery of your shoes, is handled by a dedicated service provider external to the website itself.

Hubbub is completely different. Instead of having one shop and a relatively simple delivery system, the site is composed of multiple shops (a baker, an Italian deli, a fish shop, etc) which are in a delivery area – so the Hubbub site needs to keep track of which products have been ordered from which shops. New shops and new delivery areas can be added at any time. If Hubbub decides to expand into Kensington, Chelsea, and Clapham tomorrow, it can add those three neighbourhoods as new delivery areas, and add the shops from each neighbourhood into the right area. Specialist shops (like Christchurch Fish, which can serve all of London) can be shared across multiple delivery areas. The site then keeps track of which shops have had which items ordered from them.

Items can be scheduled for delivery on a specific hour anytime in the 30 days from the date of purchase. The site has its own fulfilment system built into it, automating the scheduling of store pick-ups and item deliveries for each delivery area on an hourly basis. Delivery drivers have their pickup and delivery schedules emailed to them every day in the early afternoon, allowing them to plan out their routes for the day. All in all, the site code is much more than a simple webshop, it’s a fully integrated customized e-commerce software platform which enables Hubbub to function. It’s arguable that the Hubbub business would be impossible without the code to back it up.

Most sites take payment when a customer orders an item. On Hubbub, this isn’t possible because the price of the item may not be known when it’s ordered. In the case of a steak, a chicken, or a bag of fruit, for instance, the site may give an indicative price (steak is £3.99/lb), but the actual price of the steak which will be delivered isn’t known until the butcher actually cuts it up and weighs it on the day of delivery. This is a huge issue from a business process point of view. In order to make this relatively complex payment flow work, we wrote our own payment gateway integration code so that we can do 3-D Secure authentication with the SecureTrading Xpay gateway.

The end result is a site which looks deceptively simple. It unifies multiple webshops (which correspond to multiple real-world businesses) into a unified front-end, offering each shop its own login area for product management and order tracking. The Hubbub site administrators also have their own secure login area which allows them to track the progress of the business.

The shop code is based on a heavily-modified version of Spree retrofitted with our own internal role-based access control systems and an extensive test suite based on shoulda, machinist and cucumber.

If we made one mistake during development, it was to write too much new code before we’d adequately covered the Spree codebase with tests we liked. The default Spree comes with an rspec test suite which was incomplete (when we forked the code from Spree six months ago) – we like Shoulda and Test::Unit better, so we rewrote the suite. We should have gone all the way with the code coverage before writing any new code, though – it was amazing how much our development effort sped up once we got code coverage above ~70%. For new projects, we’ve instituted a 95% code coverage minimum on our CruiseControl server (with an actual internal target of 100%).

It’s a pretty good feeling – over the past few years, we’ve gotten way more agile. Codewise at least.

Rails Duckboards

2010-02-01T15:46:00Z

These days, Rails is no longer the hot new thing in the way that it once was – as programming frameworks go, it’s settling happily into a vigorous middle age. The good thing about this is that it means a lot of those awkward teen moments are a thing of the past, and that the experimental choices of early adulthood have given way to a stable, productive lifestyle.

However, for anyone who is new to Rails, the learning curve can be very steep. For those of us who’ve been using Rails for years (I’ve been at it since -v=0.11), it’s easy to take for granted the amount of knowledge that we’ve accumulated without thinking about it. Given the choice of several million blog posts, screencasts, and mailing list postings about Rails, I suspect it wouldn’t be an easy thing these days to start fresh.

So, it seems worth it to try and provide some duckboards for new users. There’s a swamp of old blog posts, incorrect information, and simple irrelevance out there on the internetz, and it seems worth it to try and help out newcomers. Outlining what libraries we’re using here also seems like a good idea when they are different to the framework defaults (and worth it for a newbie).

So, if I had to give some advice to people learning Rails today, I’d tell them to focus on:

a good idea of where application logic should go
a firm understanding of ActiveRecord, its relationships, and when not to use it
test-first experience with Rspec, Shoulda, and/or Test::Unit (or all three)
good understanding of what RESTful routing is and how to use it
DRY coding style

Skinny Controllers – Fat Models

It’s easy to succumb to the temptation to pack all application logic into the controllers, but it’s a mistake, as detailed in Jamis Buck’s classic blog post Skinny Controller, Fat Model. Rails controllers should respond to form input in the minimum amount of code possible – the more application logic gets offloaded into the models, the better.

RESTful routing

Routing is connects URLs to code and also guides the basic controller structures of the application. Sticking with the recommended controller actions and nesting resources properly really cleans up a given application, making it more consistent and logical, and allows the easy construction of machine-accessible APIs. A good intro to RESTful routing is available on Daryn Holmes’ blog and there’s another good intro in the Rails guides.

Test-first development

We’ve decided to settle on the Shoulda gem (in conjunction with Test::Unit) for unit and functional tests. From all the different testing frameworks we’ve tried out, it is the simplest and most readable way of ending up with a comprehensive test suite. There’s a great screencast on how to use Shoulda here We’ve also settled on using the Machinist gem instead of fixtures: http://github.com/notahat/machinist. For small projects, fixtures are fine. For anything with a complex domain model, fixtures start to suck pretty quickly.

In order to generate our models, controllers, and scaffolds using the Test::Unit/Shoulda/Machinist stack, and get 100% test coverage on the scaffolds out of the box, we built our own gem which does the boring work for us: the shoulda_machinist_generator. Check it out :).

Activerecord (especially associations)

Reading and understanding pretty much the entirety of the ActiveRecord documentation is extremely important. Busting out of ActiveRecord and using raw SQL when necessary is still smart – but the point at which that happens should be pretty far up the complexity mountain, it shouldn’t be a first resort when doing rapid development. For optimization, yes; for doing extremely complex queries, yes; for doing basic has_many, has_many :through, has_one, or polymorphic queries, no. New Rails coders often seem to have trouble breaking out of a database-table-centric mentality, especially when it comes to associations – so read the basics about the ActiveRecord class methods. Every day when you wake up, repeat the phrase in Ruby, everything is an object, so I should stop doing things like “validates_presence_of :article_id”.

DRY

The temptation to use the “old standby” CTRL-C, CTRL-V when building new functionality is always hard to avoid. Using test-first development is one way to fight this evil urge, another way is the rule of threes: writing code once is good, copying and pasting it into another part of the application is alright but not desirable, and if you find yourself copying and pasting it into a third place then you probably need to do some refactoring. Since all the code should be tested, refactoring should be easy because of the big testing safety-net. Keep it DRY: don’t repeat yourself

These are the basics of Rails development – once these are in place and working, it’s time to start configuring continuous integration servers (we use CruiseControl.rb), rcov for test coverage metrics, and figuring out how to deploy using Capistrano.

It’s interesting to note just how much the whole Rails ecosystem has grown. For someone moving across from a different environment (PHP, .NET or Java), Rails is no longer the lightweight framework that we used to know and love when we started with it. The core workflow has stayed pretty much the same – code generation does what it always did, ActiveRecord still makes us happy, etc. But the amount of problems that people are solving in the wider Ruby and Rails ecosystem means that it’s possible to do amazing things quickly, if only you know where to look. Our own Branston gem is a good example of this – it solves a set of problems that advanced Rails users do have (automatic generation of Cucumber acceptance-test code) but which new users probably don’t need to know about. Wading through the blog swamp is hard work – anybody else have any Rails duckboards to help newcomers?

Demystifying the Cloud

2010-01-19T14:54:00Z

I don’t like talking about “the cloud” for the same reason I don’t like talking about “web 2.0” – although it’s suggestive of a certain set of ideas, it’s too vague to really do anything creative with. I figure anybody who can talk about “the cloud” for more than 30 seconds without mentioning more specific technologies is thinking at a much too general level.

Whether I’m right or wrong about that, there are a lot of people gasbagging about “the cloud” just now. Let’s take a few steps back and think about applying the level of detail implied by such a non-specific phrase to stuff that isn’t software. Duke Ellington wouldn’t write a jazz tune by thinking about “woodwind instruments” – he’d do it by thinking something like “what kind of song structure would really allow Johnny Hodges to let it rip? What key should I write it in? What should the drummer do?”. Although there’s no evidence to support my claim, I doubt that Leonardo Da Vinci got anything useful out of thinking about anything as general as “oil painting”, even though that was the hot new thing at the time.

Anyone who’s serious about being creative and making cool stuff needs to think seriously about the conceptual and technical tools which they have at their disposal, and overly-generalized marketing buzzwords only get in the way of this kind of thinking. It doesn’t matter whether you’re making music, painting, writing code, or writing a novel, without clear thinking about what you’re trying to do and how to achieve it, you’re just going to make a mess.

Some conceptual tools

So what are people talking about when they talk about “the cloud”? Let’s try and break it down, because there are some interesting ideas in there.

A common breakdown of the cloud is to think of it as being made up of the following categories:

Software as a Service
Platform as a Service
Utility Computing

This categorization scheme is focused on marketing concerns rather than technical and creative ones, and it doesn’t do much to help me think about cool stuff I could build. I prefer a categorization that looks more like this:

On-demand services – You only pay for the computing power you use, and the costs of both entry and exit are low (although the running costs may be significantly higher on a per-unit basis than for “normal” systems). Capital costs are borne by the service provider.
Virtualization – Running your systems inside a software container instead of running it directly on physical hardware. This could mean virtualizing the entire operating system or virtualizing an application in a sandbox or VM.
Parallel processing – Using networked computers as one big computer to solve computationally expensive problems more quickly, also known as “grid computing”.
Distributed architectures – Linking a bunch of machines together on a network and setting up applications that run across multiple machines.
Self-managing systems – Software systems that are designed to take intelligent actions based on what’s happening in their environment.

These concepts are like Lego blocks. We can chain them together to make interesting software systems, depending on what we want to achieve.

Amazon’s EC2, for example, is basically on-demand services + operating system virtualization. Once we’ve got those two blocks snapped together, we’ve got a very interesting base to work from. We could be content with on-demand virtualization and use it to just deal with traffic spikes without massive capital investment, or we could snap a few more concepts into place.

On-demand virtualized operating systems + distributed architectures when applied to the problem of storage equals a virtual Storage Area Network which can store petabytes of information. Examples of this in the real world are things like Amazon S3 or Ubuntu One. I’m currently experimenting with a system that’s in the same general category, a CouchDB-based network-storage system called adhd which I’m working on with a friend. It adds the concept of self-management and can make its own semi-intelligent decisions about replication, although it’s still very much a work in progress.

Sometimes it’s interesting to take a Lego block away (instead of adding one) and see what happens. Thinking about distributed architectures and parallel processing without OS virtualization, for example, is interesting to me when I consider that even a relatively poor neighbourhood of London like the one I live in (Brixton/Stockwell) probably has more spare processing cycles in peoples’ houses than many data centres. This is a computing issue (what could we build if we could harness that processing power?) but it’s also a social and environmental issue (it’s wasteful to have all those machines sitting idle).

One thing which probably isn’t in anybody’s “cloud” categorization, but which I’m interested in, is the notion of event-driven systems. Polling sucks, and we should be building systems that can notify other systems when something happens, instead of repeatedly asking “has this thing happened yet?”. Google’s pubsubhubbub protocol is an interesting attempt to apply an event-driven approach to RSS feed publish notifications. Any time we can use events instead of polling, we should. Eventually we’ll hit a point where polling systems are seen for the slow, primitive beasts they are, and we’ll have a real-time internet. Until then, setTimeout() and AJAX polling will continue to rule the internetz.

If these are the basic suppositions, what tools are going to grow in importance and what kinds of systems are we going to see in the next little while? What are the implications for the popular tools of today?

Technical Tools

The increased ease of operating system virtualization could make application virtualization and portability a lot less compelling. Who cares if Java can run on every possible machine architecture and operating system, if we can virtualize operating systems and specific applications really easily?

PHP will either get a decent threading model or die – too many of our Lego blocks rely on the idea that a language runtime can walk and chew gum at the same time, or at least fake it convincingly, and PHP’s approach (fork an entirely new operating system process) just won’t cut it much longer. Java, the .NET languages, and the dynamic duo (Python and Ruby) won’t be hurt by this. With Ruby specifically, the JRuby implementation is going to keep rising in popularity since it’s got Java’s tried-and-tested concurrency code going for it and doesn’t have the Global Interpreter Lock (GIL) that MRI 1.8 and 1.9 have. For the kinds of multi-threaded distributed systems I’m talking about, Erlang and Scala seem like they can only grow in popularity, seeing as how they were designed to deal with exactly this problem set. Lastly, for persistence, distributed block storage systems and the NoSQL movement are going to be increasingly important.

Proprietary vendors will need to come up with a compelling license-management story for on-demand services or risk being bypassed by a whole generation of geeks who can’t be bothered to keep track of what licenses they’ve used. So far the massive first-mover advantage of Amazon in the virtualized on-demand operating system space, built largely on a Free Software stack (Xen and Linux), has obscured this.

If I want to set up my own OS virtualization and app infrastructure based on technologies which require license payments, and I want to bring instances up and down 300 times a month, how many licenses have I used? How much hassle will it be for me to keep track of? Why should I bother? Proprietary software vendors can mitigate this problem to some extent by cutting deals with hosting companies and getting them to take care of the bean-counting at a layer below that of the app developer.

The world’s biggest existing “cloud”?

Turning from tools to stuff we can build, what sorts of things are possible if we try snapping some of our conceptual Lego blocks together in different ways? Well, let’s take a look at the largest cloud system that everyone ignores. Sometimes people snap the conceptual Lego together in some pretty interesting ways, and it’s worth paying attention when they do.

The biggest publicly-accessible redundant storage system on the planet is one which we don’t normally think of as being a “cloud” system: the global Bittorrent network. This system uses a mix of on-demand services, a distributed storage and addressing infrastructure, and software self-management to swap movies and music, and back it all up in a redundant system running on tens of millions of computers. It’s mostly ignored in the “legitimate” application development literature because the vast majority of the content inside the network infringes copyright law, but it’s an amazing system nonetheless. To copyright owners, it’s a Frankenstein’s Monster which so far seems unkillable.

Compared to BitTorrent infrastructure, Amazon’s EC2 is a fairly miniscule system. Realizing this made me think a bit about how snobby we programmers can be. Despite a lot of rhetoric about “revolutionary” and “disruptive” technologies in blogs the world over, the fact that a bunch of 15 year old kids can distribute content more efficiently than the world’s largest media corporations mostly escapes our notice. The technology that accounts for about half the world’s TCP/IP traffic gets little mention from app development bloggers, because we’re all focused on Infrastructure as a Service discussions and flapping around with our iPhones. To his credit George Reese does mention BitTorrent (once) in his book Cloud Application Architectures. It’d be interesting to see it get a mention in the software architecture literature. Ironically, the book Essential Software Architecture is available as a torrent but doesn’t mention BitTorrent.

Some experiments

Despite all the hype about it, all of this cloud-related stuff is actually one of the most interesting things to happen in the last three years. Getting in on the game, me and my friend Manos have been messing around with distributed storage architectures, on-demand services, and self-managing systems in our adhd project. Even though it doesn’t really do enough yet to be useful, the sheer fun and challenge of building such a weird system (using CouchDB on top of an Eventmachine base) has been a blast.

I’ve also been building a little distributed video encoder called the Enigmamachine (again using Eventmachine as a base, but this time with Sinatra in a supporting role). There’s a whole area of programming just waiting to be explored. While none of this stuff is new (all of the stuff on our “Cloud Lego” list has been part of the programmer’s toolbox for decades), it does seem that the pervasiveness of HTTP and the mass availability of high-bandwith networks is opening up a lot of new prospects.

I’ll be giving a talk about all of this stuff at a British Interactive Media Association event at the swanky Hospital Club (the guys over at Ultraspeed invited me, thanks for that!).

Ruby Procs and Eventmachine callbacks

2010-01-17T17:47:00Z

For the last little while, I’ve been messing around to see whether I can whip off a video-encoding server which is easily installable. It’s put together using Sinatra, Thin, and Eventmachine (three of my favourite Ruby libraries lately).

At the point where it all started to work, I ran into a nasty order-of-operations problem. To fix it, I had to do a little research into how Ruby sets up blocks and procs.

The basic system is fairly simple. There’s a web interface written in Sinatra, which runs inside Thin. Off to the side of this running webapp is an Eventmachine process sitting around waiting to encode videos. The webapp interface allows users to easily define encoder profiles, which are sets of tasks for ffmpeg. As an example, we might define an Encoder with three EncodingTask objects attached to it:

output to 320×240 flash video
output to 720×576 ogg theora
make a 320×240 thumbnail jpg.

When this Encoder is applied to a new video, the tasks are run in order on the video. Once there are no more tasks, the video state is changed to “complete”.

The code that accomplishes this is pretty simple, except for the fact that it’s multithreaded:

  def ffmpeg(task, video)
    command_string = "ffmpeg -i #{video.file} #{task.command} #{video.file + task.output_file_suffix}" 
    encoding_operation = proc {
      video.state = "encoding" 
      video.save
      Log.info("Executing: #{task.name}")
      `nice -n 19 #{command_string}`
    }
    completion_callback = proc {|result|
      if task == encoding_tasks.last
        video.state = "complete" 
        video.save
      else
        current_task_index = encoding_tasks.index(task)
        next_task_index = current_task_index + 1
        next_task = encoding_tasks[next_task_index]
        ffmpeg(next_task, video)
      end
    }
    EventMachine.defer(encoding_operation, completion_callback)
  end

The first time through, the code is called with the first EncodingTask and whatever video is being encoded. The method then calls itself again for every EncodingTask until there are no more EncodingTasks left in the queue, at which point the video is marked as complete and everything chills out. The main motivations for this code are:

We need to execute a long-running, blocking process (the ffmpeg encoding)
We want the Sinatra/Thin application to keep running while we’re encoding, because we want to be able to queue up other videos to encode

The EventMachine.defer method does what we need here. EventMachine is mainly designed to be blindingly fast for network operations by using a single-threaded evented reactor loop, but the single-threaded approach only works in cases where non-blocking processing is being done. For blocking I/O, like the call to ffmpeg here, EventMachine.defer comes to the rescue for us: by default, EventMachine sets up a pool of 20 ruby threads which are available for use by EventMachine.defer in cases where it’s really necessary to use multiple ruby threads.

Don’t confuse EventMachine.defer with EventMachine Deferrables, which are related to the single-threaded part of EventMachine. EM.defer is a method which takes two proc objects as its arguments – the first should contain your long-running operation, the second is a callback which should be executed once the long-running operation is completed. The main problem with my code above is that it didn’t work.

I naively expected that I could set the local variable current_task_index inside my completion_callback proc and use it normally. This was a really dumb thing to do, as I realized once I did a little refresher research into Proc objects and how they work in Ruby.

Procs get their variables from their surrounding scope when they are defined, not when they are called. This led to all kinds of exciting order-of-operations bugs in my code, which was all speedily fixed once I moved the current_task_index = encoding_tasks.index(task) line so that it was the first thing defined in the method.

Setting up autotest, rails, redgreen, and cucumber

2010-01-05T12:33:00Z

Time for another brain-externalizing post to remind myself how things work when setting up new systems. The basic process for setting up a Rails environment with exciting testing goodness is:

sudo gem install ZenTest autotest-rails redgreen

Then in your home directory, make a .autotest file containing:

require 'redgreen/autotest'

The Rails Backtrace cleaner

2009-11-23T16:33:00Z

I’d missed it, but Rails now incorporates the functionality previously provided by Thoughtbot’s quietbacktrace gem. It’s pretty easy to use, too. By default, it whacks all of the stacktrace garbage that used to happen in a Rails 2.2 app. It’s also possible to easily add new filters in your test_helper.rb file, e.g.

  # Add more helper methods to be used by all tests here...
  #
  Rails.backtrace_cleaner.add_silencer { |line| line =~ /lib\/shoulda/ }

Artificially inducing a test failure then gives a beautiful little stack trace, like so:

  1) Failure:
test: the JobsController should get new. (JobsControllerTest) [/test/functional/jobs_controller_test.rb:19]:
Expected response to be a <:successsadfsdaf>, but was <200>

QCT - Back in Action

2009-11-20T10:38:00Z

Ubuntu releases with broken versions of the QCT Commit Tool appear to be something of a tradition, and Ubuntu 9.10 (Karmic Koala) is no exception. Qct is an easy to use, very lightweight commit tool which I always find myself reaching for when doing commits these days, and I go a bit mental when I need to use subversion and it’s not around. Anway, to the problem:

chimp@boomer:~/workspace/projectname $ qct
Auto-detected Subversion repository

Qct fires up, I type in a commit message, and select which files I want to commit. Everything looks good until we hit the “commit” button:

Error code 1 not expected
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/qctlib/gui_logic.py", line 801, in commitSelected
    self.vcs.commitFiles(checkedItemList, msg)
  File "/usr/lib/pymodules/python2.6/qctlib/vcs/svn.py", line 231, in commitFiles
    runProgram([self.svn_exe, 'commit', '-F', filename] + commitFileNames)
  File "/usr/lib/pymodules/python2.6/qctlib/utils.py", line 143, in runProgram
    raise ProgramError(progStr, out)
qctlib.utils.ProgramError: svn commit -F /tmp/tmpChpKyU  app/controllers/application_controller.rb: svn: '/home/chimp/workspace/projectname/ app/controllers' is not a working copy
svn: '/home/chimp/workspace/projectname/ app/controllers' does not exist

Note the space between projectname and app. An older version of qct is not loving the new subversion 1.6 in Ubuntu, which brings some new options with it. Ok, time for the source tarball off of http://qct.sourceforge.net. To compile it, you’ll need to sudo apt-get install pyqt4-dev-tools, then cd into the source directory and run sudo make site-install to install it system-wide. Did you do that? Bad news for not reading ahead, because attempting to use that qct binary gets you:

chimp@boomer:~/workspace/projectname$ qct
Traceback (most recent call last):
  File "/usr/local/bin/qct", line 13, in <module>
    from qctlib.gui_logic import CommitTool
  File "/usr/local/lib/python2.6/dist-packages/qctlib/gui_logic.py", line 9, in <module>
    from qctlib.ui_dialog import Ui_commitToolDialog
ImportError: cannot import name Ui_commitToolDialog

So, that one doesn’t even fire up.

Lastly, I grabbed the latest source for the project:

sudo apt-get install mercurial  pyqt4-dev-tools
hg clone http://bitbucket.org/sborho/qct/
cd qct
sudo make site-install

Don’t forget to apt-get install pyqt4-dev-tools first, or that’ll blow.

The latest source appears to work fine, very many thanks to the author Steve Borho for this great little tool.