lixo.org

I had an interesting chat last week with some colleagues from ThoughtWorks about a situation we currently have at a few of our clients. It’s an interesting economical challange and, even though I’ve missed most of my Economy classes, I still like the subject. Point is, companies and IT contractors tend to develop a variation of the Stockholm Syndrome over time because of the way incentives play out.

It goes like this: FooCorp has a big IT project ahead of them, which will need about a dozen developers. They’ll only need a third of that to maintain the application once development has finished. It makes sense for FooCorp to hire the remainder of the staff as contractors, so that they can show them the door without too much hassle when everything goes to production use. Contractors, therefore, take the risk of having to look for another gig sooner than expected, in case the project gets cancelled, and adjust their rates accordingly. Those high rates may lead to a project management working under budget pressure, which might drive managers towards shrinking the feature set and/or the schedules. Here’s where the fun bit starts: are there any incentives for a contractor to actually deliver any quality software at all?

As I see it, there are two positive incentives: peer pressure in justifying rates, and collecting good contacts and references. In the situations we have been witnessing, those pale next to doing whatever is possible to extend the contract, which more often than not are some incarnation of finger-pointing, back-stabbing or just plain inaction in order to make the project as late as possible. By extending the contract, they can ensure there’s always time to fix the mess created by blaming everyone else but their contracting buddies (those who’ll happily point them to new gigs) and pretending to work harder than everybody else by assuming authorship of other people’s work and ideas, while happily surfing the web all day.

Wouldn’t anyone at FooCorp notice that? They may have, but by the time they realize how serious the situation is, it’s usually too late to dismiss the contractors without seriously bleeding knowledge from the project and risking the schedules too much. In a way, there’s all this business knowledge being held hostage by the contractors, and the ransom is to be paid in monthly installments, until FooCorp’s employees pick it up and are able to run maintain the project on their own. Keith Henson wrote, in Sex, Drugs, and Cults:

Fighting hard to protect yourself and your relatives is good for your genes, but when captured and escape is not possible, giving up short of dying and making the best you can of the new situation is also good for your genes. In particular it would be good for genes that built minds able to dump previous emotional attachments under conditions of being captured and build new social bonds to the people who have captured you. The process should neither be too fast (because you may be rescued) nor too slow (because you don’t want to excessively try the patience of those who have captured you).

Of course, there are contractors with good ethics and who manage, against all odds, to deliver some truly amazing code.

I don’t remember exactly how I ended up there, but after following a discussion on a possible Refactoring browser implementation for Ruby over at Murphee’s, Lean Manufacturing grabbed my attention. Besides Just-in-Time production, there is another big aspect of it that I hadn’t paid much attention to before: Muda, or the reduction of waste.

Waste, in the Toyota Production System, can come from seven different sources: overproduction, waiting, transporting, inappropriate processing, unnecessary inventory or motion, and defects. They can all be mapped to the software development domain quite easily, and most have a direct link to agile methodology practices, even if in some cases you have to stretch your imagination a bit. Not to wonder, as most of software development processes were directly inspired by production line practices - not necessarily in the conveyor belt sense, but more as in “I have (resources, ideas) and would like to make (manufactured products, software) out of them. What is the most efficient, effective and sustainable way to do so?”

There are two things we can waste while developing software: time (and its little brother, money) and brainpower (and its big brother, talent). Interestingly, flushing one down the toilet also brings the other with it: a project stuck in endless meetings or in fixing production issues is wasting both its people’s time and brainpower, and I find it fascinating and saddening at the same time how often I see projects that are up to here in both problems: a production issue is found, meetings are held for days or weeks, but nothing gets actually done to fix the problem and avoid it happening again. During those meetings, there’s lots of talk about changing the architecture, platform, getting more programmers and tools, or my favourite, “just rewriting the whole damn thing.” And rewriting a significant part of your system, as anyone who watched Netscape die can tell, doesn’t work.

Still, why is this behaviour so common, and why aren’t we working on bringing refactoring support for purportedly extremely productive languages like Ruby like there is no tomorrow, as to avoid repeating the mistake of rebuilding things instead of beating them into shape?

It looks like we’re having a Christmas party at the Old Bank of England, next December 12. For the first time since perhaps the launch of the Apple Store, people from Python, Django, TurboGears, Ruby, Rails, Java, Perl, Catalyst, Maypole and Motorola 68k and 6052 Assembly communities are getting together.

This time, instead of lusting and buying expensive hardware covered with shiny brushed metal and highly scratchable white plastic, we’ll be having beer, the obligatory plate full of chips that last for about 10 seconds and a whole lot of great conversation. As the Python, Java and Perl inquisitions will be within a few feet apart, burning people at a stake can be considered.

Leave a comment here if you intend to turn up.

We’ve started using PMD a few days ago at our current project to analyze a whole lot of legacy code. I was browsing around the website looking for documentation on rules written in XPath (as we have a few project-specific metrics, we’ll have to write some new simple rules) and ended up clicking on to the PMD Scoreboard, only to find out that they were, apart from generating the usual metrics, linking to a DOOM map file.

The map file contains a number of those nasty explosive barrels that varies according to the number of warnings generated by PMD. Pretty much like psDooM, where Unix processes are represented as monsters, but more programmable via - guess what - a Ruby API, Ruby-DOOM. How cool is that?

Can someone with loads of free time please make DamageControl or CruiseControl do this, too? With a bit of shell-scripting glue, you could have a Continuous Integration arcade machine with a new game starting after every build. This would probably encourage developers to keep the build times short, so they can play another round

Sam Newman is surprised to know that Flickr doesn’t use many automated tests because, “being such a high profile and successful application I assumed there would be a more mature approach towards automated testing.”

Actually, that surprises me a bit too, but I think I know why they don’t really care that much about testing, and my take is that they’re not just a bunch of the regular outdated dot-com era cowboys throwing code at a wall to see what sticks, the ones who often get criticized for not testing their stuff and told to do better by the usual agilist (rhymes with pugilist?) at conferences and papers. There might be some reasons to not bother testing, and I’ll try to explain them here, even knowing this post is going to be a bit inflammatory. First, let me propose two different scenarios, with the numbers representing some imaginary unit of cost during a given period, after which a new project phase starts:

Assuming ceteris paribus, and also that the code was well-written by competent developers, which the Flickr folks surely are, a table like that sounds plausible: where A uses automated testing to find regressions and bugs, saving on the maintenance, B spends solely on maintenance, both with the same results (on that phase, at least).

In A, the company has to spend money upfront in order to get the testing suite up and running so they don’t have to spend with future maintenance. Now break out the pitchforks and torches and quote me to death, but this is a bad economic decision, unless the cost of writing the automated tests is lower than doing all the maintenance work, where the cost would be spread. The risk of this cost not being lower though, seems, at least in my experience, much higher than what most companies are willing to accept, hence the good results of this approach we hear about it from most, if not all, agilists.

Now quote this bit too: I am a fan of automated testing because I’ve seen how maintaining software sucks out every last drop of happiness I happen to have on my poor soul after a few months doing it. I’d much prefer writing fresh code, even if that code isn’t used in the final production environment, like test code, instead of going through another frustrating day hunched over a debugger. I am sure Sam and most of you reading this are also on the same boat.

B is, in my theory, what Flickr is doing: less upfront and more maintenance costs over time. As a startup (well, before they were acquired by Yahoo, anyway), this makes sense: the whole point of a startup is that you can do riskier things, and they guessed at some point that automatically testing anything but the most significant bits (smoke tests?) wasn’t as important as getting code out the door, fast, and obssessively listening and reacting to user feedback. This probably required keeping insane levels of attention to detail and commitment, which is quite rare I might add, but a great part of what I attribute to their success.

As Joel Spolsky put it, almost three years ago, over a similar situation:

You need some kind of economic model to decide where to spend your limited resources. You can’t make sensible decisions reliably by saying things like “load testing is a no-brainer” or “the server will probably survive.” Those are emotional brain droppings, not analysis. And in the long run we scientists will win.

So, I’m not sure who’s right, here: I know most of the projects I’ve participated in would have been (and some actually were) completely fucked without automated tests and heavy test-coverage-keeping work. But then, I’ve never worked for any startup quite like Flickr.