lixo.org

When setting up an automated continuous delivery pipeline for our current project, we decided to use RPMs and Yum – the native packaging and software updates platform of our staging, UAT and production environments – instead of more Ruby-esque solutions like Capistrano.

There were several reasons behind that, but by far the most important was the affinity we noticed the operations staff already had with RPMs and Yum: all of the system packages were being taken care of using it, with some deal of auditing thrown in: has this file changed since we installed the package? Why?

Making a decent RPM out of a Rails (or Sinatra) project wasn’t very hard: a bit of head-scratching and a few passes through Maximum RPM later, we had something we could work with.

The main headaches were working out which packages were necessary to build the RPM (the BuildRequires part), figuring out how to reliably install all of the gem dependencies from bundler into the packaged RPM (bundle install --deployment helped), and which changes needed to be made to the application to rely solely on environment variables set by /etc/default/[app]. This way, we wouldn’t have any configuration that could vary across environments coming from the package itself.

The next step was setting up a marathon of tests for those RPMs. They already contained unit tested code, and we built more obstacles: functional and integration tests, performance micro-benchmarks, metrics analysis and some manual inspection in UAT. With each step weeding out bad candidates, only truly excellent builds can get to production.

To be able to easily visualize how excellent our builds were, we came up with a simple and effective naming scheme, in time for the Olympics: precious metals.

A unit tested RPM would start out in the “tin” repository. Another step in the pipeline gets triggered and deploys it to a smoke test machine. If that works, it gets promoted to the “bronze” repository. Functional and integration tests cause it to be promoted to “silver”, and so on through “gold” (ready to go) and “platinum” (in production already).

To the operations staff, this makes a lot of sense: the production machines are fetch updates from “gold”, UAT environments look at “silver” and so on. It’s trivial to configure Yum to do that – chances are you alraedy did it when setting up your distribution –, and its output is very easy to read and understand when something goes wrong.

Looking back at the Capistrano days, my only regret is not having done this sooner!

Jakob Nielsen on the use of Agile methods:

Agile’s biggest threat to system quality stems from the fact that it’s a method proposed by programmers and mainly addresses the implementation side of system development. As a result, it often overlooks interaction design and usability, which are left to happen as a side effect of the coding.

In my experience, mostly as a developer, it is really easy to dismiss interaction and usability design for two reasons.

The first comes from the developers themselves, trading prettiness and consistency of user experience for cleaner and sounder domain models whenever they go in opposite directions. Signs this is happening are developers crying YAGNI when the stakeholders ask for a zoomable chart or DTSTTCPW when a sortable, paginated data table is required.

Next time you see a system where there are enormous listings of items with no search, pagination or sorting, ask the developers if they have ever watched a typical user at work; chances are they have only thought about the system as they see it: since the testing dataset is usually small, a loop spitting out a bit of HTML for each element isn’t such a big deal. They might even say there’s a story to implement all that lovely stuff later on, but they just get moved over and over to the bottom of the backlog barrel… until everyone watches a person struggle to find needles in a tabular haystack all day. This is a simple example – almost too trivial actually, but one I’ve seen happen way too many times.

Changing the perception that usability is just the icing on the cake draws attention to all that wasted time to the stakeholders, and should enable a much better dialogue: developers get to write an application users will love, stakeholders spend their money wisely on something that will actually increase return on investment (as productivity gains), users feel empowered and less likely to make mistakes. Everybody wins.

The second reason UI design and usability get overlooked, and this is the one Frank alludes to in his latest post, is that some agile teams rely a bit too heavily on the stakeholder’s descriptions of what is wanted. It instantly reminded me of one my favourite quotes from Cars:

Lightning McQueen: All right, Luigi, give me the best set of black walls you’ve got.

Luigi: No, no, no! You don’t know what you want! Luigi know what you want. Black-wall tires, they blend into the pavement, but these white-wall tires, they say look at me, here I am, love me.

Lightning McQueen: All right, you’re the expert.

I find it rare for the stakeholders to know exactly what they want, down to what the end-user experience should be like. Thinking about a reasonably-sized application at this level of detail can only be done as a series of small, incremental steps and having someone on the team who is really obsessed about making every single pixel on the screen be in the right place. And if you read the last sentence and thought “well, that’s not exactly the role of my stakeholder!” you get the point: the stakeholders should not have the final word as to what the usability and experience details should be, in the same way they simply delegate to and rely on the expertise of the development team to flesh out the details of a persistence layer.

Have look and feel expertise in your team, and trust it, in the same way you would trust the database or network connectivity expertise.

A toothpaste factory had a probem: they sometimes shipped empty boxes, without the tube inside. This was due to the way the production line was set up, and people with experience in designing production lines will tell you how difficult it is to have everything happen with timings so precise that every single unit coming out of it is perfect 100% of the time. Small variations in the environment (which can’t be controlled in a cost-effective fashion) mean you must have quality assurance checks smartly distributed across the line so that customers all the way down the supermarket don’t get pissed off and buy someone else’s product instead.

Understanding how important that was, the CEO of the toothpaste factory got the top people in the company together and they decided to start a new project, in which they would hire an external engineering company to solve their empty boxes problem, as their engineering department was already too stretched to take on any extra effort.

The project followed the usual process: budget and project sponsor allocated, RFP, third-parties selected, and six months (and $8 million) later they had a fantastic solution — on time, on budget, high quality and everyone in the project had a great time. They solved the problem by using some high-tech precision scales that would sound a bell and flash lights whenever a toothpaste box weighing less than it should. The line would stop, and someone had to walk over and yank the defective box out of it, pressing another button when done.

A while later, the CEO decides to have a look at the ROI of the project: amazing results! No empty boxes ever shipped out of the factory after the scales were put in place. Very few customer complaints, and they were gaining market share. “That’s some money well spent!” – he says, before looking closely at the other statistics in the report.

It turns out, the number of defects picked up by the scales was 0 after three weeks of production use. It should’ve been picking up at least a dozen a day, so maybe there was something wrong with the report. He filed a bug against it, and after some investigation, the engineers come back saying the report was actually correct. The scales really weren’t picking up any defects, because all boxes that got to that point in the conveyor belt were good.

Puzzled, the CEO travels down to the factory, and walks up to the part of the line where the precision scales were installed. A few feet before it, there was a $20 desk fan, blowing the empty boxes out of the belt and into a bin.

“Oh, that — one of the guys put it there ’cause he was tired of walking over every time the bell rang”, says one of the workers.

After reading A Theory Of Fun, I started seeing projects (and many other human activities) modelled as a game. In the book, Raph Koster describes a series of game development rules, and I found that they can be mapped to the software development project domain with some interesting results:

• A project, just like a game, has roles. In projects, they’re specific to each role. These rules define the possible actions for each of them. (This one isn’t described in the book, but I added it just to contextualise.)
• Role-specific rules should be unambiguous, intelligible and apply to all people in that role.
• No project can be developed without the meaningful interaction of the people in all roles.
• The outcome of a project has to be uncertain, otherwise it loses its appeal.
• Rules and representation of a project are not independent but interact with each other.
• People require clear and immediate feedback to understand the relationship between action and outcome.
• People require a clear goal so they can perform meaningful actions within the project world.
• Conflict and competition against time, budget and scope are essential for everyone’s motivation.
• The challenges of a project should match the skills of the people involved: neither too easy (boring) nor too difficult (frustrating).
• Projects can be developed without the need for even skill sets among the team. Instead the people learn through interaction, and this should be allowed and encouraged to happen.
• People perform actions within the project world and observe how these actions change the state of the project.
• People form a hypothesis about the meaning of a deliverable or action on the basis of their studies.
• People recognise and learn fundamental patterns within the project and can apply these to different situations (and, of course, other projects).

The more I look at these, the more this matches the way I see people working in agile projects. I’m sure there’s a lesson to be learned here…

Developers in most projects I have seen try to establish some sort of convention around leaving TODOs in the code. The most common seems to be “if you see something funny, try to fix it immediately, but if it’d take too long and you’ve got something else to worry about, leave a comment next to it starting with TODO, your initials and maybe a date”.

You know what? Using comments for that is not as cool in Ruby, Python or Java, which has had static imports for a while now. How about creating a TODO method that takes in the initials, date and comment text, or whatever else you might find useful?

import static my.project.DevelopmentUtils.TODO;



...



public void doStuffThatSmellsFunny() {

   TODO("CV, 21/jan", "Clean this mess up after fixing #3849");

   ...

}

There are some advantages to this: you can actually put some code inside that method to do, say, logging. Another good thing is that now the TODOs can be tracked using the same refactoring tools and features of modern IDEs as just any other code in the project.

It may seem a bit cumbersome, but I’ve been trying that for a few days now and it feels quite pleasant to use. I got my TODO method to just spit out the message to the console so when I run tests, I can quickly get an idea of what areas of the code touch stinky or incomplete ones.