Ever had nasty problems coming from multiple development environments set up differently on your team, with developers accidentally creating files with bad encodings? Are your users complaining about all that mojibake showing up on your internationalised UI?

Well, I had… plenty of times. So I created a tiny little tool to help fight that. Encoding Detector is an aptly named tool (if I do say so myself) that recursively detects the encoding of files in a project directory errors out if anything seems fishy.

I’ve only ever used it with Ant, but it should be a breeze to set up and install on any project. All you need to do is call the main script against a directory, like:

$ python encoding-detector.py src

You’ll need a somewhat recent Python installed — anything that came out in the last 5 years should be OK — and that’s about it. Multiple directories can be passed as arguments, and fixing the errors is sometimes as easy as adding a few UTF-8 characters to boost the detector’s confidence up a bit.

This hack would definitely not be possible without Mark Pilgrim‘s amazing chardet library. The code, as always, is on GitHub, and you can also grab a neat little tarball here.

It takes about 5 minutes to set up on your project, and you can thank me later

As many reading this might know, I’ve moved from London back to São Paulo about a month ago. The timing worked out wonderfully: right before a bunch of really interesting conferences happening over here. Among them, I’m speaking at:

Encontro Ágil, 10-11 October: a quick talk about the spirit of continuous integration and how important it is to stick to it even though it’s easy to get sidetracked choosing between the tools available. I’ll explain some of the pitfalls, patterns and anti-patterns, and hope to please the Demo Gods by showing what a good continuous build, test, integration and deployment cycle looks like in practice.

Rails Summit 13-14 October: another quick talk about the use of Ruby at ThoughtWorks and some of the lessons learned in the 40+ projects we’ve done so far. I’ll try not to make it a total rip-off of Martin’s Ruby at ThoughtWorks work for QCon earlier this year, but I’m making no promises.

More to come, I’m sure. It’s good to be home!

A side note: would you, kind reader, get annoyed if I start mixing Brazilian Portuguese and English in future posts? Should I set up a separate blog, category or whatever so the non-lusophones can skip the stuff written in what would make Camões spin in his grave? Any suggestions as to how to do that effectively in WordPress?

In most projects I have been on as a consultant, getting the right environments together was an incredibly consuming task on several people with too much to worry about already: project management, senior techies and the ones responsible for keeping the legacy environments running.

Recently, the idea of getting an example minimal application deployed as soon as possible during the project lifecycle has been banded about a bit as one of the things that could’ve made my last project run smoother (as always, hindsight is always 20-20). Then, one of my colleagues (either Graham Brooks or Sam Newman, but I could be mistaken) brought up a simple and perfect analogy for it, found in The Pragmatic Programmer: a tracer bullet.

From Wikipedia:

Tracer ammunition (tracers) are special bullets that are modified to accept a small pyrotechnic charge in their base. Ignited upon firing, the composition burns very brightly, making the projectile visible to the naked eye. This enables the shooter to follow the bullet trajectory relative to the target in order to make corrections to his aim.

Tracer bullets are used during the daytime as well, but their importance is heightened severely at night: when shooting in the dark, being able to see the trajectory is crucial, since other clues — such as surrounding objects being hit — are harder to recognize and do not provide quick enough feedback for the shooter to be able to adjust his aim in a timely fashion.

Since most software projects are, in a way or another, shooting in the dark, it makes sense to create a special type of application deployment that increases the visibility of the deployments coming after it.

This application does not need to have any features, besides painting an accurate picture of what the real application (coming up in the next few deployments, hopefully) will behave like. A few interesting things to analyze:

• System environment: is the PATH set properly? Are the versions of compilers, interpreters and runtimes correct? Is there enough memory and storage space?
• Library dependencies: are all the dynamically linked libraries available and in the correct versions? What’s the overhead of a particular choice of framework or data mapping strategy in memory consumption and performance?
• Interactions with other legacy systems: are they available in the correct version? Is the application authorized to talk to them? Can they be contacted reliably and at the performance levels desired, or is there a network issue that needs to be solved?
• Performance: how many users of the application (once it’s deployed) can we tolerate? What are the expected response times? Under load, what are the bottlenecks?

These questions do not necessarily have to be answered by a monolithic tracer bullet application. In my view, they’re more like deployment spikes: once they answer a question and the team learns from them, they can be thrown away or reused, but there’s no obligation to build on a previous one if doing so adds noise to the measurements.

This works for me: testing is about sustainability. If your tests don’t let you be sustainable (for whatever value of “sustainable” is
important to you) then you’re probably doing something wrong.

The interesting thing here is that you can define sustainability as “if we don’t have this live tomorrow morning, the company will fold” and then proceed with the most appropriate testing strategy – something that gets out of the way and gives you the fastest possible feedback, like a smoke test not much deeper than a ping to the server or a wget piping to diff.

If you define “sustainable” to be “this codebase needs to constantly adapt to new features and output formats over the next 5 years”, the testing strategy changes accordingly, and it is up to a team of competent professionals to choose at what level to operate. In this case, most would choose to use tools like Behaviour Driven Development/Design and Domain Driven Design to explore as much of the problem space as possible during development and seek shared understanding through automated and repeatable processes, that can be maintained along with the codebase and ensure the features are always easy to understand and modify, should the functionality require changes (and it will).

Git commits: 1688
Xbox 360s bought: 1
Projects created or started: 14
Guitar Hero levels completed with 5 stars: 98 (on medium)
Burglaries: 2
Full-time projects: 3
kiva.org loans made: 7
Languages learned: 2 (Erlang, LOLcode)
Presentations given: 3
Family Guy episodes watched: 116
Presentations attended: 12
Interviews given: 2 (akitaonrails.com and MTV)
Tweets: 979
Flats: 1 bought, 1 lived in
Arduinos: 4 bought, 2 destroyed
Non-fatal incidents with a soldering iron: 3
Flight time: 28.5 hours
Countries visited: 4 (Brazil, France, Ireland, UK)
Google Reader items read: 127524
Pints of London Pride per week, average: 4.33
Books: 18 bought, 7 read
Photos taken: 29355
Semi-naked photo shoots published: 1
Copies of said publication sold: 500k+
Blog posts written: 16
Burritos eaten: 47
Attempts at purchasing an iPhone: 9 (0 of which successful)
Emails written: 1045 (2.8 per day in average)
Times I heard George say “shiiiiiiit” per day, average: 2.19

Jakob Nielsen on the use of Agile methods:

Agile’s biggest threat to system quality stems from the fact that it’s a method proposed by programmers and mainly addresses the implementation side of system development. As a result, it often overlooks interaction design and usability, which are left to happen as a side effect of the coding.

In my experience, mostly as a developer, it is really easy to dismiss interaction and usability design for two reasons.

The first comes from the developers themselves, trading prettiness and consistency of user experience for cleaner and sounder domain models whenever they go in opposite directions. Signs this is happening are developers crying YAGNI when the stakeholders ask for a zoomable chart or DTSTTCPW when a sortable, paginated data table is required.

Next time you see a system where there are enormous listings of items with no search, pagination or sorting, ask the developers if they have ever watched a typical user at work; chances are they have only thought about the system as they see it: since the testing dataset is usually small, a loop spitting out a bit of HTML for each element isn’t such a big deal. They might even say there’s a story to implement all that lovely stuff later on, but they just get moved over and over to the bottom of the backlog barrel… until everyone watches a person struggle to find needles in a tabular haystack all day. This is a simple example – almost too trivial actually, but one I’ve seen happen way too many times.

Changing the perception that usability is just the icing on the cake draws attention to all that wasted time to the stakeholders, and should enable a much better dialogue: developers get to write an application users will love, stakeholders spend their money wisely on something that will actually increase return on investment (as productivity gains), users feel empowered and less likely to make mistakes. Everybody wins.

The second reason UI design and usability get overlooked, and this is the one Frank alludes to in his latest post, is that some agile teams rely a bit too heavily on the stakeholder’s descriptions of what is wanted. It instantly reminded me of one my favourite quotes from Cars:

Lightning McQueen: All right, Luigi, give me the best set of black walls you’ve got.

Luigi: No, no, no! You don’t know what you want! Luigi know what you want. Black-wall tires, they blend into the pavement, but these white-wall tires, they say look at me, here I am, love me.

Lightning McQueen: All right, you’re the expert.

I find it rare for the stakeholders to know exactly what they want, down to what the end-user experience should be like. Thinking about a reasonably-sized application at this level of detail can only be done as a series of small, incremental steps and having someone on the team who is really obsessed about making every single pixel on the screen be in the right place. And if you read the last sentence and thought “well, that’s not exactly the role of my stakeholder!” you get the point: the stakeholders should not have the final word as to what the usability and experience details should be, in the same way they simply delegate to and rely on the expertise of the development team to flesh out the details of a persistence layer.

Have look and feel expertise in your team, and trust it, in the same way you would trust the database or network connectivity expertise.

Lots of Words is a new experiment to mash up a Wikipedia based lexicon with images from Flickr and whatever else I can get my hands on, in the context of building a representative and informative source of translations for any particular word, in any particular language. I’m trying to keep things as machine-readable as possible for now, so others can build on it, too.

My friend Patrick Hall and I have been musing about it for some time, and only now a technology stack allowed me to do this as a relatively small hack rather than putting together months of optimization work.

It turns out, indexing something as big as Wikipedia (check out those dump file sizes!) isn’t really an “idea in the head and 500 lines of code”, unless you use the right tools for the job. In this case, a shiny new CouchDB instance at Amazon EC2, a bit of Ruby and Merb to add a some logic and presentation magic, and JQuery as a finishing touch did the trick. This gets pretty much every Web N.0 buzzword covered, although I haven’t yet made any millions in an iPhone app.

This is a spare-time project, so it made sense for me to try out as many different bits of new technology as possible and make it into a breakable toy. This is its third implementation, and the first I’m really happy with in terms of performance and malleability. CouchDB, even with 21+ million documents loaded in about 120 GB of storage, still responds in under 200ms times on all queries I’ve tried so far. It truly is, even in its pre-1.0 days, a fantastic piece of software.

Now I find myself wanting to put a nice front-end to this, and while the current Flickr mash-up is already very interesting—and, it turns out, solves the problem of cross-language information retrieval for a small subset of Flickr, I’m sure others will have much more useful ideas about what to do with this data. My colleague Robert Rees has helped put together a hackfest here in the ThoughtWorks UK office, together with the nice folks from the London JavaScript Meetup group. Come join us 12 November!

If you just want to get to the code (be forewarned it is ugly!), it’s on GitHub.

It’s been nearly 60 years since the Universal Declaration of Human Rights has been ratified, and Seth Brau has created a amazing typographical interpretation of its text:

While you’re watching it, please take a moment to reflect about the ways in which your human rights have been challenged over the powers-that-be over your lifetime, and think of ways in which you can stop these challenges and abuses from happening.

On the geeky side, it’s also interesting to note that the UDHR is the most translated text in the world. I’d expect no less, and it’s interesting to note how technically challenging that is — apart from getting translators the world over to get the wording just right so it doesn’t become ripe for abuse and misinterpretation, it is also one of the benchmarks for proper character set handling in any particular computer system — and if you think whatever you’re building is up to scratch, you must give the translations a go. Chances are you’ll end up with mojibake all over the place at some point.

Applying the agent nouns are code smells rationale, nicely worded by my colleague Peter Gillard-Moss, controllers are code smells. The more I look at REST and the more I look at how MVC gets typically mapped to it, the more I think the C in MVC is doing the wrong thing, even when it’s skinny as possible.

I’ve only come across very few and extremely limited cases where I need to do more in a Rails or Merb controller than simply delegate to the model and set some options about how the resource is going to be represented. Most of this code can be inferred by simply looking at the available routes and methods on the model, essentially making them declarative statements rather than fully capable objects that encapsulate some behaviour of my system, or coordinate inputs and outputs.

In fact, in a RESTful application where simple and CRUD-like behaviour is encouraged in (or even expected from) all resources of the domain, controllers are just plain unnecessary. If you’ve tried something like CrudTemplate, resources_controller or resource_controller, that’s the sort of thing I’m talking about… only without the the controller as a class.

Sinatra gets this right by doing away with the controllers and allowing you to tie blocks of code to URL matchers. Still, opportunity makes the thief, and allowing a block of code to decide what’s going to be rendered or what models get called is still going to put developers in a position where they have to fight to keep their control code skinny, while moving as much functionality as possible into a rich domain model. It’s fighting an incredibly difficult battle, since the refactoring weapons are just not powerful enough at the moment, and might never be. I’d prefer declarative statements: what, not how.

Because declarative, ladies and gentlemen, is good.

I fundamentally disagree with the idea that a software development team should be constrained to working 9 to 5. Most software developers don’t even have that luxury, working overtime to compensate for all the mistakes people like Fred Brooks told us about some decades ago. In this post, I’d like to pretend more of us are treated and behave as being further to the left of the scale that starts with “Factory Worker” and ends in “Knowledge Worker”, while fully understanding it’s a whole different world “out there”, for whatever definition of “out there” is that invalidates this particular rant.

It seems funny to me that we enthusiastically build highly informative and interactive environments for teams to play in (and I use the word play in the context of a project as a game), and then treat the human beings whose minds are supposed to be completely focused on delivering business value to a customer as machines that clock in, are amazing for 8 hours… and then punch out, go home and resume their personal lives.

And this is where I want to have my cake and eat it, too. The fundamental disagreement I have with the concept of working 9 to 5 (or any other 8-hour period of the day) is that creativity, enthusiasm and logical reasoning can’t be switched on and off by the magic powers of a commute—and mine these days include walking past the Camden Lock and Market, so it’s pretty close to that. And I mean it in a good way: it’s great that developers don’t just switch off when they go back home. It’s why we have so many great open source projects coming out of what seems like pure cognitive surplus.

In fact, the very existence of a cognitive surplus tells me that I actually go through all these good ideas throughout the day. It just so happens they are sprinkled all over it, as my brain happily responds to outside stimuli, which could be an information radiator telling me something’s just happened, or the taste of my favourite local ice cream from the shop down the road. More and more, I want my work to be a part of my life, not a slice of it. As Erin Brockovich, who once allegedly yelled,

Not personal!? That is my work, my sweat, and my time away from my kids! If that’s not personal, I don’t know what is!

If you ask your parents, or maybe grandparents, perhaps you’ll get the same answers as I did. They told me my job is my job, and I have to do it so I can put food on the table. It’s left to my imagination that very few people in my group a generation or two ago had the pleasure of working because they truly like what they do and believe what they do is both positive and meaningful to society.

I think I’ve moved up the pyramid a little bit, that I shouldn’t feel or be embarassed by thinking about “work stuff” for extended periods of time when I’m about to go to bed on a Saturday morning. In fact, that’d get me labelled as someone who’s “focused” and, nowadays, even praised as a workaholic. But if I’m caught talking and thinking about something else entirely for similarly extended periods of time while in the middle of my 8-hour journey, I’m slacking off.

Why?