I’ve got a good feeling about this release because unlike a lot of releases I have the feeling that we’re doing more than adding feature and fixing bugs (though we did that too). This release felt like we were paying off technical/hygiene debt at the same time, at least in a small way. Not major refactorings, just lots of small changes to make things better. Some examples are:

• Putting historical information on the download page. It is interesting to browse the history of the project on a single page and it also allowed us to delete a page off the wiki that had similar information.
• Dan added the ability to specify a log4j configuration file on the command-line. This is cool both because it allows people to make changes w/out cracking open the cruisecontrol.jar and because it allows people to use the log4j xml format. The xml format offers some options that aren’t available using the properties format so we’ve opened up another bag of tricks for our users.
• We updated to a new version of Jetty and at the same time we’ve exposed the Jetty configuration files. Like with log4j this opens up a lot of new opportunities for people to add behavior.
• The documentation is now served by Jetty. Till now our help files were there but not served but the webserver — you had to go to the public website instead. <slaps own forehead>
• The documentation for distributed usage is now available on the website. We’ve had support for distributed builds for three years but almost nobody knows about it.
• New css for the jsp reporting application. I can’t believe how dated those pages looked. I can’t believe it was so easy to change. I can’t believe we waited so long to do it. (Now to update the screenshot in the documentation…)

At the same time we’ve made these changes we’ve also laid the foundation for larger technical changes by upgrading our version of Java (to Java5), the JSP (2.1) and Servlet (2.5) APIs.

Good things happening, good things to come!

1. Developer regression testing, better yet TDD.
2. Active stakeholder participation.
3. Regular delivery of working software.
4. Self-organization.

I had special interest Scott’s criteria because I’d just posted my own attempt to the CITCON mailing list the week before. I wasn’t trying to put together a magic recipe but rather to come up with a list of practices without which — or their equivalents — you will fail. My list was:

• Iterations
• Planning game
• TDD
• Automated acceptance tests
• Continuous integration
• Retrospectives

I think these two lists are pretty compatible, but there are two outliers: self-organization and retrospectives. I was surprised that Scott didn’t have retrospectives on his list. I’m naturally a lumper not a splitter, so I’m tempted to say that retrospectives are a vehicle for self-organization and call it a day. But I don’t think that’s honoring what Scott had in mind. By self-organization Scott meant the team should be organizing the work, deciding who does what. But is that really needed to be Agile? I can imagine using all the practices listed but having a manager assign the work for each story. I can imagine it… but it isn’t something I’ve ever done. I don’t remember having the need, though this may be selective memory on my part. Certainly the majority of the time who should do what is obvious to everyone, and when there are multiple people who could do the work then it is divided easily enough. So I’m not sure that self-organization is a requirement but I clearly believe it is better. Better to have people sign-up themselves and have feel a greater ownership in the project.

So is this it, do we have our list? If you’re not doing these you’re in trouble?

Slow builds really bug me. And having lots of unit tests is no excuse for a slow build. If you think they are, there’s something you’re not doing right.

This past week or so I’ve been much more active working on CruiseControl, and so the slow build time — 15 minutes for a release build? ridiculous! — were wearing away at my nerves. Even the subproject main was now taking 4 minutes to build, and I remember being unhappy when it got to 90 seconds. Something had to be done and tonight was the night.

Oh look, that one test class it taking 2 minutes all by itself. Why?!? It’s not doing anything very interesting. Hmm, any good free profilers out there? A google suggests not, so printf debugging profiling it is…

The setup method is taking 10 seconds? Well that’s bad… Delving… getJmxInfo() and getServerName() take 5 seconds each… ah, getJmxInfo() calls getServerName()! But all that’s doing is getting the server name with InetAddress.getLocalHost().getCanonicalHostName(). How about we cache the result? I mean how often is the host name going to change? Cache and run the test. 32 seconds! One change to go from 126 second to 32 seconds? How about the subproject build? Down to 2 minutes from 4?! This is going to stay.

I wonder how many other places in the code we’re getting the server name… Oh, a bunch! Extract out ServerNameSingleton.getServerName() and use everywhere. Run build…

Hmm, unit test failed. Expected 8000, was -1? Hmm… Hmm… Ah! Apparently just calling getServerName() isn’t enough, I actually need to assign it! Details, details… :)  Run the full release build…

7 minutes.

15 minutes to 7 minutes, for making one simple stupid change. That I found from investigating a slow test. And that I would have botched if it hadn’t been for another test.

Man I love my tests.

(Ok, now I can sleep.)

My interest in topics like continuous integration, developer testing and mundane excellence have the common thread of “how do we maximize our sustainable (and sustained) velocity?” All too often I’ve worked with teams who find their velocity (and their sanity!) suffering because their codebase has grown beyond unwieldy into unlivable. These seems especially common on projects that are long lived, have large teams, or long lived projects with large teams — the common denominator really being the code passing through many hands.

Paul and I think CruiseControl provides an interesting study here because it is a long lived project that has had over 200 successful contributors. The patterns and practices that enabled this would help many of the projects I’ve seen.

(If you’re interested in SDBP but aren’t registered, this a gentle reminder that Early Registration ends this Friday.)

Even better, at least for general inspiration, Kaner’s slides included a reference on using checklists in intensive care units, described in this New Yorker article The Checklist. This is a fantastic article and you should go read it now. Go ahead, I’ll wait.

Back? Excellent, wasn’t it?

Ok, for the cheaters who didn’t read the article here’s one example:

In December, 2006, the Keystone Initiative published its findings in a landmark article in The New England Journal of Medicine. Within the first three months of the project, the infection rate in Michigan’s I.C.U.s decreased by sixty-six per cent. The typical I.C.U.—including the ones at Sinai-Grace Hospital—cut its quarterly infection rate to zero. Michigan’s infection rates fell so low that its average I.C.U. outperformed ninety per cent of I.C.U.s nationwide. In the Keystone Initiative’s first eighteen months, the hospitals saved an estimated hundred and seventy-five million dollars in costs and more than fifteen hundred lives. The successes have been sustained for almost four years—all because of a stupid little checklist.

(More in the fine article, which really is worth reading.)

What I found so exciting about that article is it is such a strong evidence that simple simple change in practice can yield wildly improved results, and in an area that is every bit as complex and demanding as creating software.

To me this is insight of a piece with what I felt this summer when I read The Mundanity of Excellence, an article by Daniel Chambliss. Chambliss studied the difference in performance among swimmers to explain why some excelled and others did not. He found that it didn’t come down to any of the things you might expect: unusual personalities, qualitative differences (doing the same things but faster) or talent. Instead “excellence requires qualitative differentiation“, of doing things different things. But not exceptionally different things, just habitually different:

… there is no secret; there is only the doing of all those little things, each one done correctly, time and again, until excellence in every detail becomes a firmly ingrained habit, an ordinary part of one’s everyday life.

I love this message! It doesn’t take magic, it doesn’t take a miracle. It takes decision and discipline and will and learning something new and putting it into practice. This is why I love agile coaching, and teaching engineers TDD and refactoring, and product managers how to use cases and personas, and all those other simple practices that are now considered part of the agile/lean toolkit.

And why it is thrilling to come across an article like this one in the New Yorker that offer a new analogy: the B-17 phase.

Substantially more complex than previous aircraft, the new plane required the pilot to attend to the four engines, a retractable landing gear, new wing flaps, electric trim tabs that needed adjustment to maintain control at different airspeeds, and constant-speed propellers whose pitch had to be regulated with hydraulic controls, among other features. … The Boeing model was deemed, as a newspaper put it, “too much airplane for one man to fly.” … Medicine today has entered its B-17 phase. … I.C.U. life support has become too much medicine for one person to fly.

Airplanes haven’t become more complex since them. Instead the belief of what is needed to pilot them has moved beyond the heroic “Right Stuff” to a sort routine mundane excellence. ICUs are doing the same. I’m ready to join them.

(Suggestions for better sites for sharing code snippets?)

I’ve always intended to write a longish blog/article/thing explaining and expanding on the ideas from the original talk and what I’ve learned since (including what came up at the Agile 2007 discovery session). But this isn’t that thing. Instead David Smart, one of the facilitators of the CITCON session, reminded me that I said I’d share the slides. So here are the slides as pdf, and a public commitment to provide more explanation in a future entry.

• David Abramson, Muse
• Steve Demeter, Demiforce LLC
• Ramin Firoozye, FrolicWare
• Rob Hunter, Scribular
• Mike Lee, Tapulous (blog title is NSFW)