Historically the build, like an installer, is a very low status job. So historically it goes to low status people, like the most junior developer on the team, or perhaps even a QA person. The result is that many build scripts, like many installers, more or less suck. I think this is more a reflection on the low status of the task than the skills of the person; there is just very little incentive for fixing a bad build. But that’s mostly because so few people have worked in environment with an excellent build they don’t know how much their missing.

We think that in the move to agile development your build can be a major blocker, or and that there are major productivity gains at stake, so it makes sense to approach your build engineering efforts as a serious task. We think this is a big enough problem with a big enough potential gain that we’ve organized a public Build Engineer Bootcamp for April 22nd-23rd in Minneapolis. Come and learn how to get your build system on a solid footing and then stay for CITCON (free, but separate registration) and learn about how to take your continuous integration and testing efforts to the limit.

Hope to see you there!

My starting point wasn’t quite ground zero but it was close. I used Ruby on a project for a couple of months a couple of years ago but I haven’t used it since. And I’ve never used Watir. The last time I wrote tests to drive a browser it was 2002 and I was using WinRunner.

My motivation for this project came as a side-effect of my recent change from full-time to consulting/contract work. I’ve been working out of NextSpace, a coworking operation in Santa Cruz, and that has given me a different set of colleagues than I’ve had in the past. I’m now daily rubbing shoulders with a range of interesting people, but a bunch of them churn out websites for a range of clients. The technologies vary — WordPress, Drupal, Flash, Ruby/Rails, etc — but one common factor is there seems to little in the way of automated testing, even for sites that involve a lot of custom coding. I’ve wanted to know if some of the tools from enterprise IT and product development could be applied in their context.

I was interested in using Watir for a few different reasons. One seeming motivation is that I’ve done some consulting for WatirCraft, but that’s getting things out of order: I was interested in what they’re doing before I started working with them. What really attracts me to Watir is the philosophy that they want to take maximum advantage of Ruby as a language, as opposed to having a language neutral API like Selenium. I enjoyed Ruby when I worked with it and I’d like to do more of it. I also really like the idea of Ruby as a language for testers. Testers have long done more coding than they get credit for; switching that coding to Ruby should offer more opportunities for collaboration with developers and more credit and status from the coding they were doing already.

I was also interested in doing this experiment with Ruby because Bret/WatirCraft have been working on a new WatirCraft framework, with the source available in GitHub, and one of the enabling changes in Watir was to allow browser independent scripts by using Watir::Browser.new. All the references I’ve seen to this reference IE and Firefox, but since I’m on OS X (and because I’m a bit of an Mac fanboi) anything that doesn’t work with Safari is a non-starter. I went so far as to ask for Safari support on the WatirCraft UserVoice page only to be told it was already there. Oh. Okay then…

So here’s what I actually did. I installed the very awesome MAMP, an all-in-one installer for Macs to run an Apache, MySQL and PHP stack. Next was WordPress 2.7. Ruby 1.8.7 came via the also awesome MacPorts. Then began the fun that started “sudo gem install …”:

• safariwatir
• firewatir
• rspec
• cucumber

I didn’t actually use Cucumber, but there was talk of it in the air and the examples look interesting so I gave into the peer pressure. (Maybe in a future OSTATLI…)

Honestly I struggled a bit getting started from that point, trying to run the Watir tests against SafariWatir… or even the SafariWatir tests on itself! But here the power of Twitter came in handy, with Dave Hoover stepping up from behind the SafariWatir twitter account and updating the directions for running the Watir tests in GitHub in real time. Dave likewise saved me from the rathole of the old SafariWatir tests. In the midst of my struggles I did come to appreciate GitHub. It was trivially easy for me to fork SafariWatir and send a couple of minor tweaks back to Dave. After years of taking submission to CruiseControl via patch files and working with CVS and SVN, I’m jealous!

Eventually I was able to sort our some basic RSpec syntax and some basic Watir syntax and put together this script which met my goal. If you have the default WordPress 2.7 site running on port 8080 (and you have the same admin password that I left hardcoded in the script) then this script will open Safari, log into WordPress, post a message and logout. Change Watir::Browser.default = ‘safari’ to Watir::Browser.default = ‘firefox’ (and install JSSH) and the script will run against Firefox instead. This setting is in script for connivence; if I were testing in anger I would be setting it with an environment variable from Rake as I ran the tests.

My plan for the future it to take this a step farther and try running the same tests from Celerity. My vision is a system where we run tests in Celerity for fast feedback and then run across browsers to surface compatibility issues.

I would love to do more work doing cross-browser testing with Watir (and fix some of the incompatibilities in the process). If you know of anyone with a contract for that kind of work they can contact me via The CI Guys website.

Slow builds really bug me. And having lots of unit tests is no excuse for a slow build. If you think they are, there’s something you’re not doing right.

This past week or so I’ve been much more active working on CruiseControl, and so the slow build time — 15 minutes for a release build? ridiculous! — were wearing away at my nerves. Even the subproject main was now taking 4 minutes to build, and I remember being unhappy when it got to 90 seconds. Something had to be done and tonight was the night.

Oh look, that one test class it taking 2 minutes all by itself. Why?!? It’s not doing anything very interesting. Hmm, any good free profilers out there? A google suggests not, so printf debugging profiling it is…

The setup method is taking 10 seconds? Well that’s bad… Delving… getJmxInfo() and getServerName() take 5 seconds each… ah, getJmxInfo() calls getServerName()! But all that’s doing is getting the server name with InetAddress.getLocalHost().getCanonicalHostName(). How about we cache the result? I mean how often is the host name going to change? Cache and run the test. 32 seconds! One change to go from 126 second to 32 seconds? How about the subproject build? Down to 2 minutes from 4?! This is going to stay.

I wonder how many other places in the code we’re getting the server name… Oh, a bunch! Extract out ServerNameSingleton.getServerName() and use everywhere. Run build…

Hmm, unit test failed. Expected 8000, was -1? Hmm… Hmm… Ah! Apparently just calling getServerName() isn’t enough, I actually need to assign it! Details, details… :)  Run the full release build…

7 minutes.

15 minutes to 7 minutes, for making one simple stupid change. That I found from investigating a slow test. And that I would have botched if it hadn’t been for another test.

Man I love my tests.

(Ok, now I can sleep.)

Even better, at least for general inspiration, Kaner’s slides included a reference on using checklists in intensive care units, described in this New Yorker article The Checklist. This is a fantastic article and you should go read it now. Go ahead, I’ll wait.

Back? Excellent, wasn’t it?

Ok, for the cheaters who didn’t read the article here’s one example:

In December, 2006, the Keystone Initiative published its findings in a landmark article in The New England Journal of Medicine. Within the first three months of the project, the infection rate in Michigan’s I.C.U.s decreased by sixty-six per cent. The typical I.C.U.—including the ones at Sinai-Grace Hospital—cut its quarterly infection rate to zero. Michigan’s infection rates fell so low that its average I.C.U. outperformed ninety per cent of I.C.U.s nationwide. In the Keystone Initiative’s first eighteen months, the hospitals saved an estimated hundred and seventy-five million dollars in costs and more than fifteen hundred lives. The successes have been sustained for almost four years—all because of a stupid little checklist.

(More in the fine article, which really is worth reading.)

What I found so exciting about that article is it is such a strong evidence that simple simple change in practice can yield wildly improved results, and in an area that is every bit as complex and demanding as creating software.

To me this is insight of a piece with what I felt this summer when I read The Mundanity of Excellence, an article by Daniel Chambliss. Chambliss studied the difference in performance among swimmers to explain why some excelled and others did not. He found that it didn’t come down to any of the things you might expect: unusual personalities, qualitative differences (doing the same things but faster) or talent. Instead “excellence requires qualitative differentiation“, of doing things different things. But not exceptionally different things, just habitually different:

… there is no secret; there is only the doing of all those little things, each one done correctly, time and again, until excellence in every detail becomes a firmly ingrained habit, an ordinary part of one’s everyday life.

I love this message! It doesn’t take magic, it doesn’t take a miracle. It takes decision and discipline and will and learning something new and putting it into practice. This is why I love agile coaching, and teaching engineers TDD and refactoring, and product managers how to use cases and personas, and all those other simple practices that are now considered part of the agile/lean toolkit.

And why it is thrilling to come across an article like this one in the New Yorker that offer a new analogy: the B-17 phase.

Substantially more complex than previous aircraft, the new plane required the pilot to attend to the four engines, a retractable landing gear, new wing flaps, electric trim tabs that needed adjustment to maintain control at different airspeeds, and constant-speed propellers whose pitch had to be regulated with hydraulic controls, among other features. … The Boeing model was deemed, as a newspaper put it, “too much airplane for one man to fly.” … Medicine today has entered its B-17 phase. … I.C.U. life support has become too much medicine for one person to fly.

Airplanes haven’t become more complex since them. Instead the belief of what is needed to pilot them has moved beyond the heroic “Right Stuff” to a sort routine mundane excellence. ICUs are doing the same. I’m ready to join them.

• “We are betting that we can build a business around making testers successful with automated testing.”
• “Pete and I are making a bet …[on] the vision … automation is about code and success requires understanding code, and making code understandable.”
• “We are also betting that people want and will be willing to pay for a framework that is easier to use.“
• “We are also betting on Ruby.”
• “We are also betting that the growing use of Agile Development practices will continue.”
• “We think testers do have a significant role in the future of automated testing.“

Based on my experiences over the years I think that Pete and Bret have made a lot of reasonable bets.

At Agitar we had literally hundreds of companies as customers, which is pretty significant when you realize that our prices started at around $35,000 and went up to seven figures. That’s a good indication that there’s a market for helping people succeed at testing their software.

From my time in QA doing test automation with the Mercury tools I agree that successful testing required actual coding. I think this has been a barrier to success with traditional automation attempts, at least in Silicon Valley. To be a successful required coding skills but if you had coding skills there was a higher paying higher prestige job waiting for you in development. It has gotten to the point that it is actually more difficult to find a good tester than it is to find a good developer because development is such a compelling brain drain.

But Bret’s right that agile development is the future and “testers on agile teams are being hammered by shorter iterations“. Because automated testing is so key to successful agile development I think this means market forces should swell the ranks of tester-developers and developer-testers. (Of course since I used to blog at DeveloperTesting.com it isn’t surprising that I feel this way…)

Finally I think WatirCraft is right to focus on the tester side of this equation. I was exposed to dozens of teams all over the world while at Agitar and what I saw was very consistent. There was widespread interest in developers testing their code but often only because you were supposed to as part of “going agile”. I saw only a few teams where the testing done by developers was considered to reduce traditional testing to any great extent. Companies want their developers developing, not testing, so I’d agree that the bulk of future test automation will belong to testers.