Test-Driven Development Justified

This post was imported from blogger, to see the original, likely better-formatted post see kalebpederson.blogspot.com.

>I stood in shock, in utter awe and amazement.  I was talking with one of the lead programmers of an open source project, with whom I’d been in discussions countless times over the last few years.  He was a programmer who commanded my respect, and the respect of everybody else in our IRC community.  He wrote software that was virtually bug free and did exactly what it was supposed to.   He used the idioms of the language, appropriate patterns familiar to capable programmers, and consistently demonstrated a sound understanding of theory vetted with practicality.  And yet, despite all of this, he didn’t know what TDD was.  To be fair to him, I bet he did, but he didn’t recognize the acronym and asked me what it meant.


The TDD Tax



Misko Hevery recently mentioned a tax associated with testing.  I personally read that as the tax associated with TDD, which may or may not be exactly what was intended.  What exactly is that tax and how can we separate the necessary part of the tax from the unnecessary?


Let me frame the details of the tax in light of a commercial project that has essentially taken 1.5 years, full time.

Project
Total
Lines
Chars
Chars
/Line
All Source 75478
1939605 27.02
Unit Tests

Backend 2,116 52,755 24.93
Model 5,972 168,279 28.18
Main 1,004 23,302 23.21
Utils 4,178 102,358 24.50
Widgets 8,747 251,869 28.79
Core 702 16,256 23.16
Support 157 3,272 20.84
Total 22,876 618,091 27.02
Project Total
75478
1939605
25.70

The statistics above describe a desktop application written with Qt.  Unit tests were written with Qt’s QTestlib which allows almost all interactions with widgets to be tested programatically.  Qt supports a model-view architecture which also helped test the GUI components. The application consists of about 75 thousand lines of source code and 23 thousand lines of tests.  The project actually should have more tests, but we weren’t diligent enough at the time (and we regret it now).

Typing Tax


The first part of the TDD tax is the required extra typing.  I’d like to believe that everybody knows that being the fastest typist in the world does not make them an uber-efficient, super effective programmer.  Rather, typing speed has very little impact on the quality or effectiveness of a programmer.


How much is the typing tax? I type about 100 words per minute (WPM) for standard text, so let’s assume that I can only type 60 when programming, because of all the extra special characters.  Although WPM has different definitions, I’m going to use five characters as a word.  Thus, 618,091 chars / 5 chars/word gives me 123618.2 words.  And 123,618.2 words / 60 WPM = 2060.3 minutes.  And 2060.3 minutes / 60 minute/hour gives 34.34 hours.


Yes, only 34.34 hours.  I could type the entirety of my unit tests in only 34 hours… in less than one forty hour work week.  In table form:



Words @ 5 chars/word
123618.2
Minutes @ 60 WPM
2060.3
Total Hours:
34.34



Unfortunately, I didn’t keep track of time writing the tests, so I don’t know what percentage of the time I spent writing tests and what percentage of the time I was developing production code.  But, for sake of this blog post, I’m going to assume 10% of my total development time was dedicated to writing tests.


That means that I spent about (1.5 * 52) * 40 / 10 = 312 hours writing tests.  Of that time spent writing tests, 34 hours, or 9.1% was spent typing.  The other 277 hours or 90.9% was testing related.


Breaking it Down Further

Where else was that 277 hours spent? Some of it was inevitably spent trying to figure out how I could test a certain part of the application.  Should that be part of the TDD tax?  Perhaps, but if I’m striving for quality, I am still going to test it somehow, aren’t I?

In some cases it will be easy to test manually and difficult to test programatically, such as with unit tests.  In these cases, it could be said that I’m paying a tax.

I’m not sure how else I might be spending that time.  I imagine some of that is spent deriving and discovering yet unknown requirements.  Time is likely also spent working on some design related activities — trying to discover how different pieces of the application can best fit together.  Another small portion of that time will be spent running the unit tests time and time again.

ROI

What I’m really doing, throughout this process, however, is making a deposit against the need to run the tests multiple times.  The more times I run the test, the more payment I get back; it compounds, like interest.

A bunch of different numbers wouldn’t mean much here.  For a one off project, or a project whose requirements are truly known up front, perhaps the tax associated with TDD isn’t worth it.  On most real-world projects, however, requirements will change… a lot.  And each time a requirement changes or part of the application is re-written, the automated tests pay interest.  I can change, replace, or re-write large chunks of functionality and know with certainty that I haven’t changed any important behavior.

If you haven’t tried paying the TDD tax, you should; you might find you’ll later need to collect some of its interest.  When you’ve diligently attempted TDD, spread the word for many still don’t know about it. Even if it doesn’t work for your situation, it might work for theirs.

2 Responses to “Test-Driven Development Justified”

  1. Kaleb Brasee December 5, 2009 at 11:38 am #

    >Thanks for taking the time to do these calculations! I've definitely seen the time-saving benefits of TDD, but had never done any analysis to try to get at a real ROI. I should try that out on a side project.

    Still, even without a definite ROI, the choice is pretty obvious. I can either:
    1. take 2 hours to TDD some code, finding and fixing about 5 defects in the process (which I just did this week), or
    2. code it in 1 hour and hand it off untested to QA, eat up 5 hours of their time finding my defects, and another couple hours of my time (or more of someone else's time) fixing them later.

  2. Kaleb Pederson December 7, 2009 at 10:04 am #

    >Kaleb, thanks for reading. I actually wish I had better records. The absolute most helpful thing I've seen with TDD is not in the development, or even the initial bug fixing. It's the focus on quality and the eventual payback when design and requirements change.

    Some of our requirements have changed dramatically in areas we never would have expected. Had we better followed TDD in those areas, we could have saved ourselves weeks of rework. Thanks to our tests, we've been able to make sweeping changes and to know, without a doubt, that we hadn't broken a single critical feature.

Leave a Reply