Archive for the ‘Agile’ Category

Subprime Technical Debt

The concept of “Technical Debt” was first introduced by Ward Cunningham in an OOPSLA 1992 experience report. Martin Fowler has a pretty concise definition of technical debt. Steve McConnell expanded the concept and even created a taxonomy of different kinds of technical debt.

Just what is technical debt? I recommend reading the above-linked articles, but here’s a brief definition if you don’t want to click away. Technical debt is speeding things up now by taking a shortcut, even though it’ll slow you down later. The metaphor holds up pretty well. Technical debt tends to grow over time if you don’t keep an eye on it. A system with a lot of technical debt is harder to work on, and therefore progress is slower, than one without; this is like paying interest. And so forth.

Given that the current recession was caused by the collapse of the subprime mortgage industry, is there a way to extend the metaphor of technical debt to include subprime debt? I think so. Subprime mortgages were those issued to people who couldn’t afford them, or who were uninformed about how the debt instrument actually works. (You mean, I can’t pay interest-only forever?) The analogy isn’t exact, but I think these are comparable to two broad classes of debt that I’ve seen on software development projects. The first class are created by managers who take on excessive debt irresponsibly, attempting to achieve short-term goals at the cost of making long-term goals almost impossible to achieve. The second class occurs when engineers take on problems that are difficult or intractable without realizing it, even though this might already be known in the industry.

As an example of management irresponsibility, consider a project that is approaching its “feature complete” or “code complete” milestone. The idea is that coding of all the new features is supposed to be completed and checked in, though it’s acknowledged that it will have bugs, and there is room later in the schedule for bugfixing. (I actually think the idea of having a schedule with a “feature complete” milestone followed by bugfixing is completely bogus, but that’s a topic for another article. Many projects do seem to be run this way, however.) Inevitably, time to implement the features starts to run short, and managers will attempt to remain “on schedule” by telling developers to write code more quickly, that they don’t need to worry about bugs (because we’ll have time to fix them later), that they can just implement a skeleton of the feature initially (and implement the corner cases as bugfixes later), and so forth. Taken to the extreme, no code for feature X is actually integrated, but a bug “feature X doesn’t work” is filed, which allows the project to reach feature complete “on time,” and which allows the feature to be developed later under the guise of bugfixing.

Of course, this is completely silly way to run a project. Any bugfixing time in the schedule is intended for fixing unknown, unforeseen bugs. Using this time to work on “known bugs” (really, incomplete features) and even on outright feature implementation leaves too little time to work on the unknown bugs, and the bugs arising from features being implemented late and in a hurry. In reality, the schedule has already slipped, but the managers haven’t realized it yet. Or worse, they’ve realized it, but they aren’t willing to admit it. This is reckless and irresponsible. It happens all the time.

As an example of engineer naïveté, let me refer you to an article from Jeff Atwood’s Coding Horror blog on cross-site scripting. Basically, if you let users input arbitrary HTML and you include it directly on your page, they can use this to take over your site! Oh, well, to avoid this you just have to sanitize the HTML before including it. Should be a few regexps, right? Wrong. To do this properly pretty much requires writing a full-blown HTML scanner and parser that mimics the behavior of the scanner and parser in the actual web browser. If you don’t get this exactly right, it means that somebody will be able to find some fragment of HTML that gets past your sanitizer but does malicious things when processed by the browser. Dealing with SQL injection attacks is similar. If you approach these problems thinking you’ll use pattern-matching instead of parsing, you condemn yourself to a never-ending series of hacks, patches, workarounds, and kludges that will make the system unmaintainable and yet doesn’t really solve the problem. Some people might consider this job security, but what a crappy job.

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.
— Jamie Zawinski

Whether through management irresponsibility or engineering naïveté, these cases create an unsustainable amount of technical debt. The result is, if not outright failure, a death march.

(Hat tip to Robert McIlree for an article comparing subprime mortgages to technical debt, which didn’t really inspire this article, but which does predate it.)


Read Full Post »

In the past few months, I’ve read most of Weinberg’s Quality Software Management series. There are four books in this series:

1. Systems Thinking
2. First Order Measurement
3. Congruent Action
4. Anticipating Change

I first heard of this series several years ago, and I had a couple reactions. The first was to be intimidated. Wow, four volumes about software quality. I bet they’re full of charts and graphs and statistics, because that’s what it takes to make quality software, right? And I bet they’re deathly dull, too. (Boehm’s Software Engineering Economics is like this.) As it turns out, Weinberg’s series is far from intimidating and is in fact quite accessible. It’s mostly about people and teams and how they interact. Software is almost incidental. There are graphs and diagrams, but they’re all pretty qualitative. That is, they don’t show numbers. They show trends and relationships. For example, as the complexity of a problem increases, the effort required to solve the problem increases exponentially, not linearly. It doesn’t matter what the exact numbers are; it’s the shape of the curve that counts.

My second reaction was about the ambiguity in the title. Are these books about management techniques that lead to high-quality software, or about high-quality management in software projects? Weinberg addresses this issue at the very beginning of volume 1. The answer is: both. In order to produce high quality software, the quality of management must be improved. The way this is done is to consider software projects as systems of people, which leads into the heart of the subject matter of the first volume.

I stalled out reading volume 4. What I read of it is very good, and it’s a logical continuation of the preceding volumes. But it’s too advanced for me and my projects. We’re still stuck working on stuff that’s covered in the first couple volumes. We’re set up in a particular way, and stuff manages to get done after a fashion. But I don’t actually believe that people consciously understand how their actions affect the system. Basically they’re reacting to situations as they come up, and dare I say, their reactions are often not congruent.

Read Full Post »