This is a very belated post that covers a session that took place at the JavaOne conference in San Francisco, October 2017.
Here’s a recap of the BOF (“birds-of-a-feather”) session I led on software maintenance. The title was Maintenance – The Silent Killer. This was my feeble attempt at clickbait. This was an evening session that was held during the dinner hour, and maintenance isn’t the most scintillating topic, so I figured attendance needed all the help I could give it.
When the start time arrived, I was standing on the podium in an empty room. I thought, well, if nobody shows up then I can go home early. Then about fifty people flooded in! It turns out they had lined up outside waiting for their badges to be scanned, but then a conference staffer came by and told them that badges weren’t scanned for the evening sessions and that they should just go in.
Overall I thought it went quite well. I gave a brief presentation, and then set up some discussion questions for the audience. The people who showed up really were interested in maintenance, they offered a variety of interesting insights and views, and they were quite serious about the topic. There was enough discussion to fill the allotted time, and there was plenty of interaction between me and the audience and among audience members themselves. I’ll declare the session to have been successful, though it’s difficult for me to draw any grand conclusions from it. I was heartened by the amount of participation. I was really concerned that nobody would show up, or perhaps that three people would show up, since most tech conferences are about the latest and greatest new shiny thing.
The session wasn’t recorded. What follows is some notes on my slide presentation, followed by some additional notes from the discussion that followed. These are unfortunately rather sparse, as I was participating at the same time. However, I did capture a few ideas that I hadn’t considered previously, which I found quite beneficial.
Slide 2: Golden Gate Bridge. I grew up in Marin County, which is connected to San Francisco by the Golden Gate Bridge. We crossed the bridge frequently. Back in 1974 or so the toll was raised from 50¢ to 75¢, and my parents complained incessantly about this. At one point I had the following conversation with my Dad about the toll:
Me: Why do they collect tolls?
Dad: To pay off the bridge.
Me: When will the bridge be paid off?
Dad: Never!
As I kid I was kind of perplexed by this. If you take out a loan, and make regular payments on it, won’t it eventually be paid off? (Sub-prime mortgages weren’t invented until much later.) Of course, the original construction loans have long since been paid off. What the tolls are used for, and which indeed will never be paid off, is the continuous maintenance that the bridge requires.
∞
Slide 3: This is me driving my car through Tunnel Log in Sequoia National Park. The point isn’t about a tunnel through a tree, but the cost of owning and operating a car. The first time I used my car for business expenses, I was surprised by the per-mile reimbursement amount. If you consider the 2017 numbers, this car’s gasoline costs about 14¢-20¢ per mile, and the IRS standard reimbursement rate is 53.5¢ per mile. Hey, I’m making money on this deal!
No. This is a 1998 BMW, and you will not be surprised to learn that the cost of maintenance on this car is quite significant. Indeed, I’ve added up the maintenance costs over the lifetime of the car, and they outweigh the cost of gasoline. Counting maintenance and depreciation, I’m decidedly not making money on mileage reimbursement.
∞
Slide 4 has some points on maintenance as a general phenomenon. One point that bears further explanation is my claim that “deferred maintenance costs can grow superlinearly.” Continuing with the car example, consider oil changes. It might cost a couple hundred dollars a year for regular oil changes. You could save money for a couple years by not changing the oil. This might eventually result in a several thousand dollar engine rebuild. “Superlinear” isn’t very precise, but the point is that the cost of remediating problems caused by deferred maintenance is often much greater than the sum of incremental maintenance costs.
∞
Slide 5, quotation from Kurt Vonnegut. Perhaps profound if you’ve never heard it before, but a cliché if you pay attention to maintenance. It does seem to be true that in general creative activities get all the attention at the expense of maintenance activities.
∞
Slides 6-7. Physical systems exhibit wear and friction and this contributes to the need to do regular maintenance. Software doesn’t “wear” out. But there are a bunch of phenomena that cause software systems to require maintenance. Primarily these seem to be related to the environment in which the software exists, not the software itself.
∞
Slides 8-9. Most planning and costing efforts around software projects are concerned with software construction. Maintenance is a significant cost, accounting for perhaps 50% to 75% (Boehm) or 40% to 80% (Glass) of the total life cycle costs. However, comparatively little planning and budgeting effort goes toward maintenance.
Glass points out that software maintenance and construction are essentially the same activity, except that maintenance requires additional effort to “understand the existing product.” As a programmer, when you’re developing software, you know what you’re trying to do and you’re familiar with the code you’re developing at the moment. When maintaining software, you often have to deal with code that you might never have seen before and figure out what it does before you can modify it successfully. The cost incurred in re-acquiring knowledge and understanding of an existing software system is significant.
∞
Slide 10. OpenJDK is an open source implementation of Java. It’s an old code base; Java 1.0 was released in 1996, and it was in development for a couple years prior to that. It’s been continually evolved and maintained since then. Evolution consists of usual software activities such as adding features, improving performance, fixing bugs, mitigating security vulnerabilities, and maintaining old releases. Maintenance activities are a large portion of the team’s activities. I’m not sure how to measure it, but the estimates from Boehm and Glass above are quite plausible.
In addition to the above development activities the team also puts effort into deprecation and removal of obsolete features. This is important because, among other things, it helps to reduce the long-term maintenance burden. See some of my prior materials on the topic of deprecation:
- JEP 277, Enhanced Deprecation
- JavaOne 2014 session, Debt and Deprecation
- JavaOne 2015 session, Innovations in Deprecation
- Deprecation of
Object.finalize() - Dr Deprecator on Twitter
The cost of knowledge re-acquisition mentioned previously is somewhat mitigated by systems maintained by the JDK group that preserve history.
The open version of the JDK source code in the Mercurial version control system, and it includes changesets dating back to December 2007. The earlier source code history is in a closed, Oracle-internal system and dates back to August 1994.
The JDK Bug System (a JIRA instance) contains over 265,000 bugs and feature requests dating back to late 1994. Many of these bugs were converted from a Sun Microsystems internal bug database.
Personally, I’ve found that the ability to search over 20 years of source code history and bug history to be of immense value in understanding existing code and diagnosing problems.
∞
Slide 11. A big driver of software maintenance is security vulnerabilities. This has gotten worse in recent years, as “everything” is connected to the internet. Another significant contributor to maintenance issues is the large number of dependencies among software components, many of which are in open source. By reusing external software components, you can reduce development time. However, doing so takes on the maintenance burden of those components. Either you have to keep all the external components up to date, or you have to maintain them yourself.
∞
Slide 12. Questions and Audience Discussion
The slide has several questions to spark discussion with the audience. We didn’t address them directly, but there was a relatively free-flowing conversation. Here are some notes from that conversation.
One audience member compared maintenance to a fence. Suppose you have a pasture, and wolves keep coming to it and attacking your sheep. So you put up a fence. The fence just sits there. The sheep grace peacefully. Wolves stay away because they realize they can’t get past the fence. Nothing happens. The fact that nothing is happening is a huge benefit! Like a fence, a well-maintained system just does its thing without calling attention to itself. This may lead people to forget about it. A poorly-maintained system is constantly breaking, attracting lots of attention.
An attendee suggested thinking about maintenance planning the same way a project manager thinks about risk management. With less maintenance there is a greater risk of failure, and vice-versa.
Another attendee suggested insurance as a model for maintenance. Maintenance costs are like insurance premiums: you pay them regularly, and you’re protected. Not paying them saves money temporarily, until some disaster strikes. (Rather like my car oil change example above.) Of course, insurance is closely related to risk management, and as a social institution it seems poorly understood by most lay individuals.
An audience member suggested just biting the bullet and declaring that maintenance is just a cost of doing business. There’s no use complaining about it; you just have to accept it. Another audience member said that his department allocated 10% of its budget to maintenance costs.
Regarding keeping up with the software updates, one attendee pointed out that it’s not necessarily important to be on the latest software release, but instead it’s important to be on the latest patch or update level even if you’re on an old release. Many commercial software products have support contracts where they will maintain old releases for many years. They don’t have the most features or the highest performance, but they are maintained with fixes for current security vulnerabilities and other high priority problems.
(This is a big component of the business of my company, Oracle. This is also true of products from many other software companies.)
Synchronous Messages 