Devoxx UK 2013: Testing Presentations

I’m finally catching up with my backlog of items dating back to Devoxx UK 2013, which was in March!

There were a couple of testing-related events I participated in. The first was an OpenJDK TestFest, sponsored by the London Java Community, IBM, and Oracle. This wasn’t officially part of Devoxx. It was held at the Oracle offices in London the Saturday prior to Devoxx itself. There were several presentations; I gave a brief talk on OpenJDK Testing Pitfalls (slides). People spent some time hacking on actual tests, but I thought the presentations and the ensuing discussion were very helpful as well.

The other testing-related event I participated in was an OpenJDK BOF with Martijn Verburg. Hm, it’s entitled “OpenJDK Hack Session” but there wasn’t that much hacking there, though Martijn did demonstrate the new OpenJDK build system. I presented some additional material on Testing OpenJDK (slides). This presentation was less about writing tests for OpenJDK than about the difficulties we have testing OpenJDK. The biggest problem, I think, is with unreliable tests. One would hope that a failing test means that there is a bug in the system being tested. Unfortunately in OpenJDK we have a lot of tests that are only 99% reliable. If you run the test suite regularly, especially on all the platforms, that makes it very unlikely that you’ll get a test run with 100% of tests passing, even if there are no bugs in the system. Worse, there are bugs in the system, so test failures caused by actual bugs are mixed with spurious test failures.

You can see this in the test results that Balchandra Vaidya has been posting to the OpenJDK quality-discuss mailing list. See the jdk8 b88 test results posting, for example. If you click through some links to find the Results Archive page, you’ll see that there have been 12-17 failures out of 4,000 or so tests in the JDK test suite, for the past twenty or so JDK 8 builds. Worse, they haven’t been the same failures every time, since the code and tests are constantly being modified, and the set of tests being run is shifting around as well.

This is clearly a serious problem, one that I and others at Oracle hope to make progress on in the coming months.