Code coverage stats

Many years ago, Michelle Caise submitted a patch to generate code coverage reports for the PostgreSQL code base, based on the lcov utility. Although I cannot find any record of an actual patch in the mailing list archives, Peter Eisentraut committed it some time later, and applied further refinements later.

Today I’m announcing a new PostgreSQL community service: code coverage reports generated automatically and updated daily using this infrastructure. This system compiles the master branch, runs make check-world, and then generates the HTML report with make coverage, which is what you see.

Sample code coverage report in src/backend/access/brin

For readers unfamiliar with code coverage, a quick summary: code is “covered” when there is some test suite that exercises it. Code that isn’t covered can easily get broken without anybody noticing, which is not good. To avoid code breaking silently, it’s important that a majority of lines are covered by tests. For a more complete explanation, here’s the Wikipedia page on the subject.

Our stats in this area have historically been rather bad; while many backend features are well covered, there are several features that are only partially covered, and others that are not covered at all. We’ve been improving in recent years; first we added the isolationtester, which enabled us to test features that only operate under concurrency. Second we added TAP tests, which were initially to cover the client utilities, but were later extended to also cover other things such as the WAL replay code and other things. But it’s clear that we have a long way to go yet.

There are some caveats to keep in mind. One is that the make check-world target (what this coverage tool reports about) is not what the buildfarm runs, so it may well be the case that the coverage reports is running more tests than the buildfarm — meaning that we’re claiming coverage without actually having it. Another is that coverage is run in a single platform (Debian on AMD64), so code for other architectures is not reported as covered.

So go out and explore! I mean, explore the report, figure out what parts of our code are not covered, and try to craft a test to fix that. We await your patch in pgsql-hackers with interest.

(Also: are there any tests that we should be running other than make check-world? Please leave feedback in comments).