How Postgres-XL is tested

The purpose of this blog is to explain the process of Quality Analysis that Postgres-XL goes through internally at 2ndQuadrant. Here, I describe the bare minimum tests that each release goes through, not to mention the other tests carried out by many 2ndQuadrant and other community members.

•   Regression tests are carried out to ensure that the defect fixes or enhancements to PostgreSQL have not affected Postgres-XL.

We want to keep up-to-date with PostgreSQL features and performance enhancements. Also, many tools may work with only newer PostgreSQL releases. So, we merge each PostgreSQL minor release in a timely fashion. When merging the regression tests therein, we need to continuously and consciously ensure if the new features are supported in Postgres-XL fine. Sometimes, there may be a gap in the expected outputs of PostgreSQL and Postgres-XL, which has to be tracked.

• Functional tests are done to validate that the functionalities of Postgres-XL are working as per business requirements. We ensure that all features are functioning as they are expected to.

Initially we created the ‘Major differences and Limitations’ tests’ module. Postgres-XL project (Postgres-XC, initially) was based on PostgreSQL 9.2. It picked up speed again, during PostgreSQL 9.5 development timeframe. Due to this, there is a known gap of features that Postgres-XL would like to support, but does not currently. We keep track of these with our xl_* functional tests. These tests cover limitations like materialized views, event triggers, foreign data wrappers, etc. Also, on the other hand they cover positive functional tests for features like BRIN, logical decoding of WAL data, jsonb, etc.

Additionally, each time a new feature or enhancement is added, we keep adding tests to validate the functionality.

• Usability tests are performed to validate the ease with which the user interfaces can be used.

We have cluster-setup-utility tests. Creating a Postgres-XL cluster manually requires quite a few steps. We have automated these for ease-of-use with pgxc_ctl utility. For simple prototype use, we have added ‘prepare minimal’ way. Also, for seasoned users, we have added the ‘prepare empty’ way where they can provision node-by-node, for their specific use. We have automated TAP tests for this utility.

• Recovery tests are done to test how well Postgres-XL is able to recover from crashes, hardware failures and other similar problems.

Postgres-XL being deployed as a cluster, we realize the importance of data consistency across node crashes. We have crash-recovery test scripts that crash/kill nodes and bring them up again. In parallel sessions, we keep on making database changes with SQL statements, transactions or prepared transactions. We verify that nodes (or their configured standbys) come up fine. We perform data sanity checks to verify proper recovery.

• Bug tracking is done for all known/reported bugs in our internal tracking system.

• Future steps

We are looking into using SQLsmith to generate random queries to detect further problems in Postgres-XL.

Also, we are in the process of setting up continuous integration server for automated builds, deployments and tests  of Postgres-XL project.