Continuous integration at Medallia
Continuous integration is an important tool for finding issues early and is especially useful in team environments. Here at Medallia, we have built our own continuous integration system which automatically builds all our code, runs the test suite and presents the results in a nice UI.
About 3 years ago we switched from Subversion to Mercurial which allows us to improve our workflow by keeping all features on separate branches until they are ready to be merged. Each release gets its own integration branch, and these are numbered sequentially; our current release branch is called e55 and the next will be e56. Once a feature has been developed and tested, it is merged to the next release branch.
It can be hard to visualize a Mercurial repository with many branches, and thus we have developed a simple web application which helps with this; this is what it looks like:
At the top, the two latest release branches are listed, followed by the unmerged branches for each release branch. It also shows the case number in our bug tracker, who made the last commit on the branch, how long ago and how many tests failed. Clicking on the changeset link shows which tests actually failed:
This screen shows the last few commits on the branch, which tests failed and the execution time of each test. Notice that each test is actually run four times for each changeset; they are run in different order to allow us to detect code which depends on static variables being initialized by other tests.
Finally there is a tree view which makes it easier to visualize branches and how they are merged:
Clicking on a node in the graph moves the viewpoint to that node, thus making it easy to navigate around in the tree. By default one level of children and 5 levels of parent are shown, and these can be adjusted as necessary.
Together these tools make it reasonably easy to keep track of our repository and manage all the branches. So how is all of this implemented? The web interface is pretty simple: the output from 'hg log' is parsed and used to build a representation of the tree. We use dot to draw the graphical tree view.
Actually running the tests is a bit more interesting. We have a stand-alone program which executes 'hg pull' every few seconds to grab any new revisions pushed to the repository; once it finds a new revision it hands it off to a separate thread which then compiles the code and starts a separate Java process which actually runs the test cases. This process writes the test results to an XML file every few seconds so we don't have to wait until all the tests have completed before seeing any results (quick feedback is very important).
Since the tests require a lot of resources to run, we actually use several servers to run them; ssh is used to execute commands and start the tests on remote servers, and they write the test results to a shared NFS mount. This way we can test as many changesets concurrently as we want (just limited by how many servers we want to dedicate).
The system is completely automated and thus requires little maintenance. A big advantage of a home-grown system such as this is that it is simple and integrates everything we need in just a few screens. It also makes it easy to add other useful features, such as integration with our bug tracker, so that it shows which features have actually been merged to the release branch.