How does Haruspex work?

Haruspex provides you with a free-form query interface for your CI data, and it does so by processing a relatively standard file across languages: JUnit XML. Initially produced by Java's JUnit test framework, early CI systems like Hudson, Jenkins and Bamboo started parsing these files in order to gain more transparency over whether or not, for example, the CI run had suceeded, failed because the tests failed, or failed because of technical issues.

Haruspex follows a similar approach, but we don't view each CI run as being independent. The accurate picture of the health of your CI system comes from aggregating and viewing trends over time, not from individual runs.

Getting Started With Your CI Pipelines.

Generating JUnit Files

First, we need to set up the generation of JUnit files by your test framework. Here's some useful links and guidelines for some of the most common languages:

Regardardless of what langauge you are using, once your setup is producing JUnit files, you're ready to continue in this guide.

Uploading to Haruspex

In this guide we assume that your setup produces one JUnit file per CI run, but for example, in Java/JUnit it is one Junit file per test class. You will need to upload each file in order to gain a complete picture.

To do this, add another step to your CI script:

curl -X POST \
    --data-binary @report.xml \
    -H "Content-Type: application/xml" \
    -H "Authorization: $HARUSPEX_DATASET_SECRET" \
    https://ui.haruspex.dev/api/$DATASET_NAME/junit

This will now upload your test results to Haruspex on every CI invocation, and allows you to get to the bottom of what is eating your CI performance.

Using Haruspex

Trigger a CI run with the repository you modified, then go to the Haruspex user interface, and start exploring your test data.

Bazel

Making Haruspex work with Bazel is straightforward, the needed files are part of the testlogs bazel generates by default. You can find them under bazel-testlogs/$TARGET/test.xml, and upload them from there.

This covers the complete invocation of your bazel test $TARGET and will only produce one file.

Java

Maven

The surefire plugin that is responsible for producing the XML Haruspex consumes usually is already included.

After running mvn test, you can find the XML result files under target/surefire-reports/*.xml. Make sure to upload all of them to Haruspex. A note here is that each resulting file will get its own UUID in Haruspex.

Gradle

Gradle also generates XML reports by default. These can be found under $buildDir/reports/.

SBT

For SBT, see here. The long story short is that it also automatically generates XML reports.

JavaScript

Note: We have not vetted any of these projects, nor can be provide support for them. Haruspex consumes JUnit XML reports right now, if you require another format, please let us know!

Mocha

There is a third-party reporter available here.

  1. add the reporter required: yarn add --dev mocha-junit-reporter
  2. run your tests: mocha test --reporter mocha-junit-reporter
  3. The resulting XML file will be produced at ./test-results.xml

Jest

There is a third-party reporter available for Jest here.

  1. Add the reporter required to your project: yarn add --dev jest-junit
  2. Tell Jest to run the reporter, by adding it to the reporter list in your Jest config.
  3. By default this file is called junit.xml. For more information, see the site of the plugin.

Jasmine

There is a collection of reporters available for formatting your results, you can find the repository here. From that list, we need the JUnitXmlReporter.

Python

Note: We have not vetted any of these projects, nor can be provide support for them. Haruspex consumes JUnit XML reports right now, if you require another format, please let us know!

Pytest

Pytest comes with the option to generate JUnit XML files built in!

Append --junitxml=report.xml to your pytest invocation, and then upload the report to Haruspex after your test run.

unittest

For the built-in way of testing, there is a third-party XML reporter. You will need to adapt your test runner to use the XML reporter, which is documented in the repository.

Nose Tests

Nose, too, has an inbuilt way of generating JUnit XML: append the parameter --with-xunit. This is documented here.

Ruby

Everyone's favorite gemstone.

Note: We have not vetted any of these projects, nor can be provide support for them. Haruspex consumes JUnit XML reports right now, if you require another format, please let us know!

RSpec.

There is a XML formatter available that you will have to install and configure in order to generate JUnit XML reports.

Minitest

There is also a compatible reporter for minitest available here.

Test::Unit

There is also a formatter for the built-in way of running unit tests, to be found here.

Scala

In SBT versions past 0.13.5, JUnit XML functionality is built into SBT, and reports are automatically generated for every test in the testsuite.

You can find these reports under target/test-reports. A note here is that each resulting file will get its own UUID in Haruspex.

Swift

Swift has a library available for both, Server-Side Swift (SSS) and XCode. You can find it here, install it by following the instructions provided in the repository.

After you're done and your test run is successfully generating Junit files, you can proceed with uploading these files to Haruspex.

Basic Queries

When opening Haruspex, you will see a table of the raw data you have uploaded, and a form used to aggregate this data.

Aggregate

The first field is there for how you want to aggregate the data. If left empty, it defaults to COUNT - simply counting how often each element appears in the dataset. The accepted aggregates are:

  • COUNT
  • COUNT_DISTINCT
  • MIN
  • MAX
  • AVERAGE
  • SUM

The syntax is simple functions separated by commas. There is no limit in the number of aggregates you can combine. Each will be its own graph.

COUNT(column_foo), COUNT_DISTINCT(column_baz)

Group By

This field is there for filtering down on which parts of the data you want to focus on.

A simple way to intuit how this field is to simply enter a column name you see in the Raw Data: this results in a graph of how often each value for that column appears in the dataset.

Filters

This is how you can narrow down the considered data with predicates. An example would be entering exists, which removes all empty values from the result set.

A query of Group By: failure.text Filters: exists(failure.text) would therefore show you all the different failure texts while removing all the results in which this was not set.

The accepted filters are:

  • exists
  • does-not-exist
  • starts-with
  • does-not-start-with
  • contains
  • does-not-contain

The syntax is simple functions separated by commas. There is no limit in the number of filters you can combine.

exists(column_foo), starts-with(column_baz, prefix to start)

WARNING: we do not support (), in the arguments to filters.