I would like some help in finding prior art for patent WO2013165367A8 'Prioritization of continuous deployment pipeline tests'.

A 2009 paper by Jiang et al describes prioritization of tests: http://www.cs.hku.hk/research/techreps/document/TR-2009-09.pdf

What other examples of prior art for CI test case prioritization are there?

(Related to: Prior art request for WO2014027990 "Performance tests in a continuous deployment pipeline")

In reference to the patent: WO2013165367A8


The 2006 paper "Time-Aware Test Suite Prioritization" by Kristen R. Walcott and Mary Lou Soffa of University of Virginia, and Gregory M. Kapfhammer and Robert S. Roos of Allegheny College states in its abstract:

Regression test prioritization is often performed in a time constrained execution environment in which testing only occurs for a fixed time period. For example, many organizations rely upon nightly building and regression testing of their applications every time source code changes are committed to a version control repository. This paper presents a regression test prioritization technique that uses a genetic algorithm to reorder test suites in light of testing time constraints


Not an answer but Following arts are already known for this application:-

  1. US20090138856 System and method for software performance testing and determining a frustration index
  2. US20080256392 Techniques for prioritizing test dependencies
  3. US20060129994 Method and apparatus for prioritizing software tests
  4. US20090265693 Method and system for test run prioritization for software code testing in automated test execution
  • The US20060129994 prior is real and exists. I've used it in Team Foundation Server. Pretty complicated stuff to get up and running because you need successful coverage data first. – Mitch Denny Mar 9 '15 at 8:38

Atlassian added support for test prioritisation to Clover 2.4 in November 2008.

From http://blogs.atlassian.com/2008/11/stop_testing_so_much/:

The second approach, used in conjunction with the first or independently, is to prioritise those tests that are run, so as to flush out any test failures as quickly as possible.


Clover 2.4’s new test optimization can dramatically reduce your build times, taking the load off your CI server


On Jan 3rd 2007, Dennis Byrne, Kent Spillner, myself and a few other people from ThoughtWorks created a open source project called ProTest. Protest stands for Prioritized Tests. It is an intelligent test ordering tool, which aims to improve the test-feedback-cycles for the developers. ProTest maintains a history of test runs and knows what code changes affect which tests. ProTest is able to run the mostly-likely-to-fail tests first and hence is more effective.


I am a software engineering researcher (read: academic) and my expertise includes regression testing techniques. I'm mostly interested in academic publications, so the context of patent is definitely to me. As far as papers go, however, the earliest prioritisation paper that I'm aware of goes back to 1999:

Rothermel G, Untch RH, Chu C, Harrold MJ. Test case prioritization: An empirical study. Proceedings of International Conference on Software Maintenance (ICSM 1999), IEEE Computer Society Press, 1999; 179–188.

Now, I don't know what qualifies for the "Continuous Integration" part. If you mean prioritisation as part of an established CI tools, such as Jenkins, than very little academic work will count for that. However, if you broadly mean the context of regression testing (where you are testing the new version using, partly, the existing test suite from the previous versions), then the above is definitely one of the earliest.

To see a bit more detailed timeline, I'd recommend the following paper, which is a survey of regression testing techniques, and also happens to be written by yours truly: http://dl.acm.org/citation.cfm?id=2284813

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.