Ask Patents is a question and answer site for people interested in improving and participating in the patent system. It only takes a minute to sign up.
Sign up to join this community
I would like some help in finding prior art for patent WO2013165367A8 'Prioritization of continuous deployment pipeline tests'.
A 2009 paper by Jiang et al describes prioritization of tests: http://www.cs.hku.hk/research/techreps/document/TR-2009-09.pdf
What other examples of prior art for CI test case prioritization are there?
(Related to: Prior art request for WO2014027990 "Performance tests in a continuous deployment pipeline")
In reference to the patent: WO2013165367A8
The 2006 paper "Time-Aware Test Suite Prioritization" by Kristen R. Walcott and Mary Lou Soffa of University of Virginia, and Gregory M. Kapfhammer and Robert S. Roos of Allegheny College states in its abstract:
Regression test prioritization is often performed in a time constrained execution environment in which testing only occurs for a fixed time period. For example, many organizations rely upon nightly building and regression testing of their applications every time source code changes are committed to a version control repository. This paper presents a regression test prioritization technique that uses a genetic algorithm to reorder test suites in light of testing time constraints
Not an answer but Following arts are already known for this application:-
Atlassian added support for test prioritisation to Clover 2.4 in November 2008.
From http://blogs.atlassian.com/2008/11/stop_testing_so_much/:
The second approach, used in conjunction with the first or independently, is to prioritise those tests that are run, so as to flush out any test failures as quickly as possible.
...
Clover 2.4’s new test optimization can dramatically reduce your build times, taking the load off your CI server
On Jan 3rd 2007, Dennis Byrne, Kent Spillner, myself and a few other people from ThoughtWorks created a open source project called ProTest. Protest stands for Prioritized Tests. It is an intelligent test ordering tool, which aims to improve the test-feedback-cycles for the developers. ProTest maintains a history of test runs and knows what code changes affect which tests. ProTest is able to run the mostly-likely-to-fail tests first and hence is more effective.
I am a software engineering researcher (read: academic) and my expertise includes regression testing techniques. I'm mostly interested in academic publications, so the context of patent is definitely to me. As far as papers go, however, the earliest prioritisation paper that I'm aware of goes back to 1999:
Rothermel G, Untch RH, Chu C, Harrold MJ. Test case prioritization: An empirical study. Proceedings of International Conference on Software Maintenance (ICSM 1999), IEEE Computer Society Press, 1999; 179–188.
Now, I don't know what qualifies for the "Continuous Integration" part. If you mean prioritisation as part of an established CI tools, such as Jenkins, than very little academic work will count for that. However, if you broadly mean the context of regression testing (where you are testing the new version using, partly, the existing test suite from the previous versions), then the above is definitely one of the earliest.
To see a bit more detailed timeline, I'd recommend the following paper, which is a survey of regression testing techniques, and also happens to be written by yours truly: http://dl.acm.org/citation.cfm?id=2284813
