Prior art request for WO2014027990 "Performance tests in a continuous deployment pipeline"

Question

I would like help finding prior art for WO2014027990 A1, please.

The patent "Performance tests in a continuous deployment pipeline" by some employees of HP (filed Aug 13 2012, published Feb 20, 2014) describes how to establish automated performance testing in a deployment pipeline, which is an approach I was first aware of in 2008 when working with Dave Farley. In July 2010 Dave Farley and Jez Humble published the book "Continuous Delivery" which described in detail how to create an automated pipeline including automated latency, throughput, and soak testing.

I consider the patent to be without merit and am looking for help in finding prior art from the "Continuous Delivery" book and/or prior to book publication, preferably pre-2012.

Answer 1

Jez, Dan and myself published a paper at the Agile 2006 conference that covered this.

The Deployment Production Line - Continuous Delivery

Answer 2

My own answer is as follows:

The "Continuous Delivery" book by Dave Farley and Jez Humble published 27 July 2010 describes in detail performance testing in a Continuous Delivery pipeline. The publication date is 2 years prior to patent filing.

http://www.amazon.co.uk/dp/0321601912 Name: Continuous Delivery Authors: Dave Farley and Jez Humble Publisher: Addison Wesley; 1 edition (27 July 2010) Language: English ISBN-10: 0321601912 ISBN-13: 978-0321601919

Chapter 9: "Testing Nonfunctional Requirements" (pages 225-248) contains the following subsections:

1. "The Capacity Testing Environment" - describes how to create an environment for testing performance requirements, such as latency and throughput expectations
2. "Automating Capacity Testing" - describes how to automate that performance testing process
3. "Adding Capacity Tests To The Deployment Pipeline" - describes how to configure a deployment pipeline such that performance testing is an automated testing stage pre-production, with performance results compared against an expected threshold to determine success

Note that "continuous delivery pipeline" and "continuous deployment pipeline" are synonyms, as are "performance testing" and "capacity testing"

EDIT: A 2007 paper from ThoughtWorks Studios by Dave Farley called "The Deployment Pipeline" refers to performance testing in a Continuous Delivery pipeline. See http://continuousdelivery.com/wp-content/uploads/2010/01/The-Deployment-Pipeline-by-Dave-Farley-2007.pdf

Answer 3

Here's a video from 2011 of Chuck Rossi describing how they do exactly this at Facebook: http://techcrunch.com/2011/05/30/facebook-source-code/

Answer 4

In my web article on Continuous Integration I mention staged build / build pipeline and that you can use that to introduce performance testing into the regular build process. That first appeared when I revised the article in May 2006

Answer 5

These arts are already known for application:-

1. US20100005341 Automatic detection and notification of test regression with automatic on-demand capture of profiles for regression analysis
2. US20080270998 Application integration testing
3. US20100103839 Testing of Transmitters for Communication Links by Software Simulation of Reference Channel and/or Reference Receiver
4. US20060146318 Method and apparatus for self-testing of test equipment
5. KR101166128 SOFTWARE TESTING DEVICE AND METHOD THEREOF  

Answer 6

Note that I can't take credit - these are links sent to me by Jack Shirazi (author of Java Performance Tuning) when I asked him about prior art in this area.

IBM (2003) describes the procedure, but for performance tuning prior to deployment, not for comparisons across deployments in their Enterprise Java Performance: Best Practices p35 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.1040&rep=rep1&type=pdf

BEA in 2005 http://www.oracle.com/au/products/database/performance-testing-095962.html " It is always a good idea to run a series of baseline tests first to establish a known, controlled environment to compare your changes with later. "

Answer 7

Here's a DrDobb's article from 2008 talking about adding performance testing into a Continuous Integration environment:

Continuous Integration and Performance Testing

Answer 8

We (Dynatrace) have been talking and doing this for years. Back in 2010 we published the following blog post showing how we do this internally:

Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl

We also implemented specific features in our product that automatically takes performance and unit test KPIs (Key Performance Indicators) and measures them per build. We also baseline these metrics and alert in case there is a degradation in one of these measures.

Answer 9

As early as 2007 this was common practice on projects at ThoughtWorks, where both, Jez and Dave, worked at the time. One example we talked about publicly is the project to rebuild the Guardian website, guardian.co.uk back then, theguardian.com now.

In an interview with Software Engineering Radio, published in May 2008, Mat Wall and I describe the exact technique of including performance tests in a build pipeline. At 39:28 I say "The build pipeline in the case of the Guardian project, of course, continues; with the performance test."

Later on in the same interview the host, Markus Völter, remarks that the Eclipse team is using a similar technique.

The interview can be found here: Episode 95: The New Guardian.co.uk website with Matt Wall and Erik DoernenBurg

Answer 10

I wrote a paper in 2002 which used the term 'continuous deployment': Making Web Services that work

I believe this is the first written use of the term, though it was an obvious extension of Continuous Integration. indeed, it was based on Cruise Control. Which I covered along with Apache Gump in the 2001 book Ant in Action, proving that I was clearly aware of that prior art.

I don't recall explicitly mentioning perf testing, but automated tests during the deployment process are certainly called out.

This paper was written while I was working at HP in 2002. This raises an interesting question: does this actually count as a disclosure of the technique prior to the current patent application being filed?

Answer 11

Few years ago we used this http://blogs.atlassian.com/2009/05/bamboo_jmeter_aggregator_getti/ to have baseline performance tests in CI/CD pipelines. This plugin existed in 2009 and can break builds when certain baseline thresholds values are exceeded, which pretty much shows this was a pattern not invented by HP, but in practice surely before.

Answer 12

There are already plenty of resources here, but I have to mention a book, that seems from today's perspective quite "devopsy":

Mike Clark (2004), "Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Applications", p55:

"How Frequently Should a Build Run?

[...] On a real-world project you’ll probably have different types of tests: unit tests, acceptance tests, performance tests, etc. You don’t want to wait for all of those tests to run just to see if your unit tests passed. To avoid that, each type of test would have a different Ant target and you’d configure CruiseControl to run each target on a different schedule."

The author, Mike Clark, also created JUnitPerf, a tool that is introduced in the "Java Extreme Programming Cookbook" (Burke, E. and Coyner, B., 2003, p187) with:

"These tools [JProbe etc] are not designed to execute automatically as part of a continuous integration process—which is where JUnitPerf enters the picture.

JUnitPerf, available from http://www.clarkware.com/software/JUnitPerf.html, is a tool for continuous performance testing."

Answer 13

I didn't look at the patent details (and would perhaps not be able to understand the subtleties), but I would say that many Eclipse Foundation projects were doing continuous integration and automated performance tests back in 2006-2008. I would think it must be possible to get references from presentations at former EclipseCon Conferences.

Answer 14

I published a pair of relevant blog posts in February 2012 (6-ish months before they filed). The posts walked through adding load testing of a website to a working, sample Continuous Delivery process. It walked through the scripts to perform the load testing, how to use prior runs as baselines, and the actual implementation in a Jenkins build process for a Continuous Delivery pipeline that had been built over 9 prior posts, starting in late 2011.

LessThanDot.com - Implementing WCAT to Load Test a Website

LessThanDot.com - Continuous Delivery - Adding the Load Testing Stage

Also, I found this quote in the definitions interesting from an "awareness of prior art" perspective:

Load and performance tests are rarely included in the tests due to cost and time constraints since the execution time of performance and load tests are longer than unit tests and application programming interface tests.

In my mind, "rarely" means "non-0".

IANAL, but here's my take on the claims:

1.1: The build server/source control defines the distinct builds

1.2: As pointed out in the post, prior builds (codebases) are the baseline to compare against

1.3: The load test scripts outlined in the posts

1.4: Uses a set of numbers for the performance value, including metrics around transactions, requests, and response time

2: see 1.4

3: I don't have an answer for this one. It doesn't make sense that you would simultaneously load test two builds and I don't believe their description assumes simultaneity either.

4: Build server monitoring source control

5: Specifically called out in the first post, see bolded "Tripwire"

6: 2 of these: source control, build history

7: build history and the produced chart/trend

8: not covered because running a build for a prior changeset was so obvious as to not be called out in this post (though it may be in prior ones)

9: build history

10: the first post

11: .1 and .2 are already covered, .3 is source control (unless HP doesn't use source control and instead is trying to reverse engineer how it got into the current state from a saved prior state, ie no source control)

11.4: I don't think I have this. it sounds like they would expect me to calculate a diff value between the two results which is not as useful as a trend of actual values because it limits you to one and only one baseline instead of an ongoing trend - but then they are also running their load testing simultaneously, so outside the magical world of "in theory they won't impact each other", their data is already a mess

12: build history + chart over time

13: build server

14: I did 14.3, generating a performance report

15: the article deploys to a VM and web server instance specifically for the load test prior to running the load tests against it, from a system dedicated for the CD process

I can see where they could claim that claim 3 and 11.4 make this hold water. It is non-obvious (to me at least) that one would run two load tests simultaneously and that you would store the results of each run as a delta from the very first baseline run. Because that seems like a very expensive way to make this whole process not very useful.

Answer 15

This should really be a comment on the question (or better yet, the question should be edited to reflect this), but: No patent or prior art can be reasonably discussed without studying the claims. This is not a patent on the "idea" of performance tests as part of continuous delivery.

The first claim is representative:

1. A processor implemented method to perform a set of performance tests on an application in a continuous deployment pipeline, the method comprising:

• identifying, with a decision engine, a set of code changes between a baseline build and a new build, the baseline build and the new build being two distinct builds in a performance test environment;

• obtaining, with a test engine, a baseline test result by executing a set of customized test scripts on the baseline build in a performance test environment, the baseline build including a first code base that successfully completed the set of performance tests in the continuous deployment pipeline;

• testing, with the test engine, the new build by executing the set of customized test scripts on the new build in a performance test environment to obtain a new test result, the new build including a second code base being tested in the continuous deployment pipeline; and

• determining, with a decision engine, a performance value by comparing the baseline test result and the new test result.

I think this fairly readable, but at a high level, this is about getting a diff between two builds, one of which is a "baseline" build that has successfully passed the performance tests, running the same tests on the new build, and comparing the performance test results. Further dependent claims talk about actually interesting things like attributing performance changes to specific code changes.

So relevant prior art would include one or more prior art references that teach each and every one of these steps alone or in combination. So far most of the answers talk about the general "idea", but this link in a comment above seems to come closest.

Relevant snippet:

Regression Analysis

The second part of Continuous Integration Performance Management is regression analysis. Regression analysis points out performance degredations caused by changing parts of the application code.

...

In order have the basis for regression analysis we need to collect performance diagnosis data for each automated test run – or at regular intervals like at the end of each development iteration. These test runs are then automatically compared analyzing differences in the execution behavior of test cases.

This does not mention comparing against "baseline build" but it should be straightforward to find other prior art references teaching that baselines are sensible things to have, the combination of which can support a 103 obviousness type rejection.

Answer 16

This was issued as US9183123B2. The examination process surfaced and considered 19 previous patent documents and 25 non-patent documents - one of which is this present stack exchange question.

No narrowing of claims took place during the U.S. prosecution - all claims were allowed as filed right out of the box. (As seen in Public PAIR)