Regression evaluation using behavior models of software applications

Patent metadata

Field	Value
Publication number	US-9594665-B2
Application number	US-201414198284-A
Country	US
Kind code	B2
Filing date	Mar 5, 2014
Priority date	Mar 5, 2014
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

Comparisons of different versions of an application may be compared using a behavior model of the application. A behavior model may be derived from n-gram analysis of observations of the application in production. The behavior model may include sequences of inputs received by the application or functions performed by the application, where each sequence is an n-gram observed in tracer data. Each n-gram may be coupled with a resource consumption to give a behavior model with performance data. A regression analysis may apply a behavior model derived from a first version of an application to the performance observations of a new version to create an expected performance metric for the new version. A similarly calculated metric from a previous version may be compared to the metric from a new version to determine an improvement or degradation of performance.

First claim

What is claimed is:

1. A method performed by at least one hardware processor, said method comprising:
receiving a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;
generating a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, (ii) determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (iii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;
receiving a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment;
generating a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and
generating a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including:
one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; and
one or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.

2. The method of claim 1 , each representative sample comprising at least 50% of a total number of n-gram observations.

3. The method of claim 1 , each representative sample comprising at least 75% of a total number of n-gram observations.

4. A system comprising:
one or more hardware processors; and
one or more computer-readable media having stored thereon computer-executable instructions that are executable by the one or more hardware processors to implement a version analyzer that is configured to perform at least the following:
receive a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;
generate a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (ii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;
receive a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment;
generate a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and
generate a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including:
one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; and
one or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.

5. The system of claim 4 , each representative sample comprising at least 50% of a total number of n-gram observations.

6. A computer program product comprising one or more memories having stored thereon computer-executable instructions that are executable one or more processors to perform at least the following:
receive a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;
generate a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (ii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;
receive a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Baril Bryce B
• Gounares Alexander G
• Krajec Russell S

Classifications

• G06F11/3428

  Benchmarking · CPC title

• G06F2201/865

  Monitoring of software · CPC title

• G06F11/3409

  for performance assessment · CPC title

• G06F11/3466

  Performance evaluation by tracing or monitoring · CPC title

• G06F11/3452

  Performance evaluation by statistical analysis · CPC title