Automatic comparison and performance analysis between different implementations

Patent metadata

Field	Value
Publication number	US-9317629-B1
Application number	US-76532410-A
Country	US
Kind code	B1
Filing date	Apr 22, 2010
Priority date	Dec 29, 2005
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

The present invention provides a method and system for automatic verification of automatically generated standalone code intended for execution on a target computing platform against its original design simulated in a simulation environment. The present invention also applies to execution comparisons between two implementations, such as two simulations, one simulation and one standalone code implementation, or two standalone code implementations. Block diagrams can be used to create a comparison model that compares two implementations. The comparison of different implementations can be performed at a block level, a subsystem level, a model level, or multi-model level. The present invention allows automatic comparison once a user supplies the intermediate outputs and/or signals that the user wants to compare and monitor. Reports can be generated to show the statistics of the comparison results.

First claim

We claim:

1. A method comprising:
automatically generating, by a processor of a host computer, an executable comparison block diagram model including:
a first portion that represents a first simulation as a plurality of interconnected graphical blocks, outputs of at least some of the interconnected graphical blocks of the first portion representing intermediate outputs in the first simulation, and
a second portion that represents a second simulation as a plurality of interconnected graphical blocks, outputs of at least some of the interconnected graphical blocks of the second portion representing intermediate outputs in the second simulation, where
the first and second simulations represent different implementations of a design, and
the first and second simulations are configured for independent execution;

storing the executable comparison block diagram in a memory;
identifying for comparison, with a difference block in the comparison block diagram model, a first intermediate output from the first simulation and a second intermediate output from the second simulation, the difference block being separate from the first and second portions that represent the first and second simulations;
providing a switching capability at the second portion of the executable comparison block diagram model, the switching capability controlling whether a next component of the second portion of the executable comparison block diagram model receives:
the first intermediate output of the first simulation, or
the second intermediate output, where
the next component of the second portion of the executable comparison block diagram model is relative to the second intermediate output in an execution order;

instructing the first simulation and the second simulation to execute simultaneously in a locked-step manner during execution of the comparison block diagram model;
automatically comparing, by the processor, at least one value of the first intermediate output with at least one value of the second intermediate output during the execution of the comparison block diagram model including the first and second portions that represent the first and second simulations; and
generating a report comparing the first simulation with the second simulation.

2. The method of claim 1 further comprising:
executing the first simulation and the second simulation on different host computers.

3. The method of claim 1 further comprising:
identifying a third intermediate output from the first simulation and a fourth intermediate output from the second simulation; and
comparing at least one value of the third intermediate output with at least one value of the fourth intermediate output during execution of the first and second simulations.

4. The method of claim 3 , wherein the third intermediate output is an output of a first block using the first intermediate output as an input, and the fourth intermediate output is an output of a second block using either the second intermediate output or the first intermediate output as an input.

5. The method of claim 1 further comprising:
receiving comparison instructions specifying at least one or a combination of the following: a number of test runs, a range of system input sources, a range of subsystem-level design parameter values, a range of independent subsystem-level error tolerances, a range of accumulated subsystem-level error tolerances, and a range of reporting metrics and formats.

6. The method of claim 1 , wherein the report includes statistics on at least one or a combination of: an error rate, an error distribution, and a sensitivity metric.

7. The method of claim 6 , wherein the error rate is at least one or a combination of: a mean time before error, a mean time between errors, a rate of error per time, and a rate of error per run.

8. The method of claim 6 , wherein the sensitivity metric comprises at least one or a combination of an independent subsystem level sensitivity to subsystem input and an independent subsystem-level sensitivity to subsystem parameters.

9. The method of claim 1 , wherein the first portion represents semantics of the first simulation and the second portion represent semantics of the second simulation.

10. A non-transitory computer-readable medium storing instructions executable by one or more computing devices, the instructions comprising instructions for:
generating automatically an executable comparison block diagram model including:
a first portion that represents a first simulation as a plurality of interconnected graphical blocks, outputs of at least some of the interconnected graphical blocks of the first portion representing intermediate outputs in the first simulation, and
a second portion that represents a second simulation as a plurality of interconnected graphical blocks, outputs of at least some of the interconnected graphical blocks of the second portion representing intermediate outputs in the second simulation, where
the first and second simulations represent different implementations of a design, and
the first and second simulations are configured for independent execution;

storing the comparison block diagram model in a memory;
identifying for comparison, with a difference block in the comparison block diagram model, a first intermediate output from the first simulation and a second intermediate output from the second simulation, the difference block being separate from the first and second portions that represent the first and second simulations;
providing a switching capability at the second portion of the executable comparison block diagram model, the switching capability controlling whether a next component of the second portion of the executable comparison block diagram model receives:
the first intermediate output of the first simulation, or
the second intermediate output, where
the next component of the second portion of the executable comparison block diagram model is relative to the second intermediate output in an execution order;

instructing the first simulation and the second simulation to execute simultaneously in a locked-step manner during execution of the comparison block diagram model;
automatically comparing at least one value of the first intermediate output from the first simulation with at least one value of the second intermediate output from the second simulation during execution of the comparison block diagram model including the first and second portions representing the first and second simulations; and
providing an output to a user comparing the first simulation with the second simulation.

11. The medium of claim 10 further comprising instructions for:
executing the first simulation and the second simulation on different host computers.

12. The medium of claim 10 further comprising instructions for:
identifying a third intermediate output from the first simulation and a fourth intermediate output from the second simulation; and
comparing at least one value of the third intermediate output with at least one value of the fourth intermediate output during execution of the first and second simulations.

13. The medium of claim 12 , wherein the third intermediate output is an output of a first block using the first intermediate output as an input, and the fourth intermediate output is an output of a second block using either the second intermediate output or the first intermediate output as an input.

14. The medium of claim 10 further comprising instructions for:
receiving comparison instructions specifying at least one or a combination of the following: a number of test runs, a range of system input sources, a

Assignees

• Koh David
• Ogilvie Brian K
• Mathworks Inc

Inventors

• Koh David
• Ogilvie Brian K

Classifications

• G06F11/3604 Primary

  Analysis of software for verifying properties of programs (testing of software G06F11/3668) · CPC title

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06F2207/7271

  Fault verification, e.g. comparing two values which should be the same, unless a computational fault occurred · CPC title

• G06F11/3668

  Testing of software · CPC title

• G06F17/5009 Primary

  Physics · mapped topic