Automatic comparison and performance analysis between different implementations

Patent metadata

Field	Value
Publication number	US-9317628-B1
Application number	US-32191905-A
Country	US
Kind code	B1
Filing date	Dec 29, 2005
Priority date	Dec 29, 2005
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

The present invention provides a method and system for automatic verification of automatically generated standalone code intended for execution on a target computing platform against its original design simulated in a simulation environment. The present invention also applies to execution comparisons between two implementations, such as two simulations, one simulation and one standalone code implementation, or two standalone code implementations. Block diagrams can be used to create a comparison model that compares two implementations. The comparison of different implementations can be performed at a block level, a subsystem level, a model level, or multi-model level. The present invention allows automatic comparison once a user supplies the intermediate outputs and/or signals that the user wants to compare and monitor. Reports can be generated to show the statistics of the comparison results.

First claim

We claim:

1. A computer-implemented method of comparing execution of a standalone code implementation and a simulation, said method comprising:
generating automatically, by a processor of a host computer, the standalone code implementation from the simulation using specification information for a target computing platform, the simulation being a model representing a design;
representing the standalone code implementation and the simulation in an executable comparison block diagram model on the host computer, where
the standalone code implementation is represented by one or more standalone code blocks of the comparison block diagram model,
the simulation is represented by one or more simulation blocks of the comparison block diagram model, and
at least one source block of the comparison block diagram model provides a same input to:
the one or more standalone code blocks that represent the standalone code implementation, and
the one or more simulation blocks that represent the simulation;

identifying a first intermediate output in the simulation and a second intermediate output in the standalone code implementation; providing a switching mechanism at the standalone code implementation in the executable comparison block diagram model, the switching mechanism configured to specify whether a next portion of the standalone code implementation receives the first intermediate output from the simulation, or the second intermediate output;
instructing the simulation to execute on the host computer and the standalone code implementation to execute simultaneously in a locked-step manner on the target computing platform; and
automatically comparing, by the processor, at least one value of the first intermediate output with at least one value of the second intermediate output during execution of the simulation and the standalone code implementation.

2. The method of claim 1 , wherein the standalone code implementation is implemented in hardware description language (HDL).

3. The method of claim 1 ,
wherein the target computing platform is external to the host computer.

4. The method of claim 1 further comprising:
sending from the host computer to the target computing platform instructions to execute the standalone code implementation.

5. The method of claim 1 , wherein at least one block in the comparison block diagram model is capable of bi-directionally communicating with the target computing platform.

6. The method of claim 1 further comprising the steps of:
identifying a third intermediate output from the simulation and a fourth intermediate output from the standalone code implementation; and
comparing a third value of the third intermediate output with a fourth value of the fourth intermediate output during execution of the simulation and the standalone code implementation.

7. The method of claim 6 , wherein the third intermediate output is an output of a first block using the first intermediate output as an input and the fourth intermediate output is an output of a second block using either the second intermediate output or the first intermediate output as an input.

8. The method of claim 1 further comprising:
receiving comparison instruction information specifying at least one or a combination of: a specified number of test runs, a range of system input sources, a range of subsystem-level design parameter values, a range of independent subsystem-level error tolerances, a range of accumulated subsystem-level error tolerances, and a range of reporting metrics and formats.

9. The method of claim 1 further comprising:
generating a comparison report comparing execution of the simulation with the standalone code implementation.

10. The method of claim 9 , wherein the report includes statistics on at least one or a combination of: an error rate, an error distribution, and one or more sensitivity metrics.

11. The method of claim 10 , wherein the error rate is at least one or a combination of: a mean time before error, a mean time between errors, a rate of error per time, and a rate of error per run.

12. The method of claim 10 , wherein the one or more sensitivity metrics describes at least one or a combination of: an independent subsystem level sensitivity to subsystem input, an accumulated subsystem-level sensitivity to application input, an independent subsystem-level sensitivity to subsystem parameters, an accumulated subsystem-level sensitivity to parameters.

13. The method of claim 1 further comprising:
automatically generating the comparison block diagram model.

14. The method of claim 1 , wherein the first intermediate output and the second intermediate output are identified in the executable comparison block diagram model.

15. A non-transitory medium storing instructions for causing a computing device to compare execution of a standalone code implementation and a simulation, the non-transitory medium comprises the instructions for:
generating automatically, by a processor of a host computer, the standalone code implementation from the simulation using specification information for a target computing platform, the simulation being a model representing a design;
representing the standalone code implementation and the simulation in an executable comparison block diagram model on the host computer, where
the standalone code implementation is represented by one or more standalone code blocks of the comparison block diagram model,
the simulation is represented by one or more simulation blocks of the comparison block diagram model, and
at least one source block of the comparison block diagram model provides a same input to
the one or more standalone code blocks that represent the standalone code implementation, and
the one or more simulation blocks that represent the simulation;

identifying a first intermediate output in the simulation and a second intermediate output in the standalone code implementation; providing a switching mechanism at the standalone code implementation in the executable comparison block diagram model, the switching mechanism configured to specify whether a next portion of the standalone code implementation receives the first intermediate output from the simulation, or the second intermediate output;
instructing the simulation to execute on the host computer and the standalone code implementation to execute on the target computing platform simultaneously in a locked-step manner; and
automatically comparing, by the processor, at least one value of the first intermediate output with at least one value of the second intermediate output during execution of the simulation and the standalone code implementation.

16. The non-transitory medium of claim 15 , wherein the standalone code implementation is implemented in hardware description language (HDL).

17. The non-transitory medium of claim 15 wherein the target computing platform is external to the host computer.

18. The non-transitory medium of claim 15 , wherein at least one block in the comparison block diagram model is capable of bi-directionally communicating with the target computing platform.

19. The non-transitory medium of claim 15 further comprising the instructions for:
identifying a third intermediate output from the simulation and a fourth intermediate output from the standalone code implementation; and
comparing a third value of the third intermediate output with a fourth value of the fourth intermediate output during execution of the simulation and the standalone code implementation.

20. The non-transi

Assignees

• Koh David
• Ogilvie Brian K
• Mathworks Inc

Inventors

• Koh David
• Ogilvie Brian K

Classifications

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06F11/3604 Primary

  Analysis of software for verifying properties of programs (testing of software G06F11/3668) · CPC title

• G06F11/3668

  Testing of software · CPC title

• G06F2207/7271

  Fault verification, e.g. comparing two values which should be the same, unless a computational fault occurred · CPC title

• G06F17/5009 Primary

  Physics · mapped topic