Malicious software detection based on API trust

Patent metadata

Field	Value
Publication number	US-11947670-B2
Application number	US-202318092355-A
Country	US
Kind code	B2
Filing date	Jan 2, 2023
Priority date	Apr 13, 2018
Publication date	Apr 2, 2024
Grant date	Apr 2, 2024

Abstract

Examples of the present disclosure describe systems and methods for malicious software detection based on API trust. In an example, a set of software instructions executed by a computing device may call an API. A hook may be generated on the API, such that a threat processor may receive an indication when the API is called. Accordingly, the threat processor may generate a trust metric based on the execution of the set of software instructions, which may be used to determine whether the set of software instructions poses a potential threat. For example, one or more call stack frames may be evaluated to determine whether a return address is preceded by a call instruction, whether the return address is associated with a set of software instructions or memory associated with a set of software instructions, and/or whether the set of software instructions satisfies a variety of security criteria.

First claim

What is claimed is:

1. A system comprising:
a processor; and
a memory storing computer-executable instructions executable by the processor, the computer-executable instructions comprising instructions for:
generating a hook on an application programming interface (API);
receiving, in response to a set of software instructions calling the API, an indication from the generated hook;
based on the received indication, accessing a call stack associated with an execution of the set of software instructions, the call stack comprising a call stack frame;
generating a trust metric for an execution of the set of software instructions, wherein generating the trust metric for the execution of the set of software instructions comprises evaluating the call stack with a level-of-rigor determined based on an attempted action by the set of software instructions to generate an indication of a potential threat, the trust metric based on the indication of the potential threat determined from evaluating the call stack frame;
determining, based on an evaluation of the generated trust metric, that the set of software instructions poses a threat to computer security; and
based on determining that the set of software instructions poses a threat to computer security, performing a corrective action.

2. The system of claim 1 , wherein the set of computer-executable instructions comprises instructions for evaluating the call stack with a first level-of-rigor, wherein the instructions for evaluating the call stack with the first level-of-rigor comprise instructions for:
making a determination that a return address associated with the call stack frame is not preceded by a call instruction; and
generating the indication of the potential threat based on the determination that the return address associated with the call stack frame is not preceded by the call instruction.

3. The system of claim 2 , wherein the instructions for evaluating the call stack with the first level-of-rigor further comprise instructions for:
determining that the set of software instructions are not digitally signed;
determining that the set of software instructions are not in a trusted security catalog;
determining that that the set of software instructions are not whitelisted;
determining that a software reputation indicates a threat; and
based on a determination that the set of software instructions are not digitally signed, a determination that the set of software instructions are not in the trusted security catalog, the set of software instructions are not whitelisted, and that the software reputation indicates the potential threat, generating the indication of the potential threat.

4. The system of claim 3 , wherein the set of computer-executable instructions comprise instructions for evaluating the call stack with a second level-of-rigor, wherein the instructions for evaluating the call stack with the second level-of-rigor comprises instructions for:
determining that the return address is not associated with a memory location associated with the set of software instructions; and
based on a determination that the return address is not associated with the set of software instructions, generating the indication of the potential threat.

5. The system of claim 1 , wherein generating the trust metric comprises generating a cache entry in a cache for a return address of the call stack frame, wherein the cache entry comprises information associated with evaluating the call stack.

6. The system of claim 1 , wherein generating the trust metric comprises evaluating reputation information and wherein evaluating the reputation information comprises:
determining whether a local reputation data store comprises the reputation information;
when it is determined that the local reputation data store comprises the reputation information, accessing the reputation information from the local reputation data store; and
when it is determined that the local reputation data store does not comprise the reputation information, accessing the reputation information from a security service.

7. The system of claim 1 , wherein performing the corrective action comprises performing at least one action from the group consisting of:
generating a prompt comprising information associated with the threat;
generating an entry in a log comprising information associated with the threat;
terminating execution of the set of software instructions;
adjusting an execution parameter of the execution of the set of software instructions; and
gathering data associated with the execution of the set of software instructions.

8. The system of claim 1 , wherein the computer-executable instructions comprise instructions for:
evaluating a computing device to generate a whitelist indicating the set of software instructions is whitelisted; and
storing the generated whitelist in a local reputation data store.

9. A computer program product comprising a non-transitory, computer-readable medium storing thereon a set of computer-executable instructions executable by processor, the computer-executable instructions comprising instructions for:
generating a hook on an application programming interface (API);
receiving, in response to a set of software instructions calling the API, an indication from the generated hook;
based on the received indication, accessing a call stack associated with an execution of the set of software instructions, the call stack comprising a call stack frame;
generating a trust metric for an execution of the set of software instructions, wherein generating the trust metric for the execution of the set of software instructions comprises evaluating the call stack with a level-of-rigor determined based on an attempted action by the set of software instructions to generate an indication of a potential threat, the trust metric based on the indication of the potential threat determined from evaluating the call stack frame;
determining, based on an evaluation of the generated trust metric, that the set of software instructions poses a threat to computer security; and
based on determining that the set of software instructions poses the threat to computer security, performing a corrective action.

10. The computer program product of claim 9 , wherein the set of computer-executable instructions comprises instructions for evaluating the call stack with a first level-of-rigor, wherein the instructions for evaluating the call stack with the first level-of-rigor comprise instructions for:
making a determination that a return address associated with the call stack frame is not preceded by a call instruction; and
generating the indication of the potential threat based on the determination that the return address associated with the call stack frame is not preceded by the call instruction.

11. The computer program product of claim 10 , wherein evaluating the call stack with the first level-of-rigor comprises:
determining that the set of software instructions are not digitally signed;
determining that the set of software instructions are not in a trusted security catalog;
determining that that the set of software instructions are not whitelisted;
determining that a software reputation indicates a threat; and
based on a determination that the set of software instructions are not digitally signed, a determination that the set of software instructions are not in the trusted security catalog, the set of software instructions are not whitelisted, and that the software reputation indicates the potential threat, generating the indication of the potential threat.

12. The computer program product of claim 11 wherein the set of computer-execu

Assignees

• Open Text Inc

Inventors

• Sandoval Andrew L
• Myers David Alan
• Shaw Ii John R
• Klonowski Eric

Classifications

• G06F21/566 Primary

  Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities · CPC title

• G06F21/554 Primary

  involving event detection and direct action · CPC title

• G06F2221/033

  Test or assess software · CPC title

• G06F21/52

  during program execution, e.g. stack integrity {; Preventing unwanted data erasure; Buffer overflow} · CPC title