Method and apparatus for detecting malicious software using handshake information

Patent metadata

Field	Value
Publication number	US-2016134646-A1
Application number	US-201414534429-A
Country	US
Kind code	A1
Filing date	Nov 6, 2014
Priority date	Nov 6, 2014
Publication date	May 12, 2016
Grant date	—

Abstract

In one embodiment, a method includes identifying unusual behavior with respect to a handshake between a first endpoint and a second endpoint that are included in a network, and determining whether the unusual behavior with respect to the handshake indicates presence of malicious software. The method also includes identifying at least one of the first endpoint and the second endpoint as potentially being infected by the malicious software if it is determined that the unusual behavior with respect to the handshake indicates the presence of malicious software.

First claim

1 . A method comprising:
identifying unusual behavior with respect to a handshake between a first endpoint and a second endpoint, wherein the first endpoint and the second endpoint are included in a network, wherein the unusual behavior is identified by an observer node included in the network, the observer node being inline on the network between the first endpoint and the second endpoint, the observer node being arranged to obtain communications between the first endpoint and the second endpoint; determining whether the unusual behavior with respect to the handshake indicates presence of malicious software; and identifying at least one of the first endpoint and the second endpoint as potentially being infected by the malicious software if it is determined that the unusual behavior with respect to the handshake indicates the presence of the malicious software.

2 . The method of claim 1 wherein the unusual behavior with respect to the handshake is an abandonment of the handshake.

3 . The method of claim 1 wherein the handshake is one selected from a group including a Transport Layer Security (TLS) handshake, a Secure Sockets Layer (SSL) handshake, and a Datagram Transport Layer Security {DTL} protocol handshake.

4 . The method of claim 3 wherein determining whether the unusual behavior with respect to the handshake indicates the presence of the malicious software includes using at least one selected from a group including telemetry data and historical data associated with the network to determine a likelihood that the unusual behavior with respect to the handshake indicates the presence of the malicious software.

5 . The method of claim 1 further including:
observing the network, wherein identifying the unusual behavior with respect to the handshake between the first endpoint and the second endpoint includes determining when an attempt has been made to proxy a connection between the first endpoint and the second endpoint and determining when the attempt made to the proxy the connection is unsuccessful.

6 . The method of claim 1 wherein identifying the at least one of the first endpoint and the second endpoint as potentially being infected by the malicious software includes identifying the first endpoint as potentially being infected by the malicious software and quarantining the first endpoint.

7 . A tangible, non-transitory computer-readable medium comprising computer program code, the computer program code, when executed, configured to:
identify unusual behavior with respect to a handshake between a first endpoint and a second endpoint, wherein the first endpoint and the second endpoint are included in a network, wherein the unusual behavior is identified by an observer node included in the network, the observer node being inline on the network between the first endpoint and the second endpoint, the observer node being arranged to obtain communications between the first endpoint and the second endpoint; determine whether the unusual behavior with respect to the handshake indicates presence of malicious software; and identify at least one of the first endpoint and the second endpoint as potentially being infected by the malicious software if it is determined that the unusual behavior with respect to the handshake indicates the presence of the malicious software.

8 . The tangible, non-transitory computer-readable medium of claim 7 wherein the unusual behavior with respect to the handshake is an abandonment of the handshake.

9 . The tangible, non-transitory computer-readable medium of claim 7 wherein the handshake is one selected from a group including a Transport Layer Security (TLS) handshake, a Secure Sockets Layer (SSL) handshake, and a Datagram Transport Layer Security {DTL} protocol handshake.

10 . The tangible, non-transitory computer-readable medium of claim 7 wherein the computer program code configured to determine whether the unusual behavior with respect to the handshake indicates the presence of the malicious software includes computer program code configured to use at least one selected from a group including telemetry data and historical data associated with the network to determine a likelihood that the unusual behavior with respect to the handshake indicates the presence of the malicious software.

11 . The tangible, non-transitory computer-readable medium of claim 7 wherein the computer program code is further configured to observe the network, wherein the computer program code configured to identify the unusual behavior with respect to the handshake between the first endpoint and the second endpoint is configured to determine when an attempt has been made to proxy a connection between the first endpoint and the second endpoint and to determine when the attempt made to the proxy the connection is unsuccessful.

12 . An apparatus comprising:
logic, the logic including a monitoring module, a detection module, and an identification module, the monitoring module being configured to monitor communications on a network by intercepting the communications between endpoints in the network, the communications on the network including handshake communications, wherein the detection module is configured to detect when the handshake communications include an unusual handshake communication, and wherein the identification module is arranged to determine when the unusual handshake communication indicates that at least one endpoint is compromised by malicious software; and a processing arrangement, wherein the logic includes computer program code and wherein the processing arrangement is configured to execute the computer program code.

13 . The apparatus of claim 12 wherein the monitoring module is configured to monitor behavior of the at least one endpoint when an interception proxy is inserted.

14 . The apparatus of claim 13 wherein the unusual handshake communication indicates sensitivity to the interception proxy.

15 . The apparatus of claim 12 wherein the handshake communications include one selected from a group including a Transport Layer Security (TLS) handshake, a Secure Sockets Layer (SSL) handshake, and a Datagram Transport Layer Security {DTL} protocol handshake.

16 . The apparatus of claim 15 wherein when the handshake communications include the TLS handshake, the unusual handshake communication is an abandoned TLS handshake, and wherein when the handshake communications include the SSL handshake, the unusual handshake communication is an abandoned SSL handshake.

17 . The apparatus of claim 12 further including:
a data storage arrangement, the data storage arrangement being configured to store least one selected from a group including historical information associated with the network and telemetry information associated with the network.

18 . The apparatus of claim 17 wherein the identification module is configured to use the at least one selected from the group including the historical information associated with the network and the telemetry information associated with the network to determine when the unusual handshake communication indicates that the at least one endpoint is compromised by the malicious software.

19 . The apparatus of claim 18 wherein the identification module is configured to cause the at least one endpoint to be quarantined when it is determined that the unusual handshake communication indicates that the at least one endpoint is compromised by the malicious software.

20 . The apparatus of claim 18 wherein the identification model is configured t

Assignees

• Wing Daniel G
• Andreasen Flemming S
• Leung Kent K
• Cisco Tech Inc

Inventors

• Wing Daniel G
• Andreasen Flemming S
• Leung Kent K

Classifications

• H04L63/145

  the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms · CPC title

• H04L63/1408

  by monitoring network traffic (monitoring network traffic per se H04L43/00) · CPC title

• H04L63/166 Primary

  at the transport layer · CPC title

• H04L63/1416 Primary

  Event detection, e.g. attack signature detection · CPC title