Systems and methods for detecting code duplication in codebases

Patent metadata

Field	Value
Publication number	US-2023185550-A1
Application number	US-202218064620-A
Country	US
Kind code	A1
Filing date	Dec 12, 2022
Priority date	Dec 13, 2021
Publication date	Jun 15, 2023
Grant date	—

Abstract

Systems and methods for detecting code duplication are disclosed. In one embodiment, a method for detecting exact code snippet duplicates may include: (1) representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as an abstract syntax trees; (2) featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax tree into vector representations; (3) generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique; (4) identifying, by the code duplication detection computer program, exact duplicate code snippet matches by apply density-based clustering to the dense feature vectors; and (5) tagging, by the code duplication detection computer program, the exact duplicate code snippets.

First claim

What is claimed is:

1 . A method for detecting exact code snippet duplicates, comprising:
representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as an abstract syntax trees; featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax tree into vector representations; generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique; identifying, by the code duplication detection computer program, exact duplicate code snippet matches by apply density-based clustering to the dense feature vectors; and tagging, by the code duplication detection computer program, the exact duplicate code snippets.

2 . The method of claim 1 , further comprising:
applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

3 . The method of claim 1 , further comprising:
applying, by the code duplication detection computer program, a de-noising filter to the plurality of code snippets or the abstract syntax trees.

4 . The method of claim 3 , wherein the de-noising filter filters code snippets or abstract syntax trees that are not actively used.

5 . The method of claim 3 , wherein the de-noising filter filters code snippets or abstract syntax trees that are irrelevant.

6 . The method of claim 3 , wherein the de-noising filter is based on a trained neural network.

7 . The method of claim 1 , wherein the corpus feature vectors comprise a list of featurized abstract syntax trees from a code corpus.

8 . The method of claim 1 , wherein the dimension reduction technique comprises truncated Singular Value Decomposition.

9 . A method for detecting near code snippet duplicates, comprising:
representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as an abstract syntax trees; featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax tree into vector representations; generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique; clustering, by the code duplication detection computer program, the dense feature vectors into dendrograms, each dendrogram having a different value for a cluster distance metric; applying, by the code duplication detection computer program, cross-correlation thresholding to identify an optimal value for the cluster distance metric; applying, by the code duplication detection computer program, iterative density-based clustering to the dendrogram for the optimal value for the cluster distance metric; tracking, by the code duplication detection computer program, data points in the dendrogram that have merged into a large cluster but were also present in small unique clusters, wherein the data points belonging to the same unique small cluster identify code snippets that are near duplicates of each other; and tagging, by the code duplication detection computer program, the near duplicate code snippets.

10 . The method of claim 8 , further comprising:
applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

11 . The method of claim 9 , further comprising:
applying, by the code duplication detection computer program, a de-noising filter to the plurality of code snippets or the abstract syntax trees.

12 . The method of claim 11 , wherein the de-noising filter filters code snippets or abstract syntax trees that are not actively used.

13 . The method of claim 11 , wherein the de-noising filter is based on a trained neural network.

14 . The method of claim 9 , wherein the corpus feature vectors comprise a list of featurized abstract syntax trees from a code corpus.

15 . The method of claim 9 , wherein the dimension reduction technique comprises truncated Singular Value Decomposition.

16 . A method for detecting exact code snippet duplicates, comprising:
loading, by a code duplication detection computer program, a near duplicate centroid, an exact duplicate centroid, a vectorizer, and a dimension reduction model; producing, by the code duplication detection computer program, dense vectors from incremental functions; representing, by a code duplication detection computer program, each of a plurality of incremental functions as an abstract syntax trees; featurizing, by the code duplication detection computer program, the abstract syntax trees into incremental function feature vectors by converting the abstract syntax tree into vector representations; generating, by the code duplication detection computer program, dense feature vectors from the incremental function feature vectors using the dimension reduction model; computing, by the code duplication detection computer program, a cosine similarity between the generated dense vectors and the near duplicate centroid and the exact duplicate centroid; ranking, by the code duplication detection computer program, the near duplicate centroid and the exact duplicate centroid in descending order based on the similarity; thresholding, by the code duplication detection computer program, the ranked near duplicate centroid and the exact duplicate centroid; selecting, by the code duplication detection computer program, a top most ranked centroid; and identifying, by the code duplication detection computer program, the incremental function as a duplicate of data points in a cluster of the top-most centroid.

17 . The method of claim 16 , wherein the dimension reduction model comprises truncated Singular Value Decomposition.

18 . The method of claim 16 , further comprising:
applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

19 . The method of claim 16 , further comprising:
applying, by the code duplication detection computer program, a de-noising filter to the plurality of code snippets or the abstract syntax trees.

20 . The method of claim 19 , wherein the de-noising filter filters code snippets or abstract syntax trees that are irrelevant.

Assignees

• Jpmorgan Chase Bank Na

Inventors

• Saphal Rohan
• Silavong Fanny
• Moran Sean
• Georgiadis Antonios
• Saha Sanat
• Singh Gaurav
• Osselin Pierre
• Otter Rob

Classifications

• G06F8/4435 Primary

  Detection or removal of dead or redundant code · CPC title

• G06F8/427

  Parsing · CPC title