Multi-lingual line-of-code completion system

Patent metadata

Field	Value
Publication number	US-2024028306-A1
Application number	US-202318232326-A
Country	US
Kind code	A1
Filing date	Aug 9, 2023
Priority date	Aug 1, 2019
Publication date	Jan 25, 2024
Grant date	—

Abstract

A code completion tool uses a neural transformer model to generate candidate sequences to complete a line of source code. The neural transformer model is trained using a conditional language modeling objective on a large unsupervised dataset that includes source code programs written in several different programming languages. The neural transformer model is used within a beam search that predicts the most likely candidate sequences for a code snippet under development.

First claim

1 . (canceled)

2 . A system comprising:
a processor; and a memory that stores a program that is configured to be executed by the processor, wherein the program comprises instructions to perform actions that: continuously track characters entered into a source code program during an editing process; and at a position in a line of the source code program, detect a partially-formed line of source code; search for at least one candidate sequence to complete the line of source code, wherein the at least one candidate sequence comprises a sequence of source code tokens, wherein the search generates the at least one candidate sequence to complete the line of source code based on a conditional probability generated by a neural transformer model with attention at each time step of the search, wherein the conditional probability indicates a likelihood of a next source code token to follow preceding source code tokens in a candidate sequence, wherein the neural transformer model with attention is given a context of the line of source code, wherein the neural transformer model with attention is trained to learn the syntax and relationships between code elements of different programming languages; and output the at least one candidate sequence for user selection in a user interface of the editing process.

3 . The system of claim 2 , wherein the program includes further instructions that when executed by the processor perform actions that:
detect the partially-formed line of source code based on a current cursor position.

4 . The system of claim 3 , wherein the search performs a beam search to generate the at least one candidate sequence.

5 . The system of claim 2 , wherein the program includes further instructions that when executed by the processor performs actions that:
transform the context of the line of source code into a sequence of token embeddings; and input the sequence of token embeddings to the neural transformer model with attention.

6 . The system of claim 5 , wherein the program includes further instructions that when executed by the processor performs actions that:
output by the neural transformer model with attention at each time step a plurality of hidden state vectors; and generate predicted embedding vectors as a product of the plurality of hidden state vectors with a linear projection layer.

7 . The system of claim 6 , wherein the program includes further instructions that when executed by the processor performs actions that:
generate a plurality of logits as a product of the predicted embedding vectors and the token embeddings.

8 . The system of claim 7 , wherein the program includes further instructions that when executed by the processor performs actions that:
normalize the plurality of logits into a probability distribution.

9 . The system of claim 2 , wherein the search is a beam search and wherein the at least one candidate sequence comprises top-k candidate sequences.

10 . A computer-implemented method, comprising:
monitoring each token input into a source code program in a source code development session; iteratively executing a beam search to generate at least one candidate to complete a partially-formed line of source code in the source code program, wherein the at least one candidate comprises a plurality of tokens, wherein the beam search generates a token, at each time step, to include a new token in the at least one candidate using token probabilities generated from a neural transformer model with attention, wherein the neural transformer model with attention is given a context of the partially-formed line of source code, wherein the neural transformer model with attention is trained on a plurality of multi-lingual source code programs; and outputting the at least one candidate upon an end-of-line token predicted as having a highest probability to input into the at least one candidate.

11 . The computer-implemented method of claim 10 , comprising:
detecting the partially-formed line of source code based on a position of a cursor in the source code development session.

12 . The computer-implemented method of claim 11 ,
wherein the context comprises a sequence of token embeddings and a sequence of positional embeddings.

13 . The computer-implemented method of claim 12 , comprising:
outputting by the neural transformer model with attention at each time step a hidden state vector; and generating predicted embedding vectors as a product of the hidden state vector and a linear projection layer.

14 . The computer-implemented method of claim 13 , comprising:
generating unnormalized logits as a product of the predicted embedding vectors and the sequence of token embeddings.

15 . The computer-implemented method of claim 14 , comprising:
normalizing the logits using a softmax function to generate the token probabilities.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Svyatkovskiy Alexey
• Fu Shengyu
• Sundaresan Neelakantan
• Deng Shao Kun

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F8/33 Primary

  Intelligent editors · CPC title

• G06F16/9027

  Trees · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title