Multi-lingual line-of-code completion system

Patent metadata

Field	Value
Publication number	US-11262984-B2
Application number	US-201916680328-A
Country	US
Kind code	B2
Filing date	Nov 11, 2019
Priority date	Aug 1, 2019
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

A code completion tool uses a neural transformer model to generate candidate sequences to complete a line of source code. The neural transformer model is trained using a conditional language modeling objective on a large unsupervised dataset that includes source code programs written in several different programming languages. The neural transformer model is used within a beam search that predicts the most likely candidate sequences for a code snippet under development.

First claim

What is claimed:

1. A system comprising:
one or more processors; and
a memory that stores one or more programs that are configured to be executed by the one or more processors, the one or more programs including instructions that:
track a sequence of characters entered into a partially-formed line of a source code program during an editing session, wherein the source code program is written in a first programming language; and
at a position in the partially-formed line of the source code program, generate a candidate sequence to complete the partially-formed line of source code using a decoder-only neural transformer model with attention given the tracked sequence of characters, wherein the decoder-only neural transformer model with attention is trained on an unsupervised dataset of source code programs written in a plurality of different programming languages.

2. The system of claim 1 , wherein the one or more programs include further instructions that when executed by the one or more processors:
initiate a beam search to build a search tree to generate the candidate sequence, wherein the search tree includes one or more nodes at one or more inference levels, each node represents an output probability distribution for a set of tokens of a vocabulary of the decoder-only neural transformer model with attention, wherein the output probability distribution is generated from the decoder-only neural transformer model with attention, each node expands k tokens/subtokens to a next inference level.

3. The system of claim 2 , wherein the beam search iteratively expands the search tree by invoking the decoder-only neural transformer model with attention to predict a next token given a sequence of tokens representing a partial candidate to complete the partially-formed line-of-code.

4. The system of claim 1 , wherein the decoder-only neural transformer model with attention includes at least one decoder block having a masked self-attention layer and a convolutional neural network layer.

5. The system of claim 1 , wherein the decoder-only neural transformer model with attention includes at least one one-dimensional convolutional neural network layer.

6. The system of claim 1 , wherein track the sequence of characters entered into the partially-formed line of the source code program further comprises:
obtain a sequence of tokens/subtokens representing a current context of the partially-formed line of code; and
find token/subtoken embedding vectors and positional embedding vectors for the sequence of tokens/subtokens, wherein the token/subtoken embedding vectors and the positional embedding vectors are generated from training the decoder-only neural transformer model with attention.

7. The system of claim 6 , wherein the one or more programs include further instructions that when executed by the one or more processors:
input the token/subtoken embedding vectors and positional embedding vectors into the decoder-only neural transformer model with attention, wherein the decoder-only neural transformer model with attention generates a probability distribution for the tokens/subtokens of a model vocabulary.

8. The system of claim 1 , wherein the plurality of different programming languages does not include the first programming language.

9. A method, comprising:
monitoring each token input into a partially-formed line-of-code of a source code program during a source code development session;
iteratively executing a beam search to generate token candidates to complete the line-of-code as a new token is input into the partially-formed line-of-code, wherein the beam search generates a token candidate using token probabilities generated from a decoder-only neural transformer model with attention trained on a plurality of multi-lingual source code programs;
concatenating the token candidates into candidate sequences to complete the partially-formed line-of-code; and
outputting at least one candidate sequence upon detection of a marker character input in the partially-formed line-of-code during the source code development session.

10. The method of claim 9 , further comprising:
invoking the decoder-only neural transformer model to predict a next token given a context vector representing a context of the partially-formed line-of-code including the new token.

11. The method of claim 9 , wherein the decoder-only neural transformer model includes a self-attention layer and a convolutional neural network.

12. The method of claim 11 , wherein the self-attention layer is preceded by layer normalization and layer normalization is applied to the outputs of the self-attention layer.

13. The method of claim 9 , wherein the decoder-only neural transformer model with attention utilizes token embeddings and positional embeddings representing a context of the line-of-code, wherein the token embeddings and the positional embeddings are learned from training the decoder neural transformer model with attention.

14. The method of claim 9 , wherein monitoring each token input into the source code program further comprises:
parsing characters input into the line-of-code into a concrete syntax tree;
performing byte pair encoding to extract tokens from the concrete syntax tree; and
concatenating ordered sequences of tokens of length T.

15. A device, comprising:
at least one processor coupled to a memory device;
wherein the at least one processor is configured to:
extract one or more ordered sequences of tokens from a plurality of source code programs written in different programming languages, wherein an ordered sequence of tokens represents a context of a segment of source code from a select one of the plurality of source code programs; and
utilize the ordered sequences of tokens to train a decoder-only neural transformer model with attention to predict a next token to complete a partial sequence of tokens, wherein the partial sequence of tokens is used to produce a candidate sequence of tokens that complete a line-of-code in a target source code program, wherein the decoder-only neural transformer model with attention includes an attention layer and at least one convolutional neural network layer.

16. The method of claim 9 , wherein the source code program is written in a programming language that differs from programming languages of the plurality of multi-lingual source code programs.

17. The device of claim 15 , wherein the ordered sequence of tokens includes one or more subtokens.

18. The device of claim 15 , wherein the ordered sequences of tokens are an unsupervised training dataset.

19. The device of claim 15 , wherein the decoder-only neural transformer model with attention generates a matrix of token probabilities that are used to predict a next token to succeed in a predicted candidate sequence.

20. The device of claim 15 , wherein the target source code program is written in a programming language that differs from the different programming languages.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Svyatkovskiy Alexey
• Fu Shengyu
• Sundaresan Neelakantan
• Deng Shao Kun

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title