On-device artificial intelligence processing in-browser

Patent metadata

Field	Value
Publication number	US-12561389-B2
Application number	US-202217807880-A
Country	US
Kind code	B2
Filing date	Jun 21, 2022
Priority date	Jun 21, 2022
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

Examples of the present disclosure describe systems and methods for on-device, in-browser AI processing. In examples, a selection of an AI pipeline is received. Content associated with the AI pipeline is also received. The content is segmented into multiple data segments and a set of data features is generated for the data segments. AI modules associated with the AI pipeline are loaded to create the AI pipeline. The set of data features is provided to the AI pipeline. The AI pipeline is executed to generate insights for the set of data features. The insights are then provided to a user.

First claim

What is claimed is:

1 . A system comprising:
a processor system comprising a processor; and memory coupled to the processor, the memory comprising computer executable instructions that, when executed by the processor, perform operations comprising:
receiving content by a web browser of a client device, the web browser comprising an in-browser architecture that includes an execution environment for executing an artificial intelligence (AI) pipeline, wherein the content includes at least one of audio content or video content;
extracting, by the in-browser architecture, a set of data features from the content;
retrieving, by the in-browser architecture, a set of AI modules;
creating the AI pipeline in the execution environment using the set of AI modules;
providing the set of data features to the AI pipeline;
executing the AI pipeline in the execution environment to process the content and create a set of insights; and
providing, by the in-browser architecture, the set of insights to an interface of the web browser during receipt or playback of the content.

2 . The system of claim 1 , wherein extracting the set of data features from the content comprises:
decoding each data stream in the content; and extracting the set of data features.

3 . The system of claim 1 , wherein creating the AI pipeline using the set of AI modules comprises:
retrieving the set of AI modules from an AI module repository comprising a plurality of AI modules, the AI module repository being external to the web browser, the set of AI modules representing a subset of the plurality of AI modules.

4 . The system of claim 1 , wherein selecting the set of AI modules comprises:
receiving a manual selection of the set of AI modules via the web browser.

5 . The system of claim 1 , wherein selecting the set of AI modules comprises:
evaluating, by the web browser, attributes of the content; and based on the attributes of the content, selecting, by the web browser, the set of AI modules.

6 . The system of claim 1 , wherein creating the AI pipeline comprises:
loading the set of AI modules in the execution environment; and arranging the set of AI modules according to a configuration order, the configuration order defining dependencies for the set of AI modules.

7 . The system of claim 1 , wherein each AI module in the set of AI modules:
represents a step in the AI pipeline; and comprises software code or instructions for performing the step.

8 . The system of claim 1 , wherein a particular AI module in the set of AI modules performs at least one of:
topic inferencing; object detection; or speech classification.

9 . The system of claim 1 , wherein, in addition to the set of AI modules, the AI pipeline further comprises at least one operator, preprocessing step, or postprocessing step.

10 . The system of claim 1 , wherein executing the AI pipeline causes a particular AI module in the set of AI modules to generate an output comprising inference data for the set of data features.

11 . The system of claim 10 , wherein the inference data comprises at least one of:
a timestamp for a video frame; a bounding box; or a confidence value for a detected object or event.

12 . A method comprising:
extracting, by a web browser implementing an in-browser architecture, a set of data features from content, wherein:
the web browser is implemented by a user device;
the in-browser architecture comprises an execution environment for executing an artificial intelligence (AI) pipeline; and
the content is a video file comprising video data and audio data;
retrieving, by the in-browser architecture, a set of AI modules; creating the AI pipeline in the execution environment using the set of AI modules; providing the set of data features to the AI pipeline; creating a set of insights by executing the AI pipeline in the execution environment; and providing, by the in-browser architecture, the set of insights to the web browser.

13 . The method of claim 12 , wherein the user device is a mobile device.

14 . The method of claim 12 , wherein the user device retrieves the set of AI modules from a server environment or a cloud-based environment separate from the user device.

15 . The method of claim 12 , wherein creating the AI pipeline and creating the set of insights is performed on-device of the user device.

16 . The method of claim 12 , the method further comprising:
prior to extracting the set of data features from the content:
receiving the content at the web browser, wherein the content is provided by a data source external to the user device.

17 . The method of claim 16 , the method further comprising:
upon receiving the content at the web browser, receiving an insight objective for the content, wherein the insight objective indicates a type of analysis to be performed by the AI pipeline and one or more types of insights to be provided for the content.

18 . The method of claim 12 , wherein providing the set of insights to the web browser comprises causing the set of insights to be displayed in in interface of the user device in real-time as the content is displayed in the interface.

19 . A user device comprising:
a processor; and memory comprising computer executable instructions that, when executed, perform operations comprising:
receiving content by a web browser of the user device, the web browser comprising an in-browser architecture that includes an execution environment for executing an artificial intelligence (AI) pipeline on-device of the user device, the content being a video file comprising video data and audio data;
extracting, by the in-browser architecture, a set of data features from the content;
retrieving, by the in-browser architecture, a set of AI modules based on the set of data features;
creating the AI pipeline in the execution environment by:
retrieving the set of AI modules from a data source outside of the execution environment; and
loading the set of AI modules into the execution environment;

providing the set of data features to the AI pipeline;
creating a set of insights by executing the AI pipeline in the execution environment; and
providing, by the in-browser architecture, the set of insights to an interface of the user device such that one or more insights in the set of insights is provided in the interface while at least a portion of the content is provided in the interface.

20 . The user device of claim 19 , wherein the set of AI modules is retrieved from a server environment or a cloud-based environment separate from the user device.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• ZIV Ori
• Kinarti Barak
• Bakhar Ben
• Figov Zvi
• Van Neerden Fardau
• Jassin Ohad
• Neeman Avi

Classifications

• G06F16/2456

  Join operations · CPC title

• G06F16/2322

  using timestamps · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F16/957

  Browsing optimisation, e.g. caching or content distillation · CPC title

• G06F16/954 Primary

  Navigation, e.g. using categorised browsing · CPC title