Method and apparatus for artificial intelligence downscaling and upscaling during video conference

Patent metadata

Field	Value
Publication number	US-12355992-B2
Application number	US-202217895561-A
Country	US
Kind code	B2
Filing date	Aug 25, 2022
Priority date	Aug 25, 2021
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

Provided is an electronic device configured to participate in video conference by using artificial intelligence (AI), the electronic device including a display and a processor configured to execute one or more instructions. The processor configured to obtain, from a server, image data generated by first encoding first image related to another electronic device participating in video conference, and AI data related to AI downscaling from original image to first image, obtain second image corresponding to first image by performing first decoding on image data, determine whether to perform AI upscaling on second image, based on importance of the other electronic device, based on determining to perform AI upscaling, obtain third image by performing AI upscaling on second image through an upscaling deep neural network and provide third image to display, and based on determining not to perform AI upscaling, provide second image to display.

First claim

The invention claimed is:

1. An electronic device configured to be in a video conference by using artificial intelligence (AI), the electronic device comprising:
a display; and
a processor configured to execute one or more instructions stored in the electronic device,
wherein the processor is configured to execute the one or more instructions to:
obtain, from a server, first image data generated as a result of first encoding a first image related to a first electronic device in the video conference, and first AI data related to AI downscaling from a first original image to the first image;
obtain, from the server, second image data generated as a result of the first encoding a second image related to a second electronic device in the video conference, and second AI data related to AI downscaling from a second original image to the second image;
obtain a third image corresponding to the first image by performing first decoding on the first image data, and fourth image corresponding to the second image by performing the first decoding on the second image data;
based on an importance of the first electronic device indicating that a user of the first electronic device is a presenter, obtain a fifth image by performing AI upscaling on the third image through an upscaling deep neural network (DNN), and provide the fifth image to the display; and based on an importance of the second electronic device indicating that a user of the second electronic device is a listener, obtain a sixth image by performing AI downscaling on the fourth image through an downscaling deep neural network (DNN), and provide the sixth image to the display.

2. The electronic device of claim 1 , wherein the importance of the first electronic device is checked from the first AI data, and the importance of the second electronic device is checked from the second AI data.

3. The electronic device of claim 1 , wherein the first image is an image to which the server AI-downscales the first original image or an image to which the first electronic device AI-downscales the first original image.

4. The electronic device of claim 1 , wherein
based on the importance of the first electronic device being changed, during the video conference, to indicate that the user of the first electronic device is the listener, it is determined that the AI upscaling is not performed.

5. The electronic device of claim 1 , wherein importance of the electronic device that establishes the video conference is initially set as the presenter.

6. A video conference image processing method performed by an electronic device in a video conference by using artificial intelligence (AI), the video conference image processing method comprising:
obtaining, from a server, first image data generated as a result of first encoding a first image related to a first electronic device in the video conference, and first AI data related to AI downscaling from a first original image to the first image;
obtaining, from the server, second image data generated as a result of the first encoding a second image related to a second electronic device in the video conference, and second AI data related to AI downscaling from a second original image to the second image;
obtaining a third image corresponding to the first image by performing first decoding on the first image data, and fourth image corresponding to the second image by performing the first decoding on the second image data;
based on an importance of the first electronic device indicating that a user of the first electronic device is a presenter, obtaining a fifth image by performing AI upscaling on the third image through an upscaling deep neural network (DNN), and provide the fifth image to a display; and
based on an importance of the second electronic device indicating that a user of the second electronic device is a listener, obtaining a sixth image by performing AI downscaling on the fourth image through an downscaling deep neural network (DNN), and provide the sixth image to the display.

7. A server managing a video conference by using artificial intelligence (AI), the server comprising a processor configured to execute one or more instructions stored in the server,
wherein the processor is configured to execute the one or more instructions to:
obtain, from a first electronic device in the video conference, first image data generated as a result of first encoding a first image, and first AI data related to the AI downscaling from a first original image to the first image;
obtain, from a second electronic device in the video conference, second image data generated as a result of the first encoding a second image, and second AI data related to AI downscaling from a second original image to the second image
obtain a third image corresponding to the first image by performing first decoding on the first image data, and fourth image corresponding to the second image by performing the first decoding on the second image data;
based on an importance of the first electronic device indicating that a user of the first electronic device is a presenter, obtain a fifth image by performing AI upscaling on the third image through an upscaling deep neural network (DNN), and transmit the fifth image to a third electronic device; and
based on an importance of the second electronic device indicating that a user of the second electronic device is a listener, obtain a sixth image by performing AI downscaling on the fourth image through a downscaling deep neural network (DNN), and transmit the sixth image to the third electronic device.

8. The server of claim 7 , wherein the importance of the first electronic device is checked from the first AI data related to the AI downscaling.

9. The server of claim 7 , wherein the first electronic device is configured to support AI downscaling.

10. The server of claim 7 , wherein the second electronic device is configured not to support AI upscaling.

11. The server of claim 7 , wherein the processor is further configured to execute the one or more instructions to:
obtain third image data by performing first encoding on a third original image from a fourth electronic device;
obtain the third original image by performing first decoding on the third image data;
based on importance of the fourth electronic device indicating that a user of the fourth electronic device is the listener, obtain the first image by performing AI downscaling on the first original image by using the downscaling DNN, and transmit fourth image data obtained by performing first encoding on the first image to the third electronic device; and
based on the importance of the fourth electronic device indicating that the user of the first electronic device is the presenter, transmit the third image data to the third electronic device.

12. The server of claim 11 , wherein the fourth electronic device is configured not to support AI downscaling.

13. The server of claim 11 , wherein, based on the third electronic device being configured to support AI upscaling, the processor is further configured to obtain the first image by performing AI downscaling on the first original image by using the downscaling DNN, and transmit, to the third electronic device, the third image data obtained by performing first encoding on the first image and the first AI data related to the AI downscaling.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Chaeeun
• Kim Jaehwan
• Park Youngo
• Lee Jongseok
• CHOI Kwangpyo

Classifications

• H04N19/436

  using parallelised computational arrangements · CPC title

• H04N19/136

  Incoming video signal characteristics or properties · CPC title

• H04N19/164

  Feedback from the receiver or from the transmission channel · CPC title

• G06N3/08

  Learning methods · CPC title

• H04N21/4402

  involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} · CPC title