AR translation processing method and electronic device

Patent metadata

Field	Value
Publication number	US-12518113-B2
Application number	US-202218278080-A
Country	US
Kind code	B2
Filing date	Aug 25, 2022
Priority date	Sep 8, 2021
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

Provided are an AR translation processing method and an electronic device, which relate to the technical field of communications. By the method, in a scenario in which an electronic device is used for AR translation, a pose change of the electronic device can be detected in real time, and feature matching can be performed on a plurality of consecutive frames of images acquired by a camera, so that whether to-be-translated text needs to be fully translated or partially translated, or needs not to be translated can be determined based on the pose change of the electronic device and a feature matching result, and therefore a corresponding translation trigger strategy is selected. In this way, repeated translation can be effectively avoided, thereby saving computing resources in the AR translation process and improving the translation efficiency to a particular extent.

First claim

What is claimed is:

1 . An augmented reality (AR) translation processing method, comprising:
acquiring, by a camera of an electronic device, a first image, wherein the first image comprises first to-be-translated text; translating the first to-be-translated text, to obtain a first translation result; displaying the first image and displaying a first virtual image on the first image in a superimposed manner, wherein the first virtual image comprises the first translation result; acquiring, by the camera, a second image, wherein the second image comprises second to-be-translated text; in a case that a pose change amount of the electronic device is less than a preset pose threshold and a feature similarity between the second image and the first image is greater than or equal to a preset similarity threshold, obtaining the first translation result as a translation result of the second to-be-translated text; in a case that a pose change amount of the electronic device is less than a preset pose threshold and a feature similarity is less than a preset similarity threshold, translating the second to-be-translated text; in a case that a pose change amount of the electronic device is greater than or equal to a preset pose threshold and a feature similarity is less than a preset similarity threshold, translating part or all of text in the second to-be-translated text; and displaying the second image and displaying a second virtual image on the second image in a superimposed manner, wherein the second virtual image comprises the translation result of the second to-be-translated text.

2 . The method according to claim 1 , the translating part or all of text in the second to-be-translated text comprises:
in a case that a first part of text in the second to-be-translated text is the same as the first to-be-translated text, obtaining the first translation result as a translation result of the first part of text; and translating a second part of text, to obtain a translation result of the second part of text, wherein the second part of text is text other than the first part of text in the second to-be-translated text; and the translation result of the second to-be-translated text comprises the translation result of the first part of text and the translation result of the second part of text.

3 . The method according to claim 1 , the translating part or all of text in the second to-be-translated text comprises:
in a case that the second to-be-translated text and the first to-be-translated text do not have same text, translating all of text in the second to-be-translated text, to obtain the translation result of the second to-be-translated text.

4 . The method according to claim 1 , further comprising:
extracting feature points in the first image and feature points in the second image; and comparing the feature points in the second image with the feature points in the first image, to obtain the feature similarity between the second image and the first image.

5 . The method according to claim 1 , further comprising:
generating an SLAM map by using a simultaneous localization and mapping SLAM method; and based on the second image, measurement data of a target sensor in the electronic device, and the SLAM map, determining a pose change amount of the electronic device, wherein the pose change amount comprises a position change amount and a posture change amount; and the target sensor comprises an inertial measurement unit IMU.

6 . The method according to claim 1 , further comprising:
determining, by using an SLAM method, a target virtual plane for displaying AR digital content, wherein the target virtual plane is located above a plane where the acquired image is located; and the displaying a second virtual image on the second image in a superimposed manner comprises: displaying the second virtual image on the target virtual plane.

7 . The method according to claim 6 , further comprising:
determining a target projection region of the second to-be-translated text on the target virtual plane, wherein the displaying the second virtual image on the target virtual plane comprises: displaying the second virtual image in the target projection region.

8 . The method according to claim 7 , the determining a target projection region of the second to-be-translated text on the target virtual plane comprises:
determining a first rectangular region occupied by the second to-be-translated text in the second image; based on two endpoints on a diagonal of the first rectangular region, determining two anchor points of the two endpoints mapped on the target virtual plane; and determining a second rectangular region on the target virtual plane with a connecting line of the two anchor points as a diagonal, wherein the second rectangular region is a target projection region of the second to-be-translated text on the target virtual plane, and a transparency of the target projection region is less than or equal to a preset transparency threshold.

9 . The method according to claim 1 , further comprising:
performing text recognition on the first image, to obtain the first to-be-translated text; performing text recognition on the second image, to obtain the second to-be-translated text; rendering the translation result of the first to-be-translated text, to obtain the first virtual image; and rendering the translation result of the second to-be-translated text, to obtain the second virtual image.

10 . An electronic device, comprising a display screen, a camera, and a processor, wherein the processor is coupled to a memory and configured to execute a computer program or instructions stored in the memory, to cause the electronic device to implement the following steps:
acquiring, by a camera of the electronic device, a first image, wherein the first image comprises first to-be-translated text; translating the first to-be-translated text, to obtain a first translation result; displaying the first image and displaying a first virtual image on the first image in a superimposed manner, wherein the first virtual image comprises the first translation result; acquiring, by the camera, a second image, wherein the second image comprises second to-be-translated text; in a case that a pose change amount of the electronic device is less than a preset pose threshold and a feature similarity between the second image and the first image is greater than or equal to a preset similarity threshold, obtaining the first translation result as a translation result of the second to-be-translated text; in a case that a pose change amount of the electronic device is less than a preset pose threshold and a feature similarity is less than a preset similarity threshold, translating the second to-be-translated text; in a case that a pose change amount of the electronic device is greater than or equal to a preset pose threshold and a feature similarity is less than a preset similarity threshold, translating part or all of text in the second to-be-translated text; and displaying the second image and displaying a second virtual image on the second image in a superimposed manner, wherein the second virtual image comprises the translation result of the second to-be-translated text.

11 . The electronic device according to claim 10 , the translating part or all of text in the second to-be-translated text comprises:
in a case that a first part of text in the second to-be-translated text is the same as the first to-be-translated text, obtaining the first translation result as a translation result of the first part of text; and translating a second part of text, to obtain a translation result of the second pa

Assignees

• Honor Device Co Ltd

Inventors

• Liu Xiaowei
• Chen Bing
• Wang Guoyi
• Zhou Junwei

Classifications

• G06T7/33

  using feature-based methods · CPC title

• G06V10/22

  by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition · CPC title

• G06V30/10

  Character recognition · CPC title

• G06V10/17

  using hand-held instruments · CPC title

• G06V20/63

  Scene text, e.g. street names · CPC title