Electronic apparatus and method for identifying content

Patent metadata

Field	Value
Publication number	US-12548322-B2
Application number	US-202418607842-A
Country	US
Kind code	B2
Filing date	Mar 18, 2024
Priority date	Oct 18, 2021
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

An electronic device includes a camera, a communication circuit, memory storing one or more computer programs, and at least one processor. The one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to obtain information on a second image comprising a plurality of visual objects and changed from a first image, transmit, to a server, the information on the second image, receive information on a three-dimensional (3D) model for a space comprising the plurality of external objects and information on a reference image, obtain a third image by removing, from the second image, at least one visual object among the plurality of visual objects, identify at least one feature point based on a comparison between the reference image and the third image, identify a pose of a virtual camera, and identify content superimposed on the first image.

First claim

What is claimed is:

1 . An electronic device comprising:
a camera; a communication circuit; one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the electronic device to:
obtain information on a second image comprising a plurality of visual objects corresponding to a plurality of external objects respectively and changed from a first image obtained using the camera,
transmit, to a server, the information on the second image,
receive information on a three-dimensional (3D) model for a space comprising the plurality of external objects and information on a reference image identified based on the information on the second image transmitted to the server,
in response to receiving the information on the reference image, obtain a third image by removing, from the second image, at least one visual object comprised in the second image of the second image and the reference image, among the plurality of visual objects,
identify at least one feature point among a plurality of feature points identified based on a comparison between the reference image and the third image,
identify, based on the at least one feature point, a pose of a virtual camera, and
identify a content superimposed on the first image by processing 3D model based on the pose of the virtual camera.

2 . The electronic device of claim 1 , further comprising:
a global positioning system (GPS) receiver, wherein the instructions, when executed by the one or more processors, cause the electronic device to:
identify a location of the electronic device using the GPS receiver, and
transmit, to the server, information on the location of the electronic device with the information on the second image, and
wherein the reference image is identified based on the information on the location of the electronic device and the information on the second image.

3 . The electronic device of claim 1 ,
wherein the second image is obtained by being changed from the first image based on semantic segmentation, wherein the second image is divided to an identified area and an unidentified area, wherein the unidentified area comprises the at least one visual object, and wherein the information on the second image comprises a binary code with respect to the identified area.

4 . The electronic device of claim 1 , wherein the pose of the virtual camera comprises a value of roll, a value of pitch, and a value of yaw of the virtual camera.

5 . The electronic device of claim 1 , wherein the instructions, when executed by the one or more processors, cause the electronic device to:
identify designated number of feature points identified based on distance information for each of the identified plurality of feature points, among the identified plurality of feature points as the at least one feature point.

6 . The electronic device of claim 5 , wherein the instructions, when executed by the one or more processors, cause the electronic device to identify, based on the at least one feature point, a first rotation error with respect to a roll of the virtual camera, a second rotation error with respect to a pitch of the virtual camera, and a third rotation error with respect to a yaw of the virtual camera.

7 . The electronic device of claim 6 , wherein the instructions, when executed by the one or more processors, cause the electronic device to:
identify, based on the at least one feature point, a homography matrix, and identify, based on the homography matrix and unique information on the camera, the first rotation error, the second rotation error, and the third rotation error.

8 . The electronic device of claim 7 , further comprising:
at least one sensor, wherein the instructions, when executed by the one or more processors, cause the electronic device to identify a pose of the electronic device based on first data related to the pose of the electronic device received from the server and second data related to the pose of the electronic device obtained through the at least one sensor.

9 . The electronic device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the electronic device to:
identify an average rotation error of the first rotation error, the second rotation error, and the third rotation error, and in response to identifying that the average rotation error is less than a designated value, identify the pose of the virtual camera with a first pose identified based on the pose of the electronic device.

10 . The electronic device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the electronic device to:
identify an average rotation error of the first rotation error, the second rotation error, and the third rotation error, in response to identifying that the average rotation error is greater than or equal to a designated value, calibrate a first pose identified based on the pose of the electronic device to a second pose using the first rotation error, the second rotation error, and the third rotation error, and identify the pose of the virtual camera with the second pose.

11 . A method of an electronic device, the method comprising:
obtaining information on a second image comprising a plurality of visual objects corresponding to a plurality of external objects respectively and changed from a first image obtained using a camera; transmitting, to a server, the information on the second image; receiving information on a three-dimensional (3D) model for a space comprising the plurality of external objects and information on a reference image identified based on the information on the second image transmitted to the server; in response to receiving the information on the reference image, obtaining a third image by removing, from the second image, at least one visual object comprised in the second image of the second image and the reference image, among the plurality of visual objects; identifying at least one feature point among a plurality of feature points identified based on a comparison between the reference image and the third image; identifying, based on the at least one feature point, a pose of a virtual camera; and identifying a content superimposed on the first image by processing 3D model based on the pose of the virtual camera.

12 . The method of claim 11 , further comprising:
identifying location of the electronic device using a global positioning system (GPS) receiver, and transmitting, to the server, information on the location of the electronic device with the information on the second image, wherein the reference image is set based on the information on the location of the electronic device and the information on the second image.

13 . The method of claim 11 ,
wherein the second image is obtained by being changed from the first image based on semantic segmentation, wherein the second image is divided to an identified area and an unidentified area, wherein the unidentified area comprises the at least one visual object, and wherein the information on the second image comprises a binary code with respect to the identified area.

14 . The method of claim 11 , wherein the pose of the virtual camera comprises a value of roll, a value of pitch, and a value of yaw of the virtual camera.

15 . The method of claim 11 , further comprising:
identifying designated number of feature points identified based on distance information for each of the identified plurality of feature points, among the identified plurality of feature points as the at least one feature point.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Kookyeon
• Nam Sangbeom
• Lee Jaemyeon

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30184

  Infrastructure · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06T7/11

  Region-based segmentation · CPC title