Home device control method, terminal device, and computer-readable storage medium

Patent metadata

Field	Value
Publication number	US-12562935-B2
Application number	US-202118004463-A
Country	US
Kind code	B2
Filing date	Jul 1, 2021
Priority date	Jul 7, 2020
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

A method includes obtaining an image corresponding to a target area for control of a home device, determining a target home device based on the image, obtaining a control menu corresponding to the target home device, displaying the control menu in the image using an augmented reality (AR) technology, and controlling, in response to a selection operation in the control menu, the target home device to execute a target control instruction corresponding to the selection operation.

First claim

What is claimed is:

1 . A method implemented by a terminal device, wherein the method comprises:
obtaining a plurality of three-dimensional models of home devices, wherein the home devices comprise a target home device; obtaining, for each of the three-dimensional models and from different angles, model images corresponding to each of the three-dimensional models; inputting each of the model images and a corresponding device identifier of a home device corresponding to each of the model images into an initial neural network model for training and to obtain a trained neural network model; obtaining an image corresponding to a target area; inputting the image corresponding to the target area into the trained neural network model to obtain a device identifier; determining the target home device based on the image and the device identifier; obtaining a control menu corresponding to the target home device; displaying the control menu in the image using augmented reality (AR) technology; and controlling, in response to a selection operation in the control menu, the target home device to execute a target control instruction corresponding to the selection operation.

2 . The method of claim 1 , further comprising:
performing device detection on the image to obtain a detected home device; obtaining a device feature of the detected home device; matching the device feature against a preset device feature in a device feature library to obtain a matching result; and further determining the target home device based on the matching result.

3 . The method of claim 1 , further comprising:
performing device detection on the image to obtain a detected home device; obtaining a device feature of the detected home device; obtaining an environment feature of an environment in which the detected home device is located; matching the device feature against a preset device feature in a device feature library to obtain a first matching result; matching the environment feature against a preset environment feature in an environment feature library to obtain a second matching result; and further determining the target home device based on the first matching result and the second matching result.

4 . The method of claim 2 , wherein before matching the device feature against the preset device feature, the method further comprises:
obtaining a plurality of three-dimensional models of home devices, wherein the home devices comprise the target home device; obtaining, for each of the three-dimensional models and from different angles, model images corresponding to each of the three-dimensional models; extracting a corresponding preset device feature from each of the model images; and establishing the device feature library based on the corresponding preset device feature corresponding to each of the model images.

5 . The method of claim 1 , further comprising:
obtaining an environment feature corresponding to the image; matching the environment feature against a preset environment feature in an environment feature library to obtain a matching result; and further determining the target home device based on the matching result.

6 . The method of claim 1 , further comprising:
obtaining a device status corresponding to the target home device; and displaying the device status in the image using the AR technology.

7 . The method of claim 1 , further comprising:
sending the image to a cloud; instructing the cloud to determine a target home device identifier based on the image; and sending the target home device identifier to the terminal device.

8 . The method of claim 1 , further comprising:
sending the device identifier to a cloud; receiving, from the cloud and in response to the device identifier, a device capability description file; and generating, based on the device capability description file, the control menu.

9 . A terminal device comprising:
a memory configured to store instructions; and a processor coupled to the memory and configured to execute the instructions to cause the terminal device to:
obtain an image corresponding to a target area;
obtain a plurality of three-dimensional models of home devices, wherein the home devices comprise a target home device;
obtain, for each of the three-dimensional models and from different angles, model images corresponding to each of the three-dimensional models;
input each of the model images and a corresponding device identifier of a home device corresponding to each of the model images into an initial neural network model to train and obtain a trained neural network model;
input the image into the trained neural network model to obtain a device identifier;
determine the target home device based on the image and the device identifier;
obtain a control menu corresponding to the target home device;
display the control menu in the image using an augmented reality (AR) technology; and
control, in response to a selection operation in the control menu, the target home device to execute a target control instruction corresponding to the selection operation.

10 . The terminal device of claim 9 , wherein the processor is further configured to execute the instructions to cause the terminal device to:
perform device detection on the image to obtain a detected home device; obtain a device feature of the detected home device; match the device feature against a preset device feature in a device feature library to obtain a matching result; and further determine the target home device based on the matching result.

11 . The terminal device of claim 9 , wherein the processor is further configured to execute the instructions to cause the terminal device to:
perform device detection on the image to obtain a detected home device; obtain a device feature of the detected home device; obtain an environment feature of an environment in which the detected home device is located; match the device feature against a preset device feature in a device feature library to obtain a first matching result; match the environment feature against a preset environment feature in an environment feature library to obtain a second matching result; and further determine the target home device based on the first matching result and the second matching result.

12 . The terminal device of claim 10 , wherein before matching the device feature against the preset device feature, the processor is further configured to execute the instructions to cause the terminal device to:
obtain a plurality of three-dimensional models of home devices, wherein the home devices comprise the target home device; obtain, for each of the three-dimensional models and from different angles, model images corresponding to each of the three-dimensional models; extract a corresponding preset device feature from each of the model images; and establish the device feature library based on the corresponding preset device feature corresponding to each of the model images.

13 . The terminal device of claim 9 , wherein the processor is further configured to execute the instructions to cause the terminal device to:
obtain an environment feature corresponding to the image; match the environment feature against a preset environment feature in an environment feature library to obtain a matching result; and further determine the target home device based on the matching result.

14 . The terminal device of claim 9 , wherein the processor is further configured to execute the instructions to cause the terminal device to:
obtain a device status corresponding to the target home device; and

Assignees

• Petal Cloud Technology Co Ltd

Inventors

• Hu Zifu
• Miao Caixia
• Zhou Feng
• Zou Wenjin

Classifications

• H04L67/125

  involving control of end-device applications over a network · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title