Systems, methods, and computer program products for populating environment models

Patent metadata

Field	Value
Publication number	US-12579733-B2
Application number	US-202318367122-A
Country	US
Kind code	B2
Filing date	Sep 12, 2023
Priority date	Sep 16, 2022
Publication date	Mar 17, 2026
Grant date	Mar 17, 2026

Abstract

Systems, methods, and computer program products for managing and populating environment models are described. An environment model is accessed which represents an environment, and the environment model is populated with instances of object models. Locations where the instances of object models should be positioned in the environment model are identified, by determining where in the environment model a respective size of each instance when viewed from a vantage point at the environment model matches a size of the object represented by the respective instance when viewed from a corresponding vantage point at the environment.

First claim

The invention claimed is:

1 . A computer program product comprising at least one non-transitory processor-readable storage medium storing processor-executable instructions and/or data that, when executed by at least one processor of a processor-based system, cause the processor-based system to:
access, by the at least one processor, an environment model representation of an environment; access, by the at least one processor, a first view of the environment from a first vantage point, the first vantage point having a position and a perspective in relation to the environment, wherein the first view includes an object in the environment; access, in a library of object models, an object model representation of the object, the object model including dimension data indicative of spatial dimensions of the object; and populate the environment model with an instance of the object model at a location in the environment model, wherein the processor-executable instructions which cause the processor-based system to populate the environment model with the instance of the object model at the location cause the processor-based system to: generate a second view of the environment model from a second vantage point, wherein a position and a perspective of the second vantage point in relation to the environment model substantially match the position and the perspective of the first vantage point in relation to the environment; identify the location in the environment model where at least one spatial dimension of the instance of the object model in the second view of the environment model from the second vantage point substantially matches a corresponding spatial dimension of the object in the first view of the environment from the first vantage point; and position the instance of the object model at the location, wherein: the at least one processor is carried by a robot body positioned at the environment; the at least one non-transitory processor-readable storage medium is carried by the robot body; the at least one non-transitory processor-readable storage medium stores the library of object models; and the at least one non-transitory processor-readable storage medium stores the environment model.

2 . The computer program product of claim 1 , wherein:
the first view comprises first image data having a first resolution; the second view comprises second image data having a second resolution; the processor-executable instructions which cause the processor-based system to identify the location in the environment model where at least one spatial dimension of the instance of the object model in the second view of the environment model from the second vantage point substantially matches a corresponding spatial dimension of the object in the first view of the environment from the first vantage point cause the processor-based system to: identify the location in the environment model where a number of pixels occupied by the instance of the object model in the second image data corresponds to a number of pixels occupied by the object in the first image data.

3 . The computer program product of claim 2 , wherein:
the first resolution is equal to the second resolution; and the processor-executable instructions which cause the processor-based system to identify the location in the environment model where a number of pixels occupied by the instance of the object model in the second image data corresponds to a number of pixels occupied by the object in the first image data cause the processor-based system to: identify the location in the environment model where a number of pixels occupied by the instance of the object model in the second image data is equal to a number of pixels occupied by the object in the first image data.

4 . The computer program product of claim 2 , wherein:
the first resolution is different from the second resolution by a fixed ratio; the processor-executable instructions which cause the processor-based system to identify the location in the environment model where a number of pixels occupied by the instance of the object model in the second image data corresponds to a number of pixels occupied by the object in the first image data cause the processor-based system to: identify the location in the environment model where a number of pixels occupied by the instance of the object model in the second image data is equal to a number of pixels occupied by the object in the first image data multiplied by the fixed ratio.

5 . The computer program product of claim 1 , wherein the processor-executable instructions further cause the processor-based system to generate the object model representing the object in the library of object models.

6 . The computer program product of claim 5 , wherein the processor-executable instructions which cause the processor-based system to generate the object model representing the object cause the processor-based system to: generate the object model representing the object, including the dimension data indicative of the spatial dimensions of the object.

7 . The computer program product of claim 5 , wherein:
the processor-executable instructions further cause the processor-based system to capture, by at least one image sensor, image data representing the object from multiple viewpoints; and the processor-executable instructions which cause the processor-based system to generate the object model representing the object in the library of object models cause the processor-based system to generate the object model based on the captured image data from multiple viewpoints.

8 . The computer program product of claim 5 wherein:
the processor-executable instructions further cause the processor-based system to capture, by at least one haptic sensor positioned at an actuatable member which contacts the object, haptic data representing the object; and
the processor-executable instructions which cause the processor-based system to generate the object model representing the object in the library of object models cause the processor-based system to generate the object model based on the captured haptic data.

9 . The computer program product of claim 1 , wherein:
the environment is a three-dimensional environment; the environment model is a three-dimensional environment model; the first view comprises first two-dimensional image data representing the environment from the first vantage point; the second view comprises second two-dimensional image data representing the environment model from the second vantage point; and the processor-executable instructions which cause the processor-based system to populate the environment model with the instance of the object model at the location further cause the processor-based system to, prior to identifying the location, position the instance of the object model in the second image data to correspond to a position of the object in the first image data.

10 . The computer program product of claim 9 , wherein the processor-executable instructions which cause the processor-based system to populate the environment model with the instance of the object model at the location further cause the processor-based system to, prior to identifying the location, orient the instance of the object model in the second image data to correspond to an orientation of the object in the first image data.

11 . The computer program product of claim 9 , wherein the processor-executable instructions further cause the processor-based system to determine a distance in the environment model between the second vantage point and the instance of the object model at the location.

12 . The computer program product of claim 1 ,

Assignees

• Sanctuary Cognitive Systems Corp

Inventors

• Gildert Suzanne
• Rose Geordie

Classifications

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T7/75

  involving models · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G05B2219/40298

  Manipulator on vehicle, wheels, mobile · CPC title