Part Identification using a Photograph and Engineering Data

Patent metadata

Field	Value
Publication number	US-2016275079-A1
Application number	US-201514659918-A
Country	US
Kind code	A1
Filing date	Mar 17, 2015
Priority date	Mar 17, 2015
Publication date	Sep 22, 2016
Grant date	—

Abstract

A database stores reference photographs of an assembly. The reference photographs are from different orientations relative to the assembly. By matching the query photograph to one or more of the reference photographs, the pose of the assembly in the query photograph is determined. Based on the pose, the pixels of the two-dimensional query photograph are related to a three-dimensional representation from engineering data. Using labeled parts from the engineering data, the parts represented in the query photograph are identified, and part information (e.g., segmentation, number, or other metadata) is provided relative to the query photograph.

First claim

What is claimed is:

1 . A system for part identification, the system comprising:
a memory configured to store a plurality of representations of reference photographs of an assembly of parts and a three-dimensional representation of the assembly with the parts separately labeled, the reference photographs having different orientations relative to the assembly and known orientations with the three-dimensional representation; an interface configured to receive a representation of a query photograph of the assembly or another assembly of a same type; a processor configured to determine an orientation of a camera to the assembly or the other assembly for the query photograph from matching with the reference photographs and the different orientations, and configured to generate an image of the query photograph with at least one of the parts labeled from the three-dimensional representation at the orientation of the camera to the assembly or other assembly.

2 . The system of claim 1 wherein the reference photographs and query photograph comprise compact representations of the photographs.

3 . The system of claim 2 wherein the compact representations comprise histograms from a scaling invariant feature transform.

4 . The system of claim 1 wherein the memory, interface, and processor are part of a server remote from the assembly or other assembly when the camera captures the query photograph.

5 . The system of claim 4 wherein the camera comprises a camera of a mobile device wirelessly connected with the interface through a computer network.

6 . The system of claim 1 wherein the processor is configured to match a plurality of the reference photographs with the query photographs and determine the orientation of the camera from the plurality of matches.

7 . The system of claim 1 wherein the processor is configured to determine a scale of the query photograph relative to the reference photographs.

8 . The system of claim 1 wherein the processor is configured to determine by assigning location in three-dimensional space of points in the query photograph.

9 . The system of claim 1 wherein the processor is configured to generate the image as a color coding overlay of the parts on the query photograph.

10 . The system of claim 1 wherein the processor is configured to generating the image by rendering from the three-dimensional representation.

11 . The system of claim 10 wherein the processor is configured to generating the image showing internal structure of the assembly or other assembly not visible in the query photograph.

12 . A method for part identification, the method comprising:
determining a pose of an assembly in a photograph; identifying a piece of the assembly from a correspondence of two-dimensional locations in the photograph to three-dimensional space of a three-dimensional representation of the assembly in the pose; and generating an image of the photograph with an overlay of information for the piece.

13 . The method of claim 12 wherein determining the pose comprises determining orientation and scale from a match of viewpoints in a plurality of references with the photograph.

14 . The method of claim 13 wherein determining comprises comparing a compact representation of the photograph with the references comprising compact representations of the assembly from different orientations and scales.

15 . The method of claim 12 further comprising:
receiving the photograph as a query from a remote mobile device; and
transmitting the image to a display of the remote mobile device.

16 . The method of claim 12 wherein determining comprises:
locating at least three points in the photograph as the two-dimensional locations; and
relating the at least three points to the three-dimensional space from a known pose relationship of reference photographs to the three-dimensional representation.

17 . The method of claim 12 wherein identifying comprises looking up parts of the assembly from labels of the three-dimensional representation.

18 . The method of claim 12 wherein identifying comprises rendering the three-dimensional representation with a view direction based on the pose, the three-dimensional representation including a label for the piece.

19 . The method of claim 12 wherein generating the image comprises generating the overlay as different color or shading for different pieces of the assembly, including the piece.

20 . A method for part identification, the method comprising:
identifying spare parts in an acquired two-dimensional view based on a visual search for a viewpoint of the two-dimensional view; and transferring part related information from three-dimensional engineering data to the two-dimensional view as a function of the viewpoint.

Assignees

• Siemens Ag

Inventors

• Kluckner Stefan
• Innanje Arun
• Ernst Jan
• Chen Terrence

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06F16/434 Primary

  using image data, e.g. images, photos, pictures taken by a user · CPC title

• G06F30/00

  Computer-aided design [CAD] · CPC title

• G06T7/75

  involving models · CPC title

• G06T2207/20104

  Interactive definition of region of interest [ROI] · CPC title