System and method for processing sensor data for the visually impaired

Patent metadata

Field	Value
Publication number	US-2016325096-A1
Application number	US-201214241182-A
Country	US
Kind code	A1
Filing date	Aug 29, 2012
Priority date	Aug 30, 2011
Publication date	Nov 10, 2016
Grant date	—

Abstract

A prosthetic processing apparatus ( 200 ) for use by a visually-impaired subject comprises at least one sensor ( 104 ) configured to capture and output physical information of a spatial field, an output interface ( 208 ) coupled to a sensory input device which is configured to apply a signal to a sensory pathway of the visually impaired subject, and a processor ( 202 ) operatively coupled to the sensor ( 104 ) and to the output interface ( 208 ). The processor ( 202 ) is configured to receive ( 402 ) the physical information of the spatial field from the sensor ( 104 ), and to process ( 404 ) the received information to identify one or more salient features of a predetermined category (such as edges, plane surfaces, human faces and/or bodies) within the spatial field. The processor ( 202 ) is further configured to generate ( 406 ) a transformed representation of the spatial field in which each identified salient feature is represented in a symbolic form subject to predetermined fidelity constraints imposed by capability of the sensory input device. The transformed representation is then output ( 408 ) from the processor ( 202 ) to the sensory input device via the output interface ( 208 ).

First claim

1 . A prosthetic processing apparatus for use by a visually-impaired subject, the apparatus comprising:
at least one sensor configured to capture and output physical information of a spatial field; an output interface coupled to a sensory input device which is configured to apply a signal to a sensory pathway of the visually impaired subject; and a processor operatively coupled to the sensor and to the output interface, and which is configured to:
receive the physical information of the spatial field from the sensor;
process the received information to identify one or more salient features of a predetermined category within the spatial field;
generate a transformed representation of the spatial field in which each identified salient feature is represented in a symbolic form subject to predetermined fidelity constraints imposed by capability of the sensory input device; and
output the transformed representation to the sensory input device via the output interface.

2 . The apparatus of claim 1 wherein the processor comprises a microprocessor with associated memory, the memory containing executable instructions which, when executed by the microprocessor, cause the microprocessor to apply transformative algorithms to the received information to generate the transformed representation of the spatial field.

3 . The apparatus of claim 1 wherein the output interface is coupled to a cortical implant arranged to apply electrical stimulation to the user's visual cortex corresponding with the transformed representation.

4 . The apparatus of claim 1 wherein the output interface is coupled to a retinal implant arranged to apply electrical stimulation to the user's retina corresponding with the transformed representation.

5 . The apparatus of claim 1 wherein the sensor comprises one or more of a visual sensor, a depth sensor, and an accelerometer.

6 . The apparatus of claim 1 wherein the processor is configured to apply a structural edge-detection algorithm whereby physical information received from a depth sensor is processed to identify locations at which discontinuities in depth are detected.

7 . The apparatus of claim 1 wherein the processor is configured to apply a face detection algorithm to two-dimensional image information received from a visual sensor in order to identify the location of faces within the image.

8 . The apparatus of claim 7 wherein the processor is configured to apply a body detection algorithm whereby physical information received from a depth sensor is processed to identify physical configuration of human bodies associated with located faces.

9 . The apparatus of claim 7 wherein the face detection algorithm comprises a boosted Haar cascade algorithm.

10 . The apparatus of claim 8 wherein the body detection algorithm comprises a proximity search of depth sensor information to identify features falling within a specified volume in the vicinity of located faces.

11 . The apparatus of claim 7 wherein the locations at which faces are detected are rendered symbolically in the transformed representation of the spatial field as facial icons or avatars constructed from contrasting pixels in the transformed representation.

12 . The apparatus of claim 8 wherein the configuration of human bodies is rendered symbolically in the transformed representation of the spatial field as corresponding contrasting pixels.

13 . The apparatus of claim 1 wherein the processor is configured to estimate a direction of gravity based upon physical information received from an accelerometer, whereby a spatial orientation of physical information received from one or more additional sensors is determined.

14 . The apparatus of claim 1 which comprises a depth sensor and an accelerometer, and wherein the processor is configured to apply a ground plane detection algorithm whereby physical information received from the depth sensor is processed along with physical information received from the accelerometer in order to identify locations corresponding with a contiguous substantially horizontal plane surface with the spatial field.

15 . The apparatus of claim 14 wherein the processor is configured to apply the ground plane detection algorithm which comprises:
generating a plane hypothesis corresponding with a hypothetical plane disposed at a predetermined elevation relative to the depth sensor;
testing the plane hypothesis by comparing a distance measure of points within the spatial field detected by the depth camera with points on the hypothetical plane; and
accepting the plane hypothesis in the event that the comparison establishes a sufficiently close correlation between the detected points and the points on the hypothetical plane.

16 . The apparatus of claim 15 wherein the ground plane detection algorithm further comprises generating multiple plane hypotheses corresponding with hypothetical planes disposed at a plurality of predetermined elevations relative to the depth sensor; and
accepting the plane hypothesis having the closest correlation between the detected points and the points on the hypothetical plane.

17 . The apparatus of claim 15 wherein physical information received from the accelerometer is used in particular for determining a direction normal to the hypothetical ground plane.

18 . The apparatus of claim 17 wherein the ground plane detector algorithm includes estimating the location of the horizontal plane surface by determining an improved plane estimate by applying an iterative method based upon sampling of depth sensor information corresponding with points within the accepted hypothetical plane.

19 . The apparatus of claim 14 wherein the locations corresponding with the estimated horizontal plane surface are rendered symbolically as contrasting pixels in the transformed representation of the spatial field.

20 . The apparatus of claim 1 wherein the processor is configured to apply a blending algorithm to generate a transformed representation of the spatial field comprising elements of corresponding representations produced by two or more transformative algorithms.

21 . The apparatus of claim 20 wherein the blending algorithm assigns a precedence to the representations produced by the two or more transformative algorithms, in a manner that results in the most effective presentation of salient information.

22 . The apparatus of claim 21 wherein the transformative algorithms comprise two or more of a ground plane detection algorithm, a structural edge-detection algorithm, and a face-and-body detection algorithm, and rendering of face and body representations has precedence over rendering of ground plane representations, which in turn has precedence over rendering of structural edges.

23 . A visual processing method for use in a prosthetic apparatus of a visually-impaired subject, the method comprising:
receiving information from at least one sensor configured to capture and output physical information of a spatial field; processing the received information to identify one or more salient features of a predetermined category within the spatial field; generating a transformed representation of the spatial field in which each identified salient feature is represented in a symbolic form subject to predetermined fidelity constraints imposed by capability of a sensory input device configured to apply a signal to a sensory pathway of the visually impaired subject; and outputting the transformed representation to the

Assignees

• Lui Wen Lik Dennis
• Browne Damien
• Drummond Tom
• Li Wai Ho
• Kleeman Lindsay
• Univ Monash

Inventors

• Lui Wen Lik Dennis
• Browne Damien
• Drummond Tom
• Li Wai Ho
• Kleeman Lindsay

Classifications

• A61N1/36046 Primary

  of the eye · CPC title

• G09B21/008

  using visual presentation of the information for the partially sighted · CPC title

• A61N1/37264

  Changing the program; Upgrading firmware · CPC title

• G09B21/001

  Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence) · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title