Image processing method of generating an image based on a user viewpoint and image processing device

Patent metadata

Field	Value
Publication number	US-11634028-B2
Application number	US-201715840382-A
Country	US
Kind code	B2
Filing date	Dec 13, 2017
Priority date	Jul 28, 2017
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

Provided are an image processing method and an image processing device. The image processing method includes generating an image based on viewpoint information of a user; rendering the image based on information about what is in front of the user; and outputting the rendered image using an optical element.

First claim

What is claimed is:

1. An image processing method comprising:
tracking first information including a first position of a left eye of a user and a second position of a right eye of the user;
generating second information associated with at least one of surroundings and an object present in front of the user;
rendering an image of a display panel so as to provide a first virtual image to be input to the left eye of the user and a second virtual image to be input to the right eye of the user on an imaging plane based on the first information and the second information; and
controlling the display panel to output the rendered image,
wherein a depth range of a 3-Dimensional (3D) Augmented Reality (AR) object is determined based on the depth of the imaging plane,
wherein a depth of the 3D AR object, within the depth range, represented by the first and second virtual images is different from a depth of the imaging plane, and
wherein the first information is acquired by directly detecting the right eye or the left eye of the user using at least one of an image camera and an infrared camera or acquired by detecting the right eye or the left eye of the user as reflected on a windshield of a vehicle.

2. The image processing method of claim 1 , wherein the second information comprises at least one of a shape, a position, and a depth of the object present in front of the user acquired by using a sensor.

3. The image processing method of claim 1 , wherein the rendering of the rendered image comprises:
generating the 3D AR object based on the second information; and
determining pixel values of the image of the display panel based on the 3D AR object and the first information.

4. The image processing method of claim 3 , wherein the determining of the pixel values comprises:
allocating a first image to be input to the left eye of the user to a plurality of sub-pixels of the display panel corresponding to the first position of the left eye of the user; and
allocating a second image to be input to the right eye of the user to a plurality of sub-pixels of the display panel corresponding to the second of the right eye of the user.

5. The image processing method of claim 1 , further comprising:
outputting the rendered image using an optical element.

6. The image processing method of claim 5 , wherein the optical element is at least one of a lenticular lens and a parallax barrier.

7. The image processing method of claim 5 , wherein the outputting the rendered image comprises enlarging the rendered image using a magnifying optical system.

8. The image processing method of claim 1 , wherein based on a determination that the depth of the imaging plane is between a first distance and a second distance, the depth of the 3D AR object is represented as a near distance image or as a far distance image,
wherein based on a determination that the depth of the imaging plane is between a third distance and a fourth distance, the depth of the 3D AR object is represented as the far distance image, and
wherein the third and the fourth distance is farther from the right and left eyes of the user than the first distance and the second distance.

9. The image processing method of claim 1 , wherein based on a determination that the depth of the imaging plane is between 2 and 3 meters, the depth of the 3D AR object is set to be between 2 to 100 meters, and
wherein based on a determination that the depth of the imaging plane is between 3 to 7 meters, the depth of the 3D AR object is set to be greater than the depth of the imaging plane.

10. An image processing device comprising:
a display panel comprising a plurality of pixels;
a controller configured to track first information including a first position of a left eye of a user and a second position of a right eye of the user, generate second information associated with at least one of surroundings and an object present in front of the user, render an image of the display panel so as to provide a first virtual image to be input to a left eye of the user and a second virtual image to be input to a right eye of the user on an imaging plane based on the first information and the second information, and control the display panel to output the rendered image, wherein a depth range of a 3-Dimensional (3D) Augmented Reality (AR) object is determined based on the depth of the imaging plane, wherein a depth of the 3D AR object, within the depth range, represented by the first and second virtual images is different from a depth of the imaging plane;
a magnifying optical system configured to enlarge the rendered image; and
at least one of an image camera and an infrared camera configured to acquire the first information based on one of a direct detection of the right eye or the left eye of the user directly and a detection of the right eye or the left eye of the user as reflected on a windshield of a vehicle.

11. The image processing device of claim 10 , further comprising a sensor configured to acquire at least one of a shape, a position, and a depth of the object present in front of the user.

12. The image processing device of claim 10 , wherein the controller is further configured to allocate first image to be input to the left eye of the user to a plurality of sub-pixels of the display panel corresponding to the first position of the left eye of the user, and to allocate a second image to be input to the right eye of the user to a plurality of sub-pixels of the display panel corresponding to the second position of the right eye of the user.

13. The image processing device of claim 10 , wherein the controller is further configured to render the rendered image based on an optical transformation of the first information.

14. The image processing device of claim 10 , further comprising:
at least one of a lenticular lens and a parallax barrier.

15. The image processing device of claim 10 , wherein the magnifying optical system comprises at least one of an aspherical mirror and a plane mirror.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Jinho
• Nam Dong Kyung
• Park Juyong
• Cho Yang Ho
• Choi Kyuhwan
• Choi Yoonsun

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• B60K35/235

  with means for detecting the driver's gaze direction or eye points · CPC title

• B60K35/233

  controlling the size or position in display areas of virtual images depending on the condition of the vehicle or the driver · CPC title

• B60K35/60

  Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10) · CPC title

• B60K2360/149

  Instrument input by detecting viewing direction not otherwise provided for · CPC title