XR device and method for controlling the same

Patent metadata

Field	Value
Publication number	US-11436757-B2
Application number	US-202016880915-A
Country	US
Kind code	B2
Filing date	May 21, 2020
Priority date	Dec 24, 2019
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

The present disclosure relates to an XR device and a method for controlling the same, and more particularly, is applicable to a 5G communication technology field, a robot technology field, an autonomous technology field and an artificial intelligence (AI) technology field. The method for controlling an XR device of a vehicle includes acquiring a camera view by capturing an image in front of the vehicle; acquiring position information of the vehicle by detecting a position of the vehicle, acquiring movement information of the vehicle by detecting movement of the vehicle, and providing navigation of an augmented reality (AR) mode displaying at least one virtual object for guiding a path by overlapping the at least one virtual object on the camera view based on at least the position information of the vehicle or the movement information of the vehicle.

First claim

What is claimed is:

1. An extended reality (XR) device of a vehicle, the XR device comprising:
a display configured to display, on a camera view screen, an image in front of the vehicle captured by at least one camera; and
a processor configured to provide navigation information of an augmented reality (AR) mode by overlapping at least one virtual object on the camera view screen,
wherein the processor is further configured to:
set a crop region in the image displayed on the camera view screen, wherein the crop region is a region that is an obstacle to providing the navigation information to a user;
crop the crop region from the camera view screen; and
perform calibration based on the camera view screen in which the crop region is cropped,
wherein guide information for the calibration is rendered on the camera view screen, and
wherein the guide information includes a graphic user interface.

2. The XR device of claim 1 , wherein the processor is further configured to set the crop region based on at least a vanish point of a road or a bonnet line of the vehicle included in the camera view screen.

3. The XR device of claim 2 , wherein the processor is further configured to detect one or more lanes of the road based on the image captured by the at least one camera and determine a point where the detected one or more lanes meet, as the vanish point.

4. The XR device of claim 2 , wherein the processor is further configured to determine the bonnet line of the vehicle included in the camera view screen based on a deep neural network (DNN) model capable of learning position and time information.

5. The XR device of claim 2 , wherein the processor is further configured to perform the calibration for matching a coordinate of the at least one virtual object with a coordinate of the camera view screen by controlling the coordinate of the camera view screen based on the vanish point of the road and the bonnet line of the vehicle included in the camera view screen.

6. The XR device of claim 1 , wherein the processor is further configured to maintain the AR mode if the calibration is successfully performed, and provide navigation information of a map mode if the calibration is failed many times.

7. The XR device of claim 1 , wherein the the crop region is at least a portion of a bonnet of the vehicle included in the camera view screen.

8. A method for controlling an extended reality (XR) device of a vehicle, the method comprising:
displaying, on a camera view screen, an image in front of the vehicle captured by at least one camera; and
providing navigation information of an augmented reality (AR) mode by overlapping at least one virtual object on the camera view screen,
wherein providing the navigation information comprises:
setting a crop region in the image displayed on the camera view screen, wherein the crop region is a region that is an obstacle to providing the navigation information to a user;
cropping the crop region from the camera view screen; and
performing calibration based on the camera view screen in which the crop region is cropped,
wherein guide information for the calibration is rendered on the camera view screen, and
wherein the guide information includes a graphic user interface.

9. The method of claim 8 , wherein the crop region is set based on at least a vanish point of a road or a bonnet line of the vehicle included in the camera view screen.

10. The method of claim 9 , further comprising:
detecting one or more lanes of the road based on the image captured by the at least one camera; and
determining a point where the detected one or more lanes meet, as the vanish point.

11. The method of claim 9 , further comprising:
determining the bonnet line of the vehicle included in the camera view screen based on a deep neural network (DNN) model capable of learning position and time information.

12. The method of claim 9 , wherein performing the calibration includes performing the calibration for matching a coordinate of the at least one virtual object with a coordinate of the camera view screen by controlling the coordinate of the camera view screen based on the vanish point of the road and the bonnet line of the vehicle included in the camera view screen.

13. The method of claim 8 , wherein of providing the navigation information includes maintaining the AR mode if the calibration is successfully performed, and providing navigation information of a map mode if the calibration is failed many times.

14. The method of claim 8 , wherein the crop region is at least a portion of a bonnet of the vehicle included in the camera view screen.

15. The XR device of claim 1 , wherein the processor is further configured to control the display to display a carpet image with the navigation information depending on at least a moving direction of the vehicle or a driving guide line on the camera view screen on which the crop region is cropped and the calibration is performed.

16. The XR device of claim 1 , wherein the crop region is at least a portion of a dashboard of the vehicle included in the camera view screen.

17. The XR device of claim 1 , wherein the at least one camera is fixed within the vehicle to capture the image in front of the vehicle.

18. The XR device of claim 1 , wherein the at least one camera is built in the XR device to capture the image in front of the vehicle.

19. The XR device of claim 1 , wherein the guide information further includes a guide message in text form for guiding and adjusting an optimal position of the XR device with the at least one camera built-in.

20. The XR device of claim 19 , wherein the guide information further includes a guide message for representing whether the position adjustment of the XR device succeeds.

21. The method of claim 8 , further comprising:
displaying a carpet image with the navigation information depending on at least a moving direction of the vehicle or a driving guide line on the camera view screen on which the crop region is removed and the calibration is performed.

22. The method of claim 8 , wherein the crop region is at least a portion of a dashboard of the vehicle included in the camera view screen.

23. The method of claim 8 , wherein the image in front of the vehicle is captured by the at least one camera that is fixed within the vehicle.

24. The method of claim 8 , wherein the image in front of the vehicle is captured by the at least one camera that is built in the XR device.

25. The method of claim 8 , wherein the guide information further includes a guide message in text form for guiding and adjusting an optimal position of the XR device with the at least one camera built-in.

26. The method of claim 25 , wherein the guide information further includes a guide message for representing whether the position adjustment of the XR device succeeds.

Assignees

• Lg Electronics Inc

Inventors

• Jung Junyoung
• Ji Aettie
• LEE Kihyung
• Jung Kyungjin

Classifications

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title