Rendering an extended-reality representation of a virtual meeting including virtual controllers corresponding to real-world controllers in a real-world location

Patent metadata

Field	Value
Publication number	US-12323736-B2
Application number	US-202318178247-A
Country	US
Kind code	B2
Filing date	Mar 3, 2023
Priority date	Mar 3, 2023
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

Provided are a computer program product, system, and method for rendering an extended-reality representation of a virtual meeting. An extended reality representation is generated of a virtual meeting including avatars associated with participants in the virtual meeting to render in extended-reality displays of the participants. An extended-reality representation of a virtual controller corresponding to a real-world controller in a real-world location is generated to render in at least one extended-reality display of at least one participant in the virtual meeting. Participant input is received from one of the at least one participant to control the virtual controller. The participant input is translated to control the virtual controller to commands to control the real-world controller. The commands are transmitted to the real-world controller at the real-world location to control the real-world controller to perform actions in the real-world location to carry out the transmitted commands.

First claim

What is claimed is:

1. A computer program product for interacting in a virtual meeting extended-reality environment, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising:
generating an extended-reality representation of a virtual meeting including avatars associated with participants in the virtual meeting to render in extended-reality displays of the participants in the virtual meeting;
generating in the virtual meeting an extended-reality representation of a virtual controller corresponding to a real-world controller in a real-world location to render in at least one extended-reality display of at least one participant in the virtual meeting;
receiving participant input from one of the at least one participant to control the virtual controller;
translating the participant input to control the virtual controller to commands to control the real-world controller; and
transmitting the commands to the real-world controller at the real-world location to control the real-world controller to perform actions in the real-world location to carry out the transmitted commands.

2. The computer program product of claim 1 , wherein the virtual controller comprises a virtual robot, and wherein the real world controller comprises a real-world robot capable of moving in the real-world location to interact with objects in the real-world location according to the transmitted commands.

3. The computer program product of claim 1 , wherein the virtual controller comprises a virtual cloud-based service and the real-world controller comprises a real-world cloud-based service device, wherein the transmitted commands cause the real-world cloud-based service device to control components in the real-world location.

4. The computer program product of claim 1 , wherein the operations further comprise:
determining the at least one participant in the meeting that has permission to access the virtual controller, wherein the extended-reality representation of the virtual meeting is rendered in the at least one extended-reality display of the at least one participant having permission to view the virtual controller; and
generating an additional extended-reality representation of the virtual meeting including the avatars associated the participants in the virtual meeting without rendering the virtual controller to render in the at least one extended-reality display of at least one participant in the virtual meeting that does not have permission to view the virtual controller.

5. The computer program product of claim 1 , wherein the operations further comprise:
determining a plurality of participants in the meeting having permission to access the virtual controller, wherein the virtual meeting is rendered in the extended-reality displays of the participants having permission to view the virtual controller;
receiving commands from the participants in the meeting that have permission to access the virtual controller to control the virtual controller; and
transmitting the commands from the participants determined to have access to the real-world controller at the real-world location to control the real-world controller to perform actions in the real-world location to carry out the commands.

6. The computer program product of claim 1 , wherein the operations further comprise:
receiving streaming video of the real-world location produced from volumetric video cameras located in the real-world location and capturing video of the real-world location in which the real-world controller is operating; and
rendering, in the at least one extended-reality display of the at least one participant, a viewing region, of the extended-reality representation of the virtual meeting, including an extended-reality representation of the real-world location generated from the received streaming video of the real-world location to enable the at least one participant to view the real-world location being acted upon by the real-world controller within the virtual meeting.

7. The computer program product of claim 1 , wherein the operations further comprise:
rendering virtual controls on the virtual controller displayed in the at least one extended-reality display of the at least one participant; and
detecting manipulations of the virtual controls by the at least one participant in the virtual meeting, wherein the received participant input comprises the detected manipulations.

8. The computer program product of claim 1 , wherein the operations further comprise:
rendering the virtual meeting to include at least one additional virtual controller, corresponding to at least one additional real-world controller in the real-world location or at least one additional real-world location, in the at least one extended-reality display of the at least one participant.

9. The computer program product of claim 1 , wherein the operations further comprise:
determining participants in the virtual meeting allowed to control the virtual controller; and
rendering a plurality of instances of the virtual controller in the virtual meeting to assign to the determined participants allowed to control the virtual controller, wherein the participants manipulate the instances of the virtual controller most proximate to the avatars of the participants in the virtual meeting.

10. A system for interacting in a virtual meeting extended-reality environment, comprising:
a processor; and
a computer readable storage medium having computer readable program code embodied therein that when executed by the processor performs operations, the operations comprising:
generating an extended-reality representation of a virtual meeting including avatars associated with participants in the virtual meeting to render in extended-reality displays of the participants in the virtual meeting;
generating in the virtual meeting an extended-reality representation of a virtual controller corresponding to a real-world controller in a real-world location to render in at least one extended-reality display of at least one participant in the virtual meeting;
receiving participant input from one of the at least one participant to control the virtual controller;
translating the participant input to control the virtual controller to commands to control the real-world controller; and
transmitting the commands to the real-world controller at the real-world location to control the real-world controller to perform actions in the real-world location to carry out the transmitted commands.

11. The system of claim 10 , wherein the virtual controller comprises a virtual robot, and wherein the real world controller comprises a real-world robot capable of moving in the real-world location to interact with objects in the real-world location according to the transmitted commands.

12. The system of claim 10 , wherein the operations further comprise:
determining the at least one participant in the meeting that has permission to access the virtual controller, wherein the extended-reality representation of the virtual meeting is rendered in the at least one extended-reality display of the at least one participant having permission to view the virtual controller; and
generating an additional extended-reality representation of the virtual meeting including the avatars associated the participants in the virtual meeting without rendering the virtual controller to render in the at least one extended-reality display of at least one participant in the virtual meeting that does not have permission to view the virtual controller.

13. The syste

Assignees

• IBM

Inventors

• Kairali Sudheesh S
• Rakshit Sarbajit K
• Padmanabhan Smitha
• Dongre Swapnil
• Bhide Manish Anand
• Thomas Binoy

Classifications

• H04N7/157

  defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827) · CPC title

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/1689

  Teleoperation · CPC title

• G05B2219/32014

  Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command · CPC title

• H04N7/152 Primary

  Multipoint control units therefor · CPC title