POSTED: 05 Aug, 2024

In advanced industries, the integration of Extended Reality (XR) technologies into Human-Robot Collaboration (HRC) presents unprecedented opportunities and challenges. XR, encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), plays a crucial role in overcoming barriers to HRC adoption across various sectors. This article introduces the current applications of XR in HRC, addressing aspects such as types and roles, design guidelines and frameworks, and devices and platforms. It also provides insights into the future direction of XR in HRC, highlighting its potential to enhance collaboration and efficiency in industrial environments.

Extended Reality

In general, Extended Reality (XR) serves as an umbrella term for immersive technologies like Virtual Reality (VR), Mixed Reality (MR), and Augmented Reality (AR). Virtual Reality immerses users in a completely computer-generated environment (including visual, acoustical, tactile information), while Augmented Reality enhances the real-world environment by overlaying digital information or objects onto it. Specifically, Mixed Reality (MR) refers to formats that bridge the gap between reality and Virtual Reality.

In Human-Robot Collaboration (HRC), XR technologies are trending towards enhancing safety, improving workspace design, data visualisation, training operators, and creating more intuitive user interfaces due to their capability to visualise unseen information in the physical world in real time. These applications are closely linked to aiding human decision-making. By enhancing safety, XR technologies reduce the cognitive workload on operators, allowing them to focus on critical decision points. Well-designed XR-enabled workspaces facilitate the seamless integration of human and robotic workflows, boosting collaboration and efficiency. Advanced visualisation and immersive training capabilities provided by XR tools give operators a better understanding and control, leading to higher quality and precision in their decisions. Intuitive XR-based interfaces improve human-robot interactions, resulting in faster and more efficient decision-making. This effective decision-making is crucial in complex and dynamic HRC environments.

Extended Reality in Human-Robot Collaboration

From 2023 onwards, research has explored various types of XR technologies applied in Human-Robot Collaboration (HRC), including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Generally, XR is primarily used as an interface. Additionally, XR serves multiple roles such as development environments, learning environments, platforms for design, visualisation, simulation, instruction and guidance, task and motion planning, and more.

Currently, VR is used as an interface, evaluation tool, simulation platform, task and motion planning aid, learning environment, design tool, and for data collection. Conversely, AR overlays digital information onto the real world, making it ideal for enhancing and augmenting real-world interactions. MR blends the physical and digital worlds, providing immersive experiences that enhance real-time interactions and task execution. The distinction between AR and MR is often unclear, with AR considered a subset of MR. Telepresence, achievable by combining VR and MR, allows multi-human-robot teams to collaborate from different locations.

In current research on XR in HRC, various XR devices such as (Head-Mounted Displays) HMDs, mobile devices, and projectors are utilised. While HMDs are commonly employed, projectors are sometimes used for AR-based interfaces in HRC. Additionally, mobile devices like tablets are utilised for AR-based visualisation, instruction and guidance, interfaces, and training.

Regarding software and tools for developing XR in HRC, the game engine Unity is the most popular choice. In specific areas such as HRC fabrication, Building Information Modelling (BIM) platforms, and Computer-Aided Design (CAD) platforms like Rhino 3D and Grasshopper are used. Unity is generally preferred because it is powerful enough to support various platforms and users.

The Future of Extended Reality in Human-Robot Collaboration

Recently released HMDs such as Varjo XR-4 and Apple Vision Pro, AR goggles such as Xreal Air 2 Pro and Viture Pro show considerable promise for future use in HRC. The newest HMDs feature enhanced display resolution, refresh rates, and reduced latency, making them increasingly powerful. Conversely, AR goggles are lightweight while still offering high resolution and refresh rates. Moreover, mobile devices such as tablets and smartphones remain highly accessible and user-friendly for mobile AR applications, continuing to be a viable option for future use. The potential of Unreal Engine and WebGL also warrants further exploration. Unreal Engine provides photorealistic visuals for the most immersive visualisations, while WebGL enables users to interact through web-based applications from various locations and devices, enhancing accessibility and flexibility.

Current designs often focus either on XR or HRC without sufficient attention to user experience and human factors. Therefore, future research should integrate human factors and user-centric approaches to enhance the effectiveness and usability of XR in HRC. This comprehensive analysis highlights the importance of combining advanced XR technologies with human-centric design to optimise human-robot collaboration.