Teleoperation is essential for manipulation in hazardous or inaccessible environments, but is often limited by reduced perception, limited control flexibility, and low operator immersion. To overcome these issues, we present a mixed-reality (MR)–assisted teleoperation system for mobile robotic arms. The system integrates stereoscopic vision via a head-mounted display and an augmented-reality (AR) interface providing real-time feedback and safety alerts. A multimodal control strategy further supports complex maneuvers. In the experiments, participants achieved a 92.42% success rate in object retrieval and placement, and success rates in narrow corridor tasks were 84.08% with conventional displays and 89.41% under MR-based collaborative control. These findings suggest potential usability benefits for the remote operation of mobile robotic arms.
- Open Access
- Article
Mixed Reality-Assisted Multimodal Teleoperation System for Mobile Robotic Arms
Author Information
Received: 25 Jun 2026 | Revised: 20 Aug 2026 | Accepted: 22 Sep 2026 | Published: 29 Sep 2026
Abstract
Keywords
References
- 1.
Purushottam, A.; Xu, C.; Jung, Y.; et al. Dynamic Mobile Manipulation via Whole-Body Bilateral Teleoperation of a Wheeled Humanoid. IEEE Robot. Autom. Lett. 2024, 9, 1214–1221. https://doi.org/10.1109/lra.2023.3334677.
- 2.
Li, X.; Yue, H.; Yang, D.; et al. A Large-Scale Inflatable Robotic Arm toward Inspecting Sensitive Environments: Design and Performance Evaluation. IEEE Trans. Ind. Electron. 2023,70, 12486–12499.
- 3.
Wang, B.; Nersesov, S.G.; Ashrafiuon, H. Formation Regulation and Tracking Control for Nonholonomic Mobile Robot Networks Using Polar Coordinates. IEEE Control Syst. Lett. 2022, 6, 1909–1914. https://doi.org/10.1109/lcsys.2021.3135753.
- 4.
Wang, B.; Ashrafiuon, H.; Nersesov, S.G. Leader–Follower Formation Stabilization and Tracking Control for Heterogeneous Planar Underactuated Vehicle Networks. Syst. Control Lett. 2021, 156, 105008. https://doi.org/10.1016/j.sysconle.2021.105008.
- 5.
Moniruzzaman, M.; Rassau, A.; Chai, D.; et al. Teleoperation Methods and Enhancement Techniques for Mobile Robots: A Comprehensive Survey. Robot. Auton. Syst. 2022, 150, 103973.
- 6.
Guo, B.; Dian, S.; Zhao, T.; et al. Dynamic Event-Driven Neural Network-Based Adaptive Fault-Attack-Tolerant Control for Wheeled Mobile Robot System. ISA Trans. 2023, 140, 71–83. https://doi.org/10.1016/j.isatra.2023.06.010.
- 7.
Han, T.; Wang, B. Safety-Critical Stabilization of Force-Controlled Nonholonomic Mobile Robots. IEEE Control Syst. Lett. 2024, 8, 2469–2474. https://doi.org/10.1109/lcsys.2024.3492999.
- 8.
Ajoudani, A.; Zanchettin, A.M.; Ivaldi, S.; et al. Progress and Prospects of the Human-Robot Collaboration. Auton. Robots 2018, 42, 957–975. https://doi.org/10.1007/s10514-017-9677-2.
- 9.
Teng, T.; Fernandes, M.; Gatti, M.; et al. Whole-Body Control on Non-holonomic Mobile Manipulation for Grapevine Winter Pruning Automation. In Proceedings of the 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM), Chongqing, China, 3–5 July 2021; pp. 37–42. https://doi.org/10.1109/icarm52023.2021.9536083.
- 10.
Wu, Y.; Lamon, E.; Zhao, F.; et al. Unified Approach for Hybrid Motion Control of MOCA Based on Weighted Whole-Body Cartesian Impedance Formulation. IEEE Robot. Autom. Lett. 2021, 6, 3505–3512. https://doi.org/10.1109/lra.2021.3062316.
- 11.
Luo, R.C.; Lee, S.L.; Wen, Y.C.; et al. Modular ROS Based Autonomous Mobile Industrial Robot System for Automated Intelligent Manufacturing Applications. In Proceedings of the 2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Boston, MA, USA, 6–9 July 2020; pp. 1673–1678. https://doi.org/10.1109/aim43001.2020.9158800.
- 12.
Aoki, J.; Sasaki, F.; Yamashina, R.; et al. Teleoperation by Seamless Transitions in Real and VirtualWorld Environments. Robot. Auton. Syst. 2023, 164, 104405. https://doi.org/10.1016/j.robot.2023.104405.
- 13.
Szczurek, K.A.; Prades, R.M.; Matheson, E.; et al. Multimodal Multi-User Mixed Reality Human-Robot Interface for Remote Operations in Hazardous Environments. IEEE Access 2023, 11, 17305–17333. https://doi.org/10.1109/access.2023.3245833.
- 14.
Aguirre, O.A.; N˜ acato, J.C.; Andaluz, V.H. Virtual Simulator for Collaborative Tasks of Aerial Manipulator Robots. In Proceedings of the 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), Sevilla, Spain, 24–27 June 2020; pp. 1–6. https://doi.org/10.23919/cisti49556.2020.9141092.
- 15.
Naceri, A.; Mazzanti, D.; Bimbo, J.; et al. The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation. J. Intell. Robot. Syst. 2021, 101, 80. https://doi.org/10.1007/s10846-021-01311-7.
- 16.
Bustamante, S.; Peters, J.; Scholkopf, B.; et al. ArmSym: A Virtual Human-Robot Interaction Laboratory for Assistive Robotics. IEEE Trans. Hum.-Mach. Syst. 2021, 51, 568–577. https://doi.org/10.1109/thms.2021.3106865.
- 17.
Whitney, D.; Rosen, E.; Ullman, D.; et al. ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 1–9. https://doi.org/10.1109/iros.2018.8593513.
- 18.
Stotko, P.; Krumpen, S.; Schwarz, M.; et al. A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot. In Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 3–8 November 2019; pp. 3630–3637. https://doi.org/10.1109/IROS40897.2019.8968598.
- 19.
Fidalgo, C.G.; Yan, Y.; Cho, H.; et al. A Survey on Remote Assistance and Training in Mixed Reality Environments. IEEE Trans. Vis. Comput. Graph. 2023, 29, 2291–2303. https://doi.org/10.1109/tvcg.2023.3247081.
- 20.
Vogel, C.; Walter, C.; Elkmann, N. Safeguarding and Supporting Future Human-robot Cooperative Manufacturing Processes by a Projection- and Camera-based Technology. Procedia Manuf. 2017, 11, 39–46. https://doi.org/10.1016/j.promfg.2017.07.127.
- 21.
Hietanen, A.; Pieters, R.; Lanz, M.; et al. AR-Based Interaction for Human-Robot Collaborative Manufacturing. Robot. Comput.-Integr. Manuf. 2020, 63, 101891. https://doi.org/10.1016/j.rcim.2019.101891.
- 22.
Papanastasiou, S.; Kousi, N.; Karagiannis, P.; et al. Towards Seamless Human Robot Collaboration: Integrating Multimodal Interaction. Int. J. Adv. Manuf. Technol. 2019, 105, 3881–3897. https://doi.org/10.1007/s00170-019-03790-3.
- 23.
Li, Q.; Sun, M.; Song, Y.; et al. Mixed Reality-Based Brain Computer Interface System Using an Adaptive Bandpass Filter: Application to Remote Control of Mobile Manipulator. Biomed. Signal Process. Control 2023, 83, 104646. https://doi.org/10.1016/j.bspc.2023.104646.
- 24.
Lee, J.; Lim, T.; Kim, W. Investigating the Usability of Collaborative Robot Control Through Hands-Free Operation Using Eye Gaze and Augmented Reality. In Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 1–5 October 2023; pp. 4101–4106. https://doi.org/10.1109/IROS55552.2023.10342045.
- 25.
Sekkat, H.; Tigani, S.; Saadane, R.; et al. Vision-Based Robotic Arm Control Algorithm Using Deep Reinforcement Learning for Autonomous Objects Grasping. Appl. Sci. 2021, 11, 7917. https://doi.org/10.3390/app11177917.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Suite 4002 Level 4, 447 Collins Street, Melbourne, Victoria 3000, Australia
General Inquiries: info@sciltp.com