In manufacturing environments, operators frequently need to input three-dimensional (3D) spatial positions to robotic systems to perform tasks such as grasping, assembly, and placement. Consequently, 3D pointing has become an important input modality in human-robot interaction for industrial applications. Previous studies have shown that different sensory cues delivered through augmented reality can improve operator performance in pointing tasks. This paper systematically evaluates the effects of visual, auditory, and haptic cues on operational performance during the manipulation of a Phantom Touch pointer to reach a target point across four real-virtual interaction configurations. The effects of different sensory modalities are compared using metrics including operational accuracy, task completion time, and subjective measures. The results indicate that sensory cues do not significantly improve pointing accuracy under aligned configurations, where both the pointer and target are either real or virtual. In contrast, under misaligned configurations, visual cues outperform other modalities in pointing accuracy, while haptic cues yield the shortest task completion time among the three modalities. These findings support a cue design principle for selecting appropriate sensory cues based on the interaction configuration and task objectives, providing practical guidance for AR-assisted human-robot interaction.
- Open Access
- Article
A Comparative Study of Augmented Reality Cues for Robotic Three-Dimensional Pointing
Author Information
Received: 04 May 2026 | Revised: 15 Aug 2026 | Accepted: 15 Aug 2026 | Published: 27 Aug 2026
Abstract
Keywords
human-robot interaction | human-machine interaction | interface design | augmented reality | pointing | sensory cues
References
- 1.
Chu, C.H.; Weng, C.Y. Experimental Analysis of Augmented Reality Interfaces for Robot Programming by Demonstration in Manufacturing. J. Manuf. Syst. 2024, 74, 463–476. https://doi.org/10.1016/j.jmsy.2024.03.016.
- 2.
Tien, K.W.; Lu, Y.J.; Chu, C.H. Task Allocation Based on Single-Limb Actions for Augmented Reality-Assisted Human-Robot Collaboration. J. Manuf. Syst. 2025, 82, 1000–1019. https://doi.org/10.1016/j.jmsy.2025.08.006.
- 3.
Chu, C.H.; Lee, I.Y. An Experimental Study on the Design of Interaction Modalities in Augmented Reality for Industrial Robot Programming. J. Comput. Inf. Sci. Eng. 2025, 25, 051005. https://doi.org/10.1115/1.4068084.
- 4.
Chu, C.H.; Liu, Y.L. Augmented Reality User Interface Design and Experimental Evaluation for Human-Robot Collaborative Assembly. J. Manuf. Syst. 2023, 68, 313–324. https://doi.org/10.1016/j.jmsy.2023.04.007.
- 5.
Ong, S.K.; Yew, A.W.W.; Thanigaivel, N.K.; et al. Augmented Reality-Assisted Robot Programming System for Industrial Applications. Robot. Comput.-Integr. Manuf. 2020, 61, 101820. https://doi.org/10.1016/j.rcim.2019.101820.
- 6.
Chen, K.J.; Chu, C.H. An Experimental Study on the Effects of Control Frame and View in Robot Teleoperation. Int. J. Hum. –Comput. Interact. 2026, 1–23. https://doi.org/10.1080/10447318.2026.2651379.
- 7.
Qin, Y.; He, W.; Zhang, X.; et al. Usability of Pseudo-Haptic Feedback for Manual Precise Manipulation of Objects in Augmented Reality. Int. J. Hum.-Comput. Interact. 2025, 41, 7923–7938.
- 8.
Wang, I.J.; Yeh, L.; Chu, C.H.; et al. Assistive Sensory Feedback for Trajectory Tracking in Augmented Reality. J. Comput. Inf. Sci. Eng. 2024, 24, 031001. https://doi.org/10.1115/1.4062543.
- 9.
Hancock, P.A.; Mercado, J.E.; Merlo, J.; et al. Improving Target Detection in Visual Search through the Augmenting Multi-Sensory Cues. Ergonomics 2013, 56, 729–738. https://doi.org/10.1080/00140139.2013.771219.
- 10.
Lin, T.C.; Krishnan, A.U.; Li, Z. Comparison of Haptic and Augmented Reality Visual Cues for Assisting Tele-Manipulation. In Proceedings of the 2022 IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA, 23–27 May 2022; pp. 9309–9316. https://doi.org/10.1109/icra46639.2022.9811669.
- 11.
Hutmacher, F. Why Is There so Much More Research on Vision than on Any Other Sensory Modality? Front. Psychol. 2019, 10, 2246. https://doi.org/10.3389/fpsyg.2019.02246.
- 12.
Suzuki, R.; Karim, A.; Xia, T.; et al. Augmented Reality and Robotics: A Survey and Taxonomy for AR-Enhanced Human-Robot Interaction and Robotic Interfaces. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April–5 May 2022; pp. 1–33. https://doi.org/10.1145/3491102.3517719.
- 13.
Krishnan, A.U.; Lin, T.C.; Li, Z. Human Preferred Augmented Reality Visual Cues for Remote Robot Manipulation Assistance: From Direct to Supervisory Control. In Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 1–5 October 2023; pp. 7034–7039. https://doi.org/10.1109/iros55552.2023.10341969.
- 14.
Teather, R.J.; Stuerzlinger, W. Visual Aids in 3D Point Selection Experiments. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction, Honolulu, Hawaii, USA, 4–5 October 2014; pp. 127–136. https://doi.org/10.1145/2659766.2659770.
- 15.
Siew, C.Y.; Ong, S.K.; Nee, A.Y.C. A Practical Augmented Reality-Assisted Maintenance System Framework for Adaptive User Support. Robot. Comput.-Integr. Manuf. 2019, 59, 115–129. https://doi.org/10.1016/j.rcim.2019.03.010.
- 16.
Wang, X.; Ong, S.K.; Nee, A.Y.C. Multi-Modal Augmented-Reality Assembly Guidance Based on Bare-Hand Interface. Adv. Eng. Informatics 2016, 30, 406–421. https://doi.org/10.1016/j.aei.2016.05.004.
- 17.
Lin, T.C.; Krishnan, A.U.; Li, Z. Perception and Action Augmentation for Teleoperation Assistance in Freeform Telemanipulation. ACM Trans. Hum.-Robot. Interact. 2024, 13, 1–40. https://doi.org/10.1145/3643804.
- 18.
Teather, R.J.; Stuerzlinger, W. Pointing at 3D Targets in a Stereo Head-Tracked Virtual Environment. In Proceedings of the 2011 IEEE Symposium on 3D User Interfaces (3DUI), Singapore, Singapore, 19–20 March 2011; pp. 87–94. https://doi.org/10.1109/3dui.2011.5759222.
- 19.
Cutting, J.E.; Vishton, P.M. Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth. In Perception of Space and Motion; Academic Press: San Diego, CA, 1995; pp. 69–117.
- 20.
Bhowmik, A.K. Virtual and Augmented Reality: Human Sensory-Perceptual Requirements and Trends for Immersive Spatial Computing Experiences. J. Soc. Inf. Disp. 2024, 32, 605–646. https://doi.org/10.1002/jsid.2001.
- 21.
Diaz, C.; Walker, M.; Szafir, D.A.; et al. Designing for Depth Perceptions in Augmented Reality. In Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nantes, France, 9–13 October 2017; pp. 111–122. https://doi.org/10.1109/ismar.2017.28.
- 22.
Liu, L.; van Liere, R.; Nieuwenhuizen, C.; et al. Comparing Aimed Movements in the Real World and in Virtual Reality. In Proceedings of the 2009 IEEE Virtual Reality Conference, Lafayette, LA, USA, 14–18 March 2009; pp. 219–222. https://doi.org/10.1109/vr.2009.4811026.
- 23.
Batmaz, A.U.; Machuca, M.D.B.; Pham, D.M.; et al. Do Head-Mounted Display Stereo Deficiencies Affect 3D Pointing Tasks in AR and VR? In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 585–592. https://doi.org/10.1109/vr.2019.8797975.
- 24.
Bashar, M.R.; Barrera Machuca, M.D.B.; Stuerzlinger, W.; et al. The Effect of Visual Depth on the Vergence–Accommodation Conflict on 3D Selection Performance within Virtual Reality Headsets. Vis. Comput. 2025, 41, 9645–9661. https://doi.org/10.1007/s00371-025-03990-x.
- 25.
Batmaz, A.U.; Turkmen, R.; Sarac, M.; et al. Re-Investigating the Effect of the Vergence-Accommodation Conflict on 3D Pointing. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, Christchurch, New Zealand, 9–11 October 2023; pp. 1–10. https://doi.org/10.1145/3611659.3615686.
- 26.
Lee, J.S.; Ham, Y.; Park, H.; et al. Challenges, Tasks, and Opportunities in Teleoperation of Excavator toward Human-in-the-Loop Construction Automation. Autom. Constr. 2022, 135, 104119. https://doi.org/10.1016/j.autcon.2021.104119.
- 27.
Ganatra, S.; Jeong, H. Enhancing Human-Robot Teleoperation through Depth Sensing and eXtended Reality. In Proceedings of the 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint Malo, France, 8–12 March 2025; pp. 1110–1114. https://doi.org/10.1109/VRW66409.2025.00221.
- 28.
Bazilinskyy, P.; van Haarlem, W.; Quraishi, H.; et al. Sonifying the Location of an Object: A Comparison of Three Methods. IFAC-Pap. 2016, 49, 531–536. https://doi.org/10.1016/j.ifacol.2016.10.614.
- 29.
Jones, J.A.; Swan, J.E., II; Singh, G.; et al. The Effects of Virtual Reality, Augmented Reality, and Motion Parallax on Egocentric Depth Perception. In Proceedings of the 5th Symposium on Applied Perception in Graphics and Visualization, Los Angeles, CA, USA, 9–10 August 2008; pp. 9–14. https://doi.org/10.1145/1394281.1394283.
- 30.
Adam, J.J.; Paas, F.G.W.C.; Buekers, M.J.; et al. Perception-Action Coupling in Choice Reaction Time Tasks. Hum. Mov. Sci. 1996, 15, 511–519. https://doi.org/10.1016/0167-9457(95)00035-6.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Suite 4002 Level 4, 447 Collins Street, Melbourne, Victoria 3000, Australia
General Inquiries: info@sciltp.com