Knowledge of Chatbots on Oriental Medicine-Based Theory and Practice of Acupuncture Is Inferior to That on Western Medicine-Based Acupuncture-Transmitted Infections

Hsi Liang; Zi-Jie Lee; Chi-Ching Tsang; Susanna K. P. Lau; Patrick C. Y. Woo

doi:10.53941/aieng.2026.100004

Abstract

Background—Chatbots are increasingly used in clinical medicine. However, all studies that examined the applications of chatbots so far were on their usefulness in western medicine. There has been no study that explored their knowledge on alternative medical practices. Objectives—To examine the performance of five free chatbots (ChatGPT, Claude, Copilot, Gemini and Perplexity) as well as the subscribed versions of ChatGPT, Claude and Perplexity in answering questions on oriental medicine-based theory and practice of acupuncture and western medicine-based acupuncture-transmitted infections. Methods—A total of 2528 multiple choices and true/false questions from four books on theory and practice of acupuncture, and 100 questions on acupuncture-transmitted infections set by two microbiology and infectious disease professors, were used for evaluation of the chatbots. Results—Overall, the median score for the eight free/subscribed chatbots in answering questions on theory and practice of acupuncture (75%) was significantly lower than that on acupuncture-transmitted infections (86%) (p < 0.001). Further analysis also showed significantly lower median score for all the five free chatbots and all the three subscribed chatbots in answering the questions on theory and practice of acupuncture (75% for both) than those on acupuncture-transmitted infections (86% and 90% respectively) (p = 0.014 and p = 0.003 respectively). For the three subscribed chatbots, GPT-4o achieved the highest median score (84%), followed by Perplexity Pro (79%) and Claude 3 Opus (66%) for the questions on theory and practice of acupuncture (p = 0.036 by Kruskal Wallis test). Post-hoc Dunn’s test revealed that the median score for GPT-4o was significantly higher than Claude 3 Opus (p = 0.032). Conclusions—Performance of the chatbots on oriental medicine-based theory and practice of acupuncture was inferior to that on western medicine-based acupuncture-transmitted infections. In order to further improve their usefulness, efforts should be spent on improving the chatbots’ knowledge on the theory and practice of acupuncture.

References

1.
Bera, K.; Braman, N.; Gupta, A.; et al. Predicting cancer outcomes with radiomics and artificial intelligence in radiology. Nat. Rev. Clin. Oncol. 2022, 19, 132–146.
2.
Försch, S.; Klauschen, F.; Hufnagl, P.; et al. Artificial intelligence in pathology. Dtsch. Ärztebl. Int. 2021, 118, 199.
3.
Shafi, S.; Parwani, A.V. Artificial intelligence in diagnostic pathology. Diagn. Pathol. 2023, 18, 109.
4.
Tsang, C.-C.; Zhao, C.; Liu, Y.; et al. Automatic identification of clinically important Aspergillus species by artificial intelligence-based image recognition: Proof-of-concept study. Emerg. Microbes Infect. 2024, 14, 2434573.
5.
Van Noorden, R.; Webb, R. ChatGPT and science: The AI system was a force in 2023—for good and bad. Nature 2023, 624, 509.
6.
Kumar, S.; Arif, T.; Alotaibi, A.S.; et al. Advances towards automatic detection and classification of parasites microscopic images using deep convolutional neural network: Methods, models and research directions. Arch. Comput. Methods Eng. 2023, 30, 2013–2039.
7.
Wang, Z.; Zhang, L.; Zhao, M.; et al. Deep neural networks offer morphologic classification and diagnosis of bacterial vaginosis. J. Clin. Microbiol. 2021, 59, e02236-20. https://doi.org/10.1128/jcm.02236-20.
8.
Song, Y.; He, L.; Zhou, F.; et al. Segmentation, splitting, and classification of overlapping bacteria in microscope images for automatic bacterial vaginosis diagnosis. IEEE J. Biomed. Health Inform. 2016, 21, 1095–1104.
9.
Goyal, M.; Knackstedt, T.; Yan, S.; et al. Artificial intelligence-based image classification methods for diagnosis of skin cancer: Challenges and opportunities. Comput. Biol. Med. 2020, 127, 104065.
10.
Wang, C.-S.; Hsiao, Y.; Tsou, C.-H.; et al. Chatbots are just as good as professors in both factual recall and clinical scenario analysis: Emergence of a new tool in clinical microbiology and infectious disease. J. Infect. 2024, 89, 106274.
11.
Jennes, F. 750 Questions and Answers about Acupuncture: Exam Preparation and Study Guide; Blue Poppy Enterprises, Inc.: Portland, OR, USA, 2003.
12.
Yongqiang, C. Tests: Chinese Acupuncture and Moxibustion; Foreign Language Press: Beijing, China, 2001.
13.
Luecke, K. Acupuncture Board Exam Practice Questions; Independently Published: Seattle, WA, USA, 2024.
14.
Luecke, K. Acupuncture Board Exam Study Questions—Differential Diagnosis; Independently Published: Seattle, WA, USA, 2024.
15.
Woo, P.C.; Li, J.H.; Tang, W.; et al. Acupuncture mycobacteriosis. N. Engl. J. Med. 2001, 345, 842–843.
16.
Woo, P.C.; Leung, K.W.; Wong, S.S.; et al. Relatively alcohol-resistant mycobacteria are emerging pathogens in patients receiving acupuncture treatment. J. Clin. Microbiol. 2002, 40, 1219–1224.
17.
Woo, P.C.; Lau, S.K.; Wong, S.S.; et al. Staphylococcus aureus subcutaneous abscess complicating acupuncture: Need for implementation of proper infection control guidelines. New Microbiol. 2003, 26, 169–174.
18.
Woo, P.C.; Lau, S.K.; Yuen, K.Y. First report of methicillin-resistant Staphylococcus aureus septic arthritis complicating acupuncture: Simple procedure resulting in most devastating outcome. Diagn. Microbiol. Infect. Dis. 2009, 63, 92–95.
19.
Woo, P.C.; Lin, A.W.; Lau, S.K.; et al. Acupuncture transmitted infections. BMJ. 2010, 340, c1268.
20.
Walsh, B. Control of infection in acupuncture. Acupunct. Med. 2001, 19, 109–111.
21.
Song, J.Y.; Sohn, J.W.; Jeong, H.W.; et al. An outbreak of post-acupuncture cutaneous infection due to Mycobacterium abscessus. BMC Infect. Dis. 2006, 6, 6.
22.
Huang, C.-C.; Kotha, P.; Tu, C.-H.; et al. Acupuncture: a review of the safety and adverse events and the strategy of potential risk prevention. Am. J. Chin. Med. 2024, 52, 1555–1587.
23.
Koh, S.J.; Song, T.; Kang, Y.; et al. An outbreak of skin and soft tissue infection caused by Mycobacterium abscessus following acupuncture. Clin. Microbiol. Infect. 2010, 16, 895–901.
24.
Murray, R.; Pearson, J.; Coombs, G.; et al. Outbreak of invasive methicillin-resistant Staphylococcus aureus infection associated with acupuncture and joint injection. Infect. Control Hosp. Epidemiol. 2008, 29, 859–865.
25.
Kim, Y.-J.; Kim, S.-H.; Lee, H.J.; et al. Infectious adverse events following acupuncture: clinical progress and microbiological etiology. J. Korean Med. Sci. 2018, 33, e164.
26.
Tang, P.; Walsh, S.; Murray, C.; et al. Outbreak of acupuncture-associated cutaneous Mycobacterium abscessus infections. J. Cutan. Med. Surg. 2006, 10, 166–169.
27.
Ryu, H.J.; Kim, W.J.; Oh, C.H.; et al. Iatrogenic Mycobacterium abscessus infection associated with acupuncture: clinical manifestations and its treatment. Int. J. Dermatol. 2005, 44, 846–850.
28.
Chung, A.; Bui, L.; Mills, E. Adverse effects of acupuncture. Which are clinically significant? Can. Fam. Physician 2003, 49, 985–989.
29.
Limited, A.A.A. Infection Control Guidelines for Acupuncture; Australian Acupuncture Association: West End, QLD, Australia, 1997.
30.
Sanderson, K. GPT-4 is here: What scientists think. Nature 2023, 615, 773.
31.
Abosi, O.J.; Kobayashi, T.; Ross, N.; et al. A head-to-head comparison of the accuracy of commercially available large language models for infection prevention and control inquiries, 2024. Infect. Control Hosp. Epidemiol. 2025, 46, 309–311.
32.
Tao, K.; Osman, Z.A.; Tzou, P.L.; et al. GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet. BMC Med. Res. Methodol. 2024, 24, 139.
33.
Lahat, A.; Sharif, K.; Zoabi, N.; et al. Assessing generative pretrained transformers (GPT) in clinical decision-making: comparative analysis of GPT-3.5 and GPT-4. J. Med. Internet Res. 2024, 26, e54571.

Scilight Press

Author Information

Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us