Background. An increasing number of individuals use online Artificial Intelligence (AI) - based chatbots to retrieve information on health-related topics. This study aims to evaluate the accuracy in answering vaccine-related answers of the currently most commonly used, advanced chatbots - ChatGPT-4.0 and Google Gemini Advanced. Methods. We compared the answers provided by the World Health Organization (WHO) to 38 open questions on vaccination myths and misconception, with the answers created by ChatGPT-4.0 and Gemini Advanced. Responses were considered as “appropriate”, if the information provided was coherent and not in contrast to current WHO recommendations or to drug regulatory indications. Results and Conclusions. The rate of agreement between WHO answers and Chat-GPT-4.0 or Gemini Advanced was very high, as both provided 36 (94.7%) appropriate responses. The few discrepancies between WHO and AI-chatbots answers could not be considered “harmful”, and both chatbots often invited the user to check reliable sources, such as CDC or the WHO websites, or to contact a local healthcare professional. In their current versions, both AI-chatbots may already be powerful instrument to support the traditional communication tools in primary prevention, with the potential to improve health literacy, medication adherence, and vaccine hesitancy and concerns. Given the rapid evolution of AI-based systems, further studies are strongly needed to monitor their accuracy and reliability over time.

Fiore, M., Bianconi, A., Acuti Martellucci, C., Rosso, A., Zauli, E., Flacco, M.E., et al. (2025). Vaccination hesitancy: agreement between WHO and ChatGPT-4.0 or Gemini Advanced. ANNALI DI IGIENE MEDICINA PREVENTIVA E DI COMUNITÀ, 37(3), 390-396 [10.7416/ai.2024.2657].

Vaccination hesitancy: agreement between WHO and ChatGPT-4.0 or Gemini Advanced

Fiore M.
Co-primo
Conceptualization
;
Bianconi A.
Co-primo
Conceptualization
;
Acuti Martellucci C.
Methodology
;
Rosso A.
Methodology
;
Flacco M. E.
Penultimo
Methodology
;
Manzoli Lamberto
Ultimo
Writing – Review & Editing
2025

Abstract

Background. An increasing number of individuals use online Artificial Intelligence (AI) - based chatbots to retrieve information on health-related topics. This study aims to evaluate the accuracy in answering vaccine-related answers of the currently most commonly used, advanced chatbots - ChatGPT-4.0 and Google Gemini Advanced. Methods. We compared the answers provided by the World Health Organization (WHO) to 38 open questions on vaccination myths and misconception, with the answers created by ChatGPT-4.0 and Gemini Advanced. Responses were considered as “appropriate”, if the information provided was coherent and not in contrast to current WHO recommendations or to drug regulatory indications. Results and Conclusions. The rate of agreement between WHO answers and Chat-GPT-4.0 or Gemini Advanced was very high, as both provided 36 (94.7%) appropriate responses. The few discrepancies between WHO and AI-chatbots answers could not be considered “harmful”, and both chatbots often invited the user to check reliable sources, such as CDC or the WHO websites, or to contact a local healthcare professional. In their current versions, both AI-chatbots may already be powerful instrument to support the traditional communication tools in primary prevention, with the potential to improve health literacy, medication adherence, and vaccine hesitancy and concerns. Given the rapid evolution of AI-based systems, further studies are strongly needed to monitor their accuracy and reliability over time.
2025
Fiore, M., Bianconi, A., Acuti Martellucci, C., Rosso, A., Zauli, E., Flacco, M.E., et al. (2025). Vaccination hesitancy: agreement between WHO and ChatGPT-4.0 or Gemini Advanced. ANNALI DI IGIENE MEDICINA PREVENTIVA E DI COMUNITÀ, 37(3), 390-396 [10.7416/ai.2024.2657].
Fiore, M.; Bianconi, A.; Acuti Martellucci, C.; Rosso, A.; Zauli, Enrico; Flacco, M. E.; Manzoli, Lamberto
File in questo prodotto:
File Dimensione Formato  
Manzoli Ann Ig 2025.pdf

accesso aperto

Descrizione: Article
Tipo: Versione (PDF) editoriale / Version Of Record
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione - Non commerciale (CCBYNC)
Dimensione 121.52 kB
Formato Adobe PDF
121.52 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/1072410
Citazioni
  • ???jsp.display-item.citation.pmc??? 4
  • Scopus 9
  • ???jsp.display-item.citation.isi??? 8
  • OpenAlex ND
social impact