Multilingual large language models (LLMs) are increasingly used for factual question answering, yet their accuracy varies across languages in ways that are difficult to interpret. A central challenge is that many multilingual probing benchmarks conflate multiple factors: the language used to ask the question, the cultural-linguistic context of the entities being queried, and the popularity skew of entities. In our paper, we disentangle these factors by asking: (i) how strongly does the Language of the Question (LoQ) affect factual recall, (ii) does matching LoQ to an entity-associated Language of the Entity (LoE) improve performance, and (iii) do these effects persist when entity popularity is controlled. To this end, we introduce WILA-PopQA, a new Wikidata-grounded benchmark spanning 9 languages with matched popularity profiles, and probe 12 open-weight models of varying sizes and architectures under aligned and misaligned LoQ–LoE conditions. We evaluate models’ answers to 4 types of questions about entity biographical properties in all selected languages. Results show that LoQ is the dominant source of variation. LoQ–LoE alignment does not consistently yield the highest accuracy, and performance depends on the property being asked. These results suggest that prompt language is an actionable experimental factor for multilingual factual evaluation.

Iferroudjene, M., Poggel, L., Schimmenti, A., Yang, D., Shivashankar, K., Kalo, J., et al. (2026). A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models [10.63317/29b354pejyrt].

A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models

Schimmenti, Andrea
Co-primo
;
2026

Abstract

Multilingual large language models (LLMs) are increasingly used for factual question answering, yet their accuracy varies across languages in ways that are difficult to interpret. A central challenge is that many multilingual probing benchmarks conflate multiple factors: the language used to ask the question, the cultural-linguistic context of the entities being queried, and the popularity skew of entities. In our paper, we disentangle these factors by asking: (i) how strongly does the Language of the Question (LoQ) affect factual recall, (ii) does matching LoQ to an entity-associated Language of the Entity (LoE) improve performance, and (iii) do these effects persist when entity popularity is controlled. To this end, we introduce WILA-PopQA, a new Wikidata-grounded benchmark spanning 9 languages with matched popularity profiles, and probe 12 open-weight models of varying sizes and architectures under aligned and misaligned LoQ–LoE conditions. We evaluate models’ answers to 4 types of questions about entity biographical properties in all selected languages. Results show that LoQ is the dominant source of variation. LoQ–LoE alignment does not consistently yield the highest accuracy, and performance depends on the property being asked. These results suggest that prompt language is an actionable experimental factor for multilingual factual evaluation.
2026
Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26
190
209
Iferroudjene, M., Poggel, L., Schimmenti, A., Yang, D., Shivashankar, K., Kalo, J., et al. (2026). A Wikidata-Based Framework to Measure Cross-Lingual Bias in Multilingual Large Language Models [10.63317/29b354pejyrt].
Iferroudjene, Mouloud; Poggel, Lisa; Schimmenti, Andrea; Yang, Duo; Shivashankar, Kanchan; Kalo, Jan-Christoph; Boscariol, Marta
File in questo prodotto:
File Dimensione Formato  
unpaywall-bitstream-340667589.pdf

accesso aperto

Tipo: Versione (PDF) editoriale / Version Of Record
Licenza: Creative commons
Dimensione 4.55 MB
Formato Adobe PDF
4.55 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/1086113
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact