The Prometeia Financial Benchmark (PFB) is the EVALITA 2026 shared task on finance questions across 3 languages: Italian, English, and Turkish, and 3 difficulty levels: easy, medium, and hard. The challenge is organized in two subtasks, one on Italian data and one on all three languages. For each subtask, we have received 2 submissions. Our main takeaways are that no significant performance differences stand out across languages and difficulty levels, and that PFB appears to be a challenging benchmark for models smaller than 3B, whereas 20B models already reach an overall accuracy of 90%.

Pietro Bardelli, A., Çekiç, T., Demirtas, I., Filannino, M., Scala, S., Galassi, A., et al. (2026). PFB at EVALITA 2026: Overview of the Prometeia Financial Benchmark. CEUR-WS.

PFB at EVALITA 2026: Overview of the Prometeia Financial Benchmark

Simona Scala
;
Andrea Galassi;Gianmarco Pappacoda;Paolo Torroni
2026

Abstract

The Prometeia Financial Benchmark (PFB) is the EVALITA 2026 shared task on finance questions across 3 languages: Italian, English, and Turkish, and 3 difficulty levels: easy, medium, and hard. The challenge is organized in two subtasks, one on Italian data and one on all three languages. For each subtask, we have received 2 submissions. Our main takeaways are that no significant performance differences stand out across languages and difficulty levels, and that PFB appears to be a challenging benchmark for models smaller than 3B, whereas 20B models already reach an overall accuracy of 90%.
2026
Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)
1
7
Pietro Bardelli, A., Çekiç, T., Demirtas, I., Filannino, M., Scala, S., Galassi, A., et al. (2026). PFB at EVALITA 2026: Overview of the Prometeia Financial Benchmark. CEUR-WS.
Pietro Bardelli, Alessandro; Çekiç, Tolga; Demirtas, Irem; Filannino, Michele; Scala, Simona; Galassi, Andrea; Pappacoda, Gianmarco; Torroni, Paolo...espandi
File in questo prodotto:
File Dimensione Formato  
66.pdf

accesso aperto

Tipo: Versione (PDF) editoriale / Version Of Record
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 1.07 MB
Formato Adobe PDF
1.07 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/1069398
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact