Storytelling is a fundamental human practice through which people share experiences and make sense of the world. But while conversational storytelling itself has been extensively studied, little is known about the multimodal cues that signal the emergence of storytelling in spontaneous interaction and whether these cues are shared across cultures. The EU-co-funded project Storytel addresses this gap by building a corpus of naturally occurring conversations which feature stretches of embedded storytelling, and which is collected across several European countries. Students from four different universities record videos of multiparty interactions with their smartphones capturing speech, sign and other bodily actions, such as gestures, gaze, posture, facial expressions and practical actions. By building on manually identified storytelling sequences, the project investigates whether machine learning methods can automatically detect such sequences and determine their interactional structure. By using a customized processing pipeline, we extract lexical, acoustic, and visual features and integrate them into an interactive visualization tool with fully anonymized videos, enabling users to explore multimodal features. We discuss the challenges of manually identifying storytelling sequences, developing the feature extraction pipeline, and applying comprehensive image and voice anonymization. The paper combines a cross-cultural multimodal case study with the evaluation of AI-based methods for analyzing culturally situated interaction, contributing to multimodal communication research and supporting the development of AI systems capable of recognizing storytelling in human interaction.

Bonifazi, A., Al-qaoud, J., Carrión, D.A., Crosilla, G., Debreslioska, S., Del Rosario Illan Castillo, M., et al. (2026). Enabling AI to Find Multimodal Signals of Storytelling Cross-Culturally: A Multi-Step Approach [10.1145/3776591.3836646].

Enabling AI to Find Multimodal Signals of Storytelling Cross-Culturally: A Multi-Step Approach

Crosilla, Giorgia;Vitali, Fabio;Yu, Mengyuan
2026

Abstract

Storytelling is a fundamental human practice through which people share experiences and make sense of the world. But while conversational storytelling itself has been extensively studied, little is known about the multimodal cues that signal the emergence of storytelling in spontaneous interaction and whether these cues are shared across cultures. The EU-co-funded project Storytel addresses this gap by building a corpus of naturally occurring conversations which feature stretches of embedded storytelling, and which is collected across several European countries. Students from four different universities record videos of multiparty interactions with their smartphones capturing speech, sign and other bodily actions, such as gestures, gaze, posture, facial expressions and practical actions. By building on manually identified storytelling sequences, the project investigates whether machine learning methods can automatically detect such sequences and determine their interactional structure. By using a customized processing pipeline, we extract lexical, acoustic, and visual features and integrate them into an interactive visualization tool with fully anonymized videos, enabling users to explore multimodal features. We discuss the challenges of manually identifying storytelling sequences, developing the feature extraction pipeline, and applying comprehensive image and voice anonymization. The paper combines a cross-cultural multimodal case study with the evaluation of AI-based methods for analyzing culturally situated interaction, contributing to multimodal communication research and supporting the development of AI systems capable of recognizing storytelling in human interaction.
2026
ICMI Companion '26: Companion Publication of the 28th International Conference on Multimodal Interaction
370
374
Bonifazi, A., Al-qaoud, J., Carrión, D.A., Crosilla, G., Debreslioska, S., Del Rosario Illan Castillo, M., et al. (2026). Enabling AI to Find Multimodal Signals of Storytelling Cross-Culturally: A Multi-Step Approach [10.1145/3776591.3836646].
Bonifazi, Anna; Al-qaoud, Juman; Carrión, Daniel Alcaraz; Crosilla, Giorgia; Debreslioska, Sandra; Del Rosario Illan Castillo, Maria; Eide, Øyvind; Fr...espandi
File in questo prodotto:
File Dimensione Formato  
Enabling ai to find multimodal signals of storytelling cross-culturally.pdf

accesso aperto

Tipo: Versione (PDF) editoriale / Version Of Record
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 433.78 kB
Formato Adobe PDF
433.78 kB Adobe PDF Visualizza/Apri
Appendix.pdf

accesso aperto

Tipo: File Supplementare
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 229.6 kB
Formato Adobe PDF
229.6 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/1089151
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact