Storytelling is a fundamental human practice through which people share experiences and make sense of the world. But while conversational storytelling itself has been extensively studied, little is known about the multimodal cues that signal the emergence of storytelling in spontaneous interaction and whether these cues are shared across cultures. The EU-co-funded project Storytel addresses this gap by building a corpus of naturally occurring conversations which feature stretches of embedded storytelling, and which is collected across several European countries. Students from four different universities record videos of multiparty interactions with their smartphones capturing speech, sign and other bodily actions, such as gestures, gaze, posture, facial expressions and practical actions. By building on manually identified storytelling sequences, the project investigates whether machine learning methods can automatically detect such sequences and determine their interactional structure. By using a customized processing pipeline, we extract lexical, acoustic, and visual features and integrate them into an interactive visualization tool with fully anonymized videos, enabling users to explore multimodal features. We discuss the challenges of manually identifying storytelling sequences, developing the feature extraction pipeline, and applying comprehensive image and voice anonymization. The paper combines a cross-cultural multimodal case study with the evaluation of AI-based methods for analyzing culturally situated interaction, contributing to multimodal communication research and supporting the development of AI systems capable of recognizing storytelling in human interaction.
Bonifazi, A., Al-qaoud, J., Carrión, D.A., Crosilla, G., Debreslioska, S., Del Rosario Illan Castillo, M., et al. (2026). Enabling AI to Find Multimodal Signals of Storytelling Cross-Culturally: A Multi-Step Approach [10.1145/3776591.3836646].
Enabling AI to Find Multimodal Signals of Storytelling Cross-Culturally: A Multi-Step Approach
Crosilla, Giorgia;Vitali, Fabio;Yu, Mengyuan
2026
Abstract
Storytelling is a fundamental human practice through which people share experiences and make sense of the world. But while conversational storytelling itself has been extensively studied, little is known about the multimodal cues that signal the emergence of storytelling in spontaneous interaction and whether these cues are shared across cultures. The EU-co-funded project Storytel addresses this gap by building a corpus of naturally occurring conversations which feature stretches of embedded storytelling, and which is collected across several European countries. Students from four different universities record videos of multiparty interactions with their smartphones capturing speech, sign and other bodily actions, such as gestures, gaze, posture, facial expressions and practical actions. By building on manually identified storytelling sequences, the project investigates whether machine learning methods can automatically detect such sequences and determine their interactional structure. By using a customized processing pipeline, we extract lexical, acoustic, and visual features and integrate them into an interactive visualization tool with fully anonymized videos, enabling users to explore multimodal features. We discuss the challenges of manually identifying storytelling sequences, developing the feature extraction pipeline, and applying comprehensive image and voice anonymization. The paper combines a cross-cultural multimodal case study with the evaluation of AI-based methods for analyzing culturally situated interaction, contributing to multimodal communication research and supporting the development of AI systems capable of recognizing storytelling in human interaction.| File | Dimensione | Formato | |
|---|---|---|---|
|
Enabling ai to find multimodal signals of storytelling cross-culturally.pdf
accesso aperto
Tipo:
Versione (PDF) editoriale / Version Of Record
Licenza:
Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione
433.78 kB
Formato
Adobe PDF
|
433.78 kB | Adobe PDF | Visualizza/Apri |
|
Appendix.pdf
accesso aperto
Tipo:
File Supplementare
Licenza:
Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione
229.6 kB
Formato
Adobe PDF
|
229.6 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.



