Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.

Andrea Galassi, Marco Lippi, Paolo Torroni (2021). Attention in Natural Language Processing. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 32(10), 4291-4308 [10.1109/TNNLS.2020.3019893].

Attention in Natural Language Processing

Andrea Galassi;Paolo Torroni
2021

Abstract

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.
2021
Andrea Galassi, Marco Lippi, Paolo Torroni (2021). Attention in Natural Language Processing. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 32(10), 4291-4308 [10.1109/TNNLS.2020.3019893].
Andrea Galassi; Marco Lippi; Paolo Torroni
File in questo prodotto:
File Dimensione Formato  
AttentionPrePub.pdf

accesso aperto

Descrizione: Pre-publication
Tipo: Postprint
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 2.73 MB
Formato Adobe PDF
2.73 MB Adobe PDF Visualizza/Apri
attention-in-natural-language-processing.pdf

accesso aperto

Descrizione: Published paper
Tipo: Versione (PDF) editoriale
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 2.57 MB
Formato Adobe PDF
2.57 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/663866
Citazioni
  • ???jsp.display-item.citation.pmc??? 25
  • Scopus 287
  • ???jsp.display-item.citation.isi??? 238
social impact