Symbolic knowledge injection (SKI) represents a promising paradigm for bridging symbolic knowledge and sub-symbolic predictors in intelligent autonomous agents. Given the wide availability of SKI methods from the literature, we observe that SKI effectiveness is commonly measured in terms of predictive performance variation – e.g., accuracy improvement – introduced by SKI. However, other aspects such as the injection mechanism’s ability to maintain its performance and generalisation capability, despite encountering unexpected or anomalous input during the training process, are equally relevant. Accordingly, in this paper we propose a new metric to evaluate the robustness of SKI techniques, defined as a measure of performance degradation in response to systematic dataset variations. The proposed metric enables precise quantification of the robustness degree across injection approaches and different perturbations. Details on generating and quantifying perturbations are also provided. We evaluate the effectiveness of our metric through several experiments, where we apply multiple SKI techniques to three datasets and measure how robustness varies as perturbations increase.

Andrea Rafanelli, M.M. (2024). An Empirical Study on the Robustness of Knowledge Injection Techniques Against Data Degradation. Sun SITE Central Europe, RWTH Aachen University.

An Empirical Study on the Robustness of Knowledge Injection Techniques Against Data Degradation

Matteo Magnini;Andrea Agiollo;Giovanni Ciatto;Andrea Omicini
2024

Abstract

Symbolic knowledge injection (SKI) represents a promising paradigm for bridging symbolic knowledge and sub-symbolic predictors in intelligent autonomous agents. Given the wide availability of SKI methods from the literature, we observe that SKI effectiveness is commonly measured in terms of predictive performance variation – e.g., accuracy improvement – introduced by SKI. However, other aspects such as the injection mechanism’s ability to maintain its performance and generalisation capability, despite encountering unexpected or anomalous input during the training process, are equally relevant. Accordingly, in this paper we propose a new metric to evaluate the robustness of SKI techniques, defined as a measure of performance degradation in response to systematic dataset variations. The proposed metric enables precise quantification of the robustness degree across injection approaches and different perturbations. Details on generating and quantifying perturbations are also provided. We evaluate the effectiveness of our metric through several experiments, where we apply multiple SKI techniques to three datasets and measure how robustness varies as perturbations increase.
2024
WOA 2024 – 25th Workshop "From Objects to Agents 2024"
20
32
Andrea Rafanelli, M.M. (2024). An Empirical Study on the Robustness of Knowledge Injection Techniques Against Data Degradation. Sun SITE Central Europe, RWTH Aachen University.
Andrea Rafanelli, Matteo Magnini, Andrea Agiollo, Giovanni Ciatto, Andrea Omicini
File in questo prodotto:
File Dimensione Formato  
paper_02.pdf

accesso aperto

Tipo: Versione (PDF) editoriale
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione (CCBY)
Dimensione 1.4 MB
Formato Adobe PDF
1.4 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/975934
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact