Symbolic Knowledge-Extraction Evaluation Metrics: The FiRe Score

Symbolic knowledge-extraction (SKE) techniques are becoming of key importance for AI applications since they enable the explanation of opaque black-box predictors, enhancing trust and transparency. Among all the available SKE techniques, the best option for the case at hand should be selected. However, an automatic comparison between different options can be performed only if an adequate metric - such as a scoring function resuming all the interesting features of the extractors - is provided. Regrettably, the literature currently lacks definitions of effective evaluation metrics for symbolic knowledge extractors. This paper proposes the novel FiRe score metric, which comprehensively assesses the quality of an SKE procedure by considering both its predictive performance and the readability of the extracted knowledge. FiRe is compared to another existing scoring metric and a rigorous mathematical formulation is provided along with several practical examples to highlight its effectiveness to the end of being exploited inside automatic hyper-parameter tuning procedures.