Yet another approximation of human semantic judgments using LLMs... but with quantized local models on novel data

This study investigates the automatic generation of semantic norms on word specificity using various quantized open-source local Large Language Models (LLMs), including a comparison with a proprietary model (i.e. GPT-4). Word specificity norms on English are still not public, thus they are not included in the training datasets of all tested models. This offers a novel contribution by assessing LLMs ability to generalize beyond pre-trained knowledge. Our findings reveal that smaller, local quantized models such as Llama3, Phi3, and Mistral underperform in generating human-like judgments of word specificity, while a larger model such as Mixtral, even if slightly less accurate than GPT-4, represents a viable alternative to proprietary models if adequate computational resources are accessible. These findings open up new perspectives for research on linguistic features and on the scalablility of semantic norms without relying on proprietary models.