While text-conditional 3D object generation and manipulation have seen rapid progress, the evaluation of coherence between generated 3D shapes and input textual descriptions lacks a clear benchmark. The reason is twofold: a) the low quality of the textual descriptions in the only publicly available dataset of text-shape pairs; b) the limited effectiveness of the metrics used to quantitatively assess such coherence. In this paper, we propose a comprehensive solution that addresses both weaknesses. Firstly, we employ large language models to automatically refine textual descriptions associated with shapes. Secondly, we propose a quantitative metric to assess text-to-shape coherence, through cross-attention mechanisms. To validate our approach, we conduct a user study and compare quantitatively our metric with existing ones. The refined dataset, the new metric and a set of text-shape pairs validated by the user study comprise a novel, fine-grained benchmark that we publicly release to foster research on text-to-shape coherence of text-conditioned 3D generative models. Benchmark available at https://cvlab-unibo.github.io/CrossCoherence-Web/.

Looking at words and points with attention: a benchmark for text-to-shape coherence / Amaduzzi A.; Lisanti G.; Salti S.; Di Stefano L.. - ELETTRONICO. - (2023), pp. 2860-2869. (Intervento presentato al convegno IEEE/CVF International Conference on Computer Vision Workshops tenutosi a Francia nel 2023) [10.1109/ICCVW60793.2023.00309].

Looking at words and points with attention: a benchmark for text-to-shape coherence

Amaduzzi A.;Lisanti G.;Salti S.;Di Stefano L.
2023

Abstract

While text-conditional 3D object generation and manipulation have seen rapid progress, the evaluation of coherence between generated 3D shapes and input textual descriptions lacks a clear benchmark. The reason is twofold: a) the low quality of the textual descriptions in the only publicly available dataset of text-shape pairs; b) the limited effectiveness of the metrics used to quantitatively assess such coherence. In this paper, we propose a comprehensive solution that addresses both weaknesses. Firstly, we employ large language models to automatically refine textual descriptions associated with shapes. Secondly, we propose a quantitative metric to assess text-to-shape coherence, through cross-attention mechanisms. To validate our approach, we conduct a user study and compare quantitatively our metric with existing ones. The refined dataset, the new metric and a set of text-shape pairs validated by the user study comprise a novel, fine-grained benchmark that we publicly release to foster research on text-to-shape coherence of text-conditioned 3D generative models. Benchmark available at https://cvlab-unibo.github.io/CrossCoherence-Web/.
2023
Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops - ICCVW
2860
2869
Looking at words and points with attention: a benchmark for text-to-shape coherence / Amaduzzi A.; Lisanti G.; Salti S.; Di Stefano L.. - ELETTRONICO. - (2023), pp. 2860-2869. (Intervento presentato al convegno IEEE/CVF International Conference on Computer Vision Workshops tenutosi a Francia nel 2023) [10.1109/ICCVW60793.2023.00309].
Amaduzzi A.; Lisanti G.; Salti S.; Di Stefano L.
File in questo prodotto:
Eventuali allegati, non sono esposti

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/955652
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact