In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic.

FrankenMask: Manipulating semantic masks with transformers for face parts editing / Fontanini T.; Ferrari C.; Lisanti G.; Galteri L.; Berretti S.; Bertozzi M.; Prati A.. - In: PATTERN RECOGNITION LETTERS. - ISSN 0167-8655. - ELETTRONICO. - 176:(2023), pp. 14-20. [10.1016/j.patrec.2023.10.010]

FrankenMask: Manipulating semantic masks with transformers for face parts editing

Lisanti G.;
2023

Abstract

In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic.
2023
FrankenMask: Manipulating semantic masks with transformers for face parts editing / Fontanini T.; Ferrari C.; Lisanti G.; Galteri L.; Berretti S.; Bertozzi M.; Prati A.. - In: PATTERN RECOGNITION LETTERS. - ISSN 0167-8655. - ELETTRONICO. - 176:(2023), pp. 14-20. [10.1016/j.patrec.2023.10.010]
Fontanini T.; Ferrari C.; Lisanti G.; Galteri L.; Berretti S.; Bertozzi M.; Prati A.
File in questo prodotto:
File Dimensione Formato  
1-s2.0-S0167865523002829-main.pdf

accesso aperto

Tipo: Versione (PDF) editoriale
Licenza: Licenza per Accesso Aperto. Creative Commons Attribuzione - Non commerciale - Non opere derivate (CCBYNCND)
Dimensione 2.37 MB
Formato Adobe PDF
2.37 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11585/955648
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 0
social impact