A geometrical instability for ultra-light fields during inflation?