In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington–Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model’s architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
Alberici D., Barra A., Contucci P., Mingione E. (2020). Annealing and Replica-Symmetry in Deep Boltzmann Machines. JOURNAL OF STATISTICAL PHYSICS, 180(1-6), 665-677 [10.1007/s10955-020-02495-2].
Annealing and Replica-Symmetry in Deep Boltzmann Machines
Alberici D.;Contucci P.
;Mingione E.
2020
Abstract
In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington–Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model’s architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.File | Dimensione | Formato | |
---|---|---|---|
abcm-11585-741586_def.pdf
Open Access dal 06/02/2021
Tipo:
Postprint
Licenza:
Licenza per accesso libero gratuito
Dimensione
1.09 MB
Formato
Adobe PDF
|
1.09 MB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.