Symmetries in Neural Networks

There was a phenomenal discussion on Twitter sparked by the following tweet. I was left with many more questions than answers, but I think this is a good thing. I summarize various ideas shared below.

Here’s a list of some interesting papers mentioned.

First, papers referenced in main tweet:

Lagrangian Neural Network

Now, some papers mentioned in the thread: @KrippendorfSven 2020- identifies symmetries by considering generating operators in a compressed latent space

@erikjbekkers 2019(images/grids) & @m_finzi - (alt. technique + extends to point clouds + Hamiltonian GNs) - Enforce symmetry in CNNs over a variety of Lie groups (2D rotational symmetry ~ “SO(2) Lie group”)

@markvanderwilk 2018- moves the symmetry enforcement from the architecture to the loss function

@DaniloJRezende 2019- demonstrates how to impose specific Lie group symmetries on densities in a Hamiltonian flow model

Kyle Cranmer:

@plastiq 2019- Extends the VAE to learn symmetries by simultaneously learning a transformation between different latent states or “views”

Rao & Ruderman (1999!)- demonstrates how to learn a Lie group operator (matrix) from data.

Ansemi et al 2017 - describes some weight regularizations to guide the NN in learning a symmetry rather than enforcing it



Leave a comment