This library contains a TensorFlow implementation of the rotation equivariant CNNs for spherical signals (e.g. omnidirectional images, signals on the globe) as presented in [1]. Equivariant networks for the plane are available here.
- TensorFlow: https://www.tensorflow.org/ (>= 2.0)
- cython: https://github.com/cython/cython (<= 3.0)
- lie_learn: https://github.com/kalekundert/lie_learn@fix-install
(commands to install all the dependencies on a new environment)
# tf-s2cnn deps
python -m pip install --upgrade pip wheel cython<3.0
pip install git+https://github.com/kalekundert/lie_learn@fix-install
pip install tensorflow==2.14.0 # get correct command line at https://www.tensorflow.org/
pip install tqdm
# update venv\lib\site-packages\lie_learn\representations\SO3\wigner_d.py
import collections
collections.Iterable = collections.abc.Iterable
Please have a look at the examples.
Please cite [1] in your work when using this library in your experiments.
Spherical CNNs come with different choices of grids and grid hyperparameters which are on the first look not obviously related to those of conventional CNNs.
The s2_near_identity_grid and so3_near_identity_grid are the preferred choices since they correspond to spatially localized kernels, defined at the north pole and rotated over the sphere via the action of SO(3).
In contrast, s2_equatorial_grid and so3_equatorial_grid define line-like (or ring-like) kernels around the equator.
To clarify the possible parameter choices for s2_near_identity_grid:
Adapts the size of the kernel as the angle measured from the north pole.
Conventional CNNs on flat space usually use a fixed kernel size but pool the signal spatially.
This spatial pooling gives the kernels in later layers an effectively increased field of view.
One can emulate a pooling by a factor of 2 in spherical CNNs by decreasing the signal bandwidth by 2 and increasing max_beta by 2.
Number of rings of the kernel around the equator, equally spaced in
[β=0, β=max_beta].
The choice n_beta=1 corresponds to a small 3x3 kernel in conv2d since in both cases the resulting kernel consists of one central pixel and one ring around the center.
Gives the number of learned parameters of the rings around the pole.
These values are per default equally spaced on the azimuth.
A sensible number of values depends on the bandwidth and max_beta since a higher resolution or spatial extent allows to sample more fine kernels without producing aliased results.
In practice this value is typically set to a constant, low value like 6 or 8.
A reduced bandwidth of the signal is thereby counteracted by an increased max_beta to emulate spatial pooling.
The so3_near_identity_grid has two additional parameters max_gamma and n_gamma.
SO(3) can be seen as a (principal) fiber bundle SO(3)→S² with the sphere S² as base space and fiber SO(2) attached to each point.
The additional parameters control the grid on the fiber in the following way:
The kernel spans over the fiber SO(2) between γ∈[0, max_gamma].
The fiber SO(2) encodes the kernel responses for every sampled orientation at a given position on the sphere.
Setting max_gamma≨2π results in the kernel not seeing the responses of all kernel orientations simultaneously and is in general unfavored.
Steerable CNNs [3] usually always use max_gamma=2π.
Number of learned parameters on the fiber.
Typically set equal to n_alpha, i.e. to a low value like 6 or 8.
See the deep model of the MNIST example for an example of how to adapt these parameters over layers.
[1] Taco S. Cohen, Mario Geiger, Jonas Köhler, Max Welling, Spherical CNNs. International Conference on Learning Representations (ICLR), 2018.
[2] Taco S. Cohen, Mario Geiger, Jonas Köhler, Max Welling, Convolutional Networks for Spherical Signals. ICML Workshop on Principled Approaches to Deep Learning, 2017.
[3] Taco S. Cohen, Mario Geiger, Maurice Weiler, Intertwiners between Induced Representations (with applications to the theory of equivariant neural networks), ArXiv preprint 1803.10743, 2018.