Deep Learning Architecture

This project is inspired by Fighting CV's project. Also, other references are included to get code insights.

Other references:

• x-transformers
• xFormers
• OpenMixup
• Awesome-Vision-Attentions
• vit-pytorch
• Networks-Beyond-Attention
• Efficient-AI-Backbones

# code test env.
python == 3.10.8
pytorch == 1.12.1

---

Contents

• Kolmogorov-Arnold Models
• State Space Models
• Spatial Attentions
• Channel Attentions
• MLPs
• Convolutions
• Spectral Features
• Graph
• Hybrid
• Spatio-Temporal (ST)
• Activations
• Patch Embedding
• Branch Scaling
• Normalization
• Backbones

---

Kolmogorov-Arnold Models

• Kolmogorov-Arnold Transformer (KAT, 2024) --- (pytorch_v1)

State Space Models

• CSM (VMamba, 2024) --- (pytorch_v1)(graph)
• Bidirectional Mamba (Vim, 2024) --- (pytorch_v1)(graph)
• ConvSSM (ConvS5, 2023) --- (pytorch_v1)(graph)
• Selective SSMs (Mamba, 2023) --- (pytorch_v1)(graph)

Spatial Attentions

• HiLo Attention (LITv2, 2022) --- (pytorch_v1)(graph)
• Pay Less Attention (LITv1, 2022) --- (pytorch_v1)(graph)
• External Attention (EANet, 2021) --- (pytorch_v1)(graph)
• Non-local Multi-head Self-Attention (Transformer, 2017) --- (pytorch_v1)(graph)

Channel Attentions

• Frequency Channel Attention (FcaNet, 2021) --- (pytorch_v1)(graph)
• Global Context Attention (GCNet, 2019) --- (pytorch_v1)(graph)
• Efficient Channel Attention (ECA-Net, 2019) --- (pytorch_v1)(graph)
• Squeeze-and-Excitation (SE-Net, 2018) --- (pytorch_v1)(graph)

MLPs

• Mixing and Shifting for global and local (MS-MLP, 2022) --- (pytorch_v1)(graph)
• Phase-Aware Token Mixing (Wave-MLP, 2022) --- (pytorch_v1)(graph)
• An all-MLP Architecture for Vision (MLP-Mixer, 2021) --- (pytorch_v1)(graph)

Convolutions

• GatedCNNBlock (MambaOut, 2024) --- (pytorch_v1)(graph)
• Efficient Modulation (EfficientMod. 2024) --- (pytorch_v1)
• SMT (Scale-Aware Modulation Meet Transformer, 2023) --- (pytorch_v1)
• InceptionDWConv2d (InceptionNeXt, 2023) --- (pytorch_v1)(graph)
• ParC operator (ParC-Net, 2022) --- (pytorch_v1)(graph)
• Dynamic DWConv (DWNet, 2022) --- (pytorch_v1)(graph)
• Focal Module (FocalNets, 2022) --- (pytorch_v1)(graph)
• gnConv (HorNet, 2022) --- (pytorch_v1)(graph)
• MSCA (SegNeXt, 2022) --- (pytorch_v1)(graph)
• LKA (VAN, 2022) --- (pytorch_v1)(graph)
• Pooling (MetaFormer, 2022) --- (pytorch_v1)(graph)
• RandomMixing (MetaFormer, 2022) --- (pytorch_v1)(graph)
• Inverted SepConv (MobileNetV2, 2018) --- (pytorch_v1)(graph)
• Deformable Conv (Deformable ConvNet, 2017) --- (pytorch_v1)

Spectral Features

• Dynamic Filter (DFFormer, 2024) --- (pytorch_v1)
• Global Filter (GFNet, 2021) --- (pytorch_v1)(graph)
• Fourier Mixer (FNet, 2021) --- (pytorch_v1)(graph)
• Fourier_test --- (pytorch_v1)
• img_FFT --- (.py)(.ipynb)(2D FFT)

Graph

• Mobile ViG (SVGA, 2023) --- (pytorch_v1)(graph)
• Vision GNN (ViG, 2022) --- (pytorch_v1)(graph)
• GINconv2d (GIN, 2019) --- (pytorch_v1)(graph)
• DeepGCNs (MR-GCN, 2019) --- (pytorch_v1)(graph)

Hybrid

• Inception Mixer (iFormer, 2022) --- (pytorch_v1)(graph)
• MHRA (Uniformer v1, 2022) --- (pytorch_v1)(graph)

Spatio-Temporal (ST)

• MHRA (Uniformer v2, 2022) --- (pytorch_v1)(graph)(mmaction2)
• MHRA (Uniformer v1, 2022) --- (pytorch_v1)(graph)(mmaction2)

Activations

• StarReLU (2022) --- (pytorch_v1)
• SquaredReLU (2021) --- (pytorch_v1)
• Swish (2017) --- (pytorch_v1)
• GELU (2016) --- (pytorch_v1)(scratch_code)(quick_version)
• ReLU (2010) --- (pytorch_v1)
• Softsign (2009) --- (pytorch_v1)
• Zero-centered s-shaped functions --- (pytorch_v1)
• S-shape activation function comparision --- (pytorch_v1)

Patch Embedding

• patch embed --- (pytorch_v1)

Branch Scaling

• Layer-/Residual-branch scaling (LayerScale 2021, ResScale 2021) --- (pytorch_v1)(timm)

Normalization

• GroupNorm
• LayerNorm (2016) --- (pytorch_v1)
• BatchNorm (2015) --- (pytorch_v1)

Backbones

• Efficient Modulation Network (2024) --- (pytorch_v1)
• VanillaNet (2023) --- (pytorch_v1)
• InceptionNeXt; MetaNeXt (2023) --- (pytorch_v1)
• MetaFormer baseline (2022) --- (pytorch_v1)
• PoolFormer (2022) --- (pytorch_v1)