memorax.networks.blocks

Building blocks for constructing network architectures.

Feed-Forward

FFN - Feed-forward network block with expansion.

Normalization

PreNorm - Pre-normalization wrapper.

PostNorm - Post-normalization wrapper.

Residual

Residual - Residual connection wrapper.

Composition

Stack - Stacks multiple blocks sequentially.

Mixture of Experts

MoE - Mixture of Experts layer.

TopKRouter - Top-K routing for MoE.