memorax.networks.positional_embeddings

Positional embedding schemes for attention-based models.

RoPE - Rotary Position Embeddings.

ALiBi - Attention with Linear Biases.

LearnablePositionalEmbedding - Learned embeddings per position.