varmology

• The RoPE Compatibility Problem in DeepSeek's Multi Head Latent Attention
  April 27, 2025

  A deep dive into why naively combining Rotary Position Embeddings with compressed KV caches fails mathematically, and how DeepSeek's decoupled RoPE strategy elegantly solves the non-commutativity problem by separating content and position paths.

• Analysis of Matrix Multiplications in Transformer Architectures
  January 1, 2025

  Performance analysis of transformer computations on H100 architecture, with roofline model examination and optimization strategies.

Less Readworthy Posts

• Balancing Memory & Compute: Strategies to Manage KV Cache in LLMs
  May 27, 2024

  Strategies to reduce KV cache memory in LLMs.

• Layer Normalization as a Projection: The Complete Geometric Interpretation
  May 17, 2025

  Layer normalization as geometric projections.