Download
DeepGEMM is a specialized CUDA library for efficient, high-performance general matrix multiplication (GEMM) operations, with particular focus on low-precision formats such as FP8 (and experimental support for BF16). The library is designed to work cleanly and simply, avoiding overly templated or heavily abstracted code, while still delivering performance that rivals expert-tuned libraries. It supports both standard and “grouped” GEMMs, which is useful for architectures like Mixture of Experts (MoE) that require segmented matrix multiplications. One distinguishing aspect is that DeepGEMM compiles its kernels at runtime (via a lightweight Just-In-Time (JIT) module), so users don’t need to precompile CUDA kernels before installation. Despite its lean design, it includes scaling strategies (fine-grained scaling) and optimizations inspired by cutting edge systems (drawing from ideas in CUTLASS, CuTe) but in a more streamlined form.
Features
- High-performance GEMM kernels focused on FP8 precision, with optional BF16 support
- Support for grouped GEMM (segmented matrix operations) useful for MoE scenarios
- Runtime JIT compilation of kernels (no heavy ahead-of-time kernel compilation needed)
- Clean, modular code structure (less dependence on heavy template programming)
- Fine-grained scaling strategies (to adapt precision dynamically)
- Benchmark and test suite (e.g. test_fp8.py), performance monitoring, and ongoing issue tracking
Project Samples
