Research artefact
random-moe-pruning: dataset-free expert pruning for Mixture-of-Experts checkpoints
A Valendra research artefact for shrinking Mixture-of-Experts checkpoints through expert pruning, without a calibration dataset and reproducible on CPU. Current release validated on GPT-OSS.
random-moe-pruning is a Valendra research artefact for applying expert pruning to Mixture-of-Experts checkpoints hosted on the Hugging Face Hub.
The key idea is simple: drop the experts that the router is least likely to use, without requiring a calibration dataset, GPU inference, or retraining.
The current release has been validated on the GPT-OSS family, but the approach is designed to work with any MoE architecture.
What it does
The pruning procedure walks the MoE layers of a checkpoint, scores each expert based on how the router responds to random Gaussian probes, and rewrites the safetensors shards while keeping only the experts that the router would be most likely to use.
The result is a smaller derived checkpoint that keeps the same general architecture, tokenizer, and chat template as the base model—and loads directly through the standard transformers API.
Because the procedure only needs the router weights, the pre-MoE RMSNorm scale, and the expert tensors themselves, it can run on a standard machine without a GPU and without a calibration dataset.
Published checkpoints
The experimental checkpoints published from this work are:
These are pruned GPT-OSS variants intended as reproducible artefacts for studying the effect of reducing the number of experts per layer across different model sizes. They are not intended to replace the base checkpoints or to be presented as production-ready models.
Why it matters
Expert pruning gives teams a practical way to explore the trade-off between model size, memory footprint, and inference cost without retraining the full model. In real AI deployments, infrastructure decisions, model operations, and cost control have to be reconciled together. This work was built as a small, auditable implementation that can be integrated into pipelines already working with Hugging Face checkpoints. The focus is deliberately narrow: make expert pruning easy to reproduce, easy to inspect, and easy to run locally.
Limitations
This work is a research artefact, not a production recipe. The quality of each pruned checkpoint should be evaluated case by case against its base model. The current release has been validated only on the GPT-OSS family. The approach is designed to work with any Mixture-of-Experts architecture, but support beyond GPT-OSS should not be assumed until explicitly tested and documented.
Next steps
The most relevant lines of work are:
- Benchmarking the published checkpoints. Measuring quality, latency, and memory against the base models.
- Hardening GPT-OSS support. Consolidating the current compatibility before expanding support to other MoE families.
- Exploring other pruning strategies. Investigating whether alternative scoring methods or heuristics improve pruned checkpoint quality.
- Extending to other MoE families. Once GPT-OSS support is stable, adapting the approach to other Mixture-of-Experts architectures.