Dhir-5B-4bit

Dhīr-5B-4bit is the 4-bit quantized release of Dhīr 5B-A1.7B , pretrained from scratch in India on approximately 6.13 trillion tokens. Architecture and Efficiency Dhīr 5B uses a decoder-only MoE Transformer with 12 experts and top-2 routing , activating approximately 1.79 billion parameters per token out of 5.09 billion total. The model supports English and ten Indian languages , with a context length of 16,384 tokens . 4-bit Quantization and Offline Deployment At just 3.4 GB , Dhīr-5B-4bit is designed for private, offline inference on compatible laptops, desktops, mobile devices and embedded hardware using llama.cpp. The model is distributed in GGUF Q4_K_M format . Its weights were quantized directly from the released BF16 model, without additional training or fine-tuning. ZenteiQ's reported CPU benchmarks also demonstrate faster decoding than Dhīr 2B on the tested hardware, although actual performance depends on the device and inference configuration. Quantization Fidelity ZenteiQ evaluated Dhīr-5B-4bit against the original BF16 model using WikiText-2 perplexity, KL divergence of next-token probability distributions, next-token prediction agreement and standard benchmarks. The evaluation covered approximately 1.5 million tokens across 11 languages, code and mathematics . Key evaluation results: Within 1.8% relative perplexity difference compared with the original BF16 model on WikiText-2. 91% top-1 next-token agreement with the BF16 model. Evaluation across eleven languages, code and mathematics, including comparisons of next-token distributions using KL divergence. No additional training following quantization. For the full evaluation, refer to ZenteiQ's technical paper and research updates: https://zenteiq.ai/research/blog/ Choose a Dhīr Model ModelSizeRecommended Deployment Dhīr-2B-4bit 1.3 GB Edge devices, mobiles and embedded chips with limited memory Dhīr-2B-8bit 2.2 GB Laptops and mobiles; closest to the original 2B model Dhīr-5B-A1.7B-4bit 3.4 GB Laptops and desktops; higher-capacity MoE model with efficient CPU inference Dhīr-5B-A1.7B-8bit 5.4 GB Laptops, desktops and small GPUs; closest to the original 5B model Deployment with ZenteiQ's llama.cpp ZenteiQ's llama.cpp repository is a Dhīr-compatible fork of the open-source llama.cpp inference engine. The repository contains the Dhīr-specific architecture support required to load and run Dhīr models in GGUF format, including the custom Mixture-of-Experts architecture used by Dhīr 5B. It provides: Offline local inference on compatible CPUs and GPUs. Command-line execution using llama-cli . OpenAI-compatible HTTP API serving using llama-server . Application integration through frameworks such as LangChain. Deployment support for compatible laptops, desktops and edge devices. The repository provides inference software; model weights are distributed separately through ZenteiQ's model repositories. Until Dhīr support is merged into upstream llama.cpp, users should build the dedicated dhir-support branch . Repository: https://github.com/zenteiq-research/llama.cpp The README provides installation, compilation, inference, CPU optimization and Python integration commands. Model Specifications Base model: Dhīr 5B-A1.7B Total parameters: 5.09 billion Active parameters: Approximately 1.79 billion per token Architecture: Decoder-only Mixture-of-Experts Transformer, 12 experts, top-2 routing Quantization: Q4_K_M (4-bit) Model format: GGUF Model file: Dhir-5B-A1.7B-Q4_K_M.gguf Model size: 3.4 GB Context length: 16,384 tokens Languages: English and ten Indian languages License: Apache License 2.0, the same as the original Dhīr 5B model Developed by ZenteiQ AiTech Innovations Pvt. Ltd. https://zenteiq.ai/dhir/