KV-Cache-Serving-Risk-Dataset-Open-Weight-LLMs

Simulated memory footprint, decode latency, and OOM early-warning labels for 210 open-weight LLM serving deployments (Gemma, Gemma 2, Llama 3, Mistral) swept across GPU, precision, batch size, and context length, grounded in published KV-cache, GQA/MQA, and memory-bandwidth roofline equations. Forward-looking task predicts 4x-context OOM risk from current-context features only.

About Dataset

This dataset simulates how GPU memory and decode-step latency scale for realistic open-weight LLM serving deployments. Each deployment fixes a model architecture (approximate published configs for Gemma 2B/7B, Gemma 2 9B/27B, Llama 3 8B/70B, and Mistral 7B), a GPU (T4 through H100-80GB, public datasheet memory and bandwidth), a weight precision (fp16/int8/int4), and a KV-cache allocator strategy (naive contiguous vs. paged), then sweeps batch size (1-64) and context length (512-131072 tokens). Every number is computed from deterministic formulas cited to ten sources: Attention Is All You Need, Scaling Laws for Neural Language Models, Chinchilla, MQA, GQA, FlashAttention, PagedAttention/vLLM, and the Gemma, Gemma 2, and Llama 3 technical reports. No real cluster telemetry is used - every value is reproducible from public model configs and public GPU specs.

Purpose of Dataset

Lets Ml Infra And Platform Engineers Estimate, Before Deployment, Whether A Given Open-weight Model Plus Gpu Plus Precision Plus Context-length Combination Will Run Out Of Memory Or Degrade Below An Interactive-latency Sla, Including An Early-warning Signal For Whether Growing Context 4x Will Push A Currently-working Deployment Into Oom.