Long-Context-Retrieval-Risk-Dataset-Open-Weight-LLMs

This dataset simulates how retrieval and QA accuracy vary with the position of relevant information inside the context window and with how far context length is pushed past a model's native trained length, for 168 configs spanning seven approximate open-weight model configs, four context-extension methods (none, Position Interpolation, YaRN, and the training-free Self-Extend method), and whether the serving stack preserves StreamingLLM-style attention-sink tokens. Retrieval accuracy combines a symmetric U-shaped position-in-context penalty calibrated to the shape reported in Lost in the Middle (2023), an attention-sink boost, and a method-specific exponential decay in extension ratio calibrated to each cited method's reported relative degradation. All ten grounding sources, spanning 2021 to 2024, and every equation are documented in long_context_theory.py and the README Research basis. Simulated model configs reflect 2024-2025 open-weight releases (Gemma 2, Llama 3.1, Mistral); the position-encoding and context-extension mechanisms grounding the equations trace back to RoPE in 2021.