SEBI Circulars Dataset

About Dataset

Schema Details corpus Columns: circular_number, issue_date, effective_date, subject, issuing_department, supersession_status, version_lineage, source_url, text, excerpt, extraction_date, circular_type, validity_status, superseded_by_id, supersession_edges circular_number (str): Unique identifier (e.g., SEBI/HO/CFD/P/CIR/2023/123). issue_date (date): Publication date. effective_date (date, nullable): When the circular takes effect. subject (str): Circular title/summary. issuing_department (str): Issuing SEBI department (e.g., CFD, MRD). Known limitation: 158/705 records have issuing_department=UNKNOWN due to pre-existing parsing artifacts. supersession_status (str): in_force, superseded, or amended. version_lineage (list[str]): Prior circular numbers this updates/references. source_url (str): Original SEBI publication page. text (str): Full circular text. excerpt (bool): Whether the text is a partial excerpt. extraction_date (date): When this record was extracted from source. Known data-quality caveat: Some master-circular subject fields capture body text (~2900 chars) due to a pre-existing PDF parsing artifact in src/sebi_rag/ingest_pdf.py. This is not a regression from this work; document it in your analysis. chunks Columns: chunk_id, doc_id, section, context_header, text, circular_number, issue_date, effective_date, subject, issuing_department, supersession_status, version_lineage, circular_type, validity_status, superseded_by_id 77,859 section-aware retrieval chunks derived from corpus text, one row per chunk. chunk_id (str): Unique chunk identifier (e.g., SEBI/HO/CFD/P/CIR/2023/123#preamble#0). doc_id (str): Parent circular number. section (str): Section path (e.g., SEBI/HO/CFD/.../preamble/p0). context_header (str): Repeated contextual header extracted from chunk text (circular_number | subject | section). text (str): Chunk text body (header removed for clarity). Flattened metadata: circular_number, issue_date, effective_date, subject, issuing_department, supersession_status, version_lineage. lineage Columns: source_circular, relation, target_circular, source_issue_date, target_in_corpus 4,483 regulatory supersession/amendment edges (forward-direction only). source_circular (str): Circular that supersedes/amends another. relation (str): supersedes or amends. target_circular (str): Older circular being superseded/amended. source_issue_date (date): Publication date of source (for temporal reasoning). target_in_corpus (bool): Whether the target circular is in this corpus (allows filtering for pair-classification tasks). Note: Inverse relationships (superseded_by, amended_by) are omitted to avoid duplication; regenerate them at query time. citation-normalization Columns: 8,802 in-text reference citations mined and normalized. raw_reference (str): Raw citation text as it appears in the circular (e.g., CIR/CFD/CMD/4/2015). normalized_circular_number (str): Canonical form (lowercase, standardized). context_window (str): Surrounding text (~60 characters on each side, whitespace collapsed). source_doc_id (str): Circular containing the reference. format_family (str): Reference format category: new-standard: SEBI/HO/DEPT/P/CIR/YYYY/NNN (post-2015 format). old-standard: CIR/DEPT/YYYY/NNN (legacy format). dept-order-2026: HO/(NN)YYYY-DEPT (departmental order format, 2026). Task: Seq2seq/NER: predict normalized circular number from raw reference; or use as training set for reference extraction/normalization models. supersession-pairs Columns: 2,769 labeled circular pairs: positives from lineage, negatives sampled same-department (2:1 ratio). circular_a_number (str): First circular. circular_a_subject (str): Subject of circular A. circular_b_number (str): Second circular. circular_b_subject (str): Subject of circular B. label (str): supersedes, amends, or unrelated. Task: Pair classification: does circular A supersede/amend circular B? Positives from lineage edges (both endpoints in corpus); negatives sampled deterministically (seed=42) from same-department non-linked pairs.