Output Contract

This specification describes public BenchmarkRunner exports. It is distinct from the Python dataclass schema and the paper's frozen research archive. The current contract below is included directly from configs/contracts/public_output_contract.json during the docs build; tests/unit/test_output_contract.py checks that it equals lrdbench.output_contract.PUBLIC_OUTPUT_CONTRACT.

Run root and file requirements

Runs write under <report.export_root>/<run_id>/. Required files and minimum CSV headers, conditional files, dynamic columns and the contract version are defined in the embedded source. Extra columns are allowed; readers must not assume an exact column count or a fixed set of manifest-dependent metric/stratum columns.

{
  "contract_version": "1.1.0",
  "run_root": "<report.export_root>/<run_id>",
  "required_files": {
    "summary": [
      "tables/run_summary.csv",
      "tables/per_stratum_metrics.csv",
      "tables/leaderboard.csv",
      "tables/estimator_metadata.csv",
      "tables/failures.csv",
      "tables/failure_map.csv",
      "tables/uncertainty_calibration.csv",
      "tables/benchmark_uncertainty.csv",
      "tables/estimator_disagreement.csv",
      "tables/scale_window_sensitivity.csv",
      "html/report.html",
      "manifest/environment.json",
      "artefacts/artefact_index.csv"
    ],
    "raw_result_store": [
      "raw/records.csv",
      "raw/estimates.csv",
      "raw/metrics.csv",
      "raw/artefacts.csv"
    ]
  },
  "conditional_files": {
    "tables/stress_metrics.csv": "present for stress_test reports",
    "tables/correction_metrics.csv": "present when correction/preprocessing metrics are requested",
    "raw/leaderboards.csv": "present when leaderboard rows are generated",
    "raw/truths.csv": "present when any record declares ground truth (all synthetic modes)",
    "manifest/benchmark_manifest.yaml": "present when the manifest was loaded from YAML",
    "figures/*.png": "present when requested report.figure_set entries have data",
    "latex/*.tex": "present when latex report output is requested"
  },
  "required_columns": {
    "tables/run_summary.csv": ["run_id", "manifest_id", "benchmark_name", "mode"],
    "tables/per_stratum_metrics.csv": ["estimator_name", "metric_name", "value", "stratum_json", "metadata_json"],
    "tables/leaderboard.csv": ["estimator_name", "rank", "score"],
    "tables/estimator_metadata.csv": ["estimator_name", "family", "target_estimand", "supports_ci", "supports_diagnostics", "assumptions_json", "params_json", "version"],
    "tables/failures.csv": ["estimator_name", "stratum_json", "n_metric_rows", "n_missing_values", "n_missing_uncertainty", "n_validity_observations", "n_invalid_estimates", "invalid_rate", "missing_value_rate"],
    "tables/failure_map.csv": ["estimator_name", "stratum_json"],
    "tables/uncertainty_calibration.csv": ["estimator_name", "metric_name", "value", "nominal"],
    "tables/benchmark_uncertainty.csv": ["estimator_name", "metric_name", "value", "uncertainty_type", "aggregation", "nominal", "ci_low", "ci_high", "stratum_json", "metadata_json"],
    "tables/estimator_disagreement.csv": ["scope", "record_id", "estimator_name", "metric_name", "value", "stratum_json", "metadata_json"],
    "tables/scale_window_sensitivity.csv": ["scope", "record_id", "estimator_name", "metric_name", "value", "stratum_json", "metadata_json"],
    "tables/stress_metrics.csv": ["record_id", "estimator_name", "metric_name", "value", "stratum_json", "contamination_operator", "clean_record_id", "nominal"],
    "tables/correction_metrics.csv": ["record_id", "estimator_name", "metric_name", "value", "stratum_json", "preprocessing_operator", "preprocessing_kind", "correction_target", "raw_record_id", "oracle_record_id", "oracle_operator"],
    "artefacts/artefact_index.csv": ["artefact_id", "run_id", "artefact_type", "format", "path", "hash", "created_at", "depends_on_json"],
    "raw/records.csv": ["record_id", "source_type", "source_name", "n", "values_path", "truth_family", "target_estimand", "target_value", "annotations_json"],
    "raw/estimates.csv": ["record_id", "estimator_name", "point", "ci_low", "ci_high", "stderr", "valid", "runtime_seconds", "failure_reason", "warnings", "bootstrap_cis_json"],
    "raw/metrics.csv": ["scope", "record_id", "estimator_name", "metric_name", "value", "stratum_json", "metadata_json"],
    "raw/leaderboards.csv": ["estimator_name", "rank", "score", "components_json"],
    "raw/truths.csv": ["record_id", "target_estimand", "target_value", "process_family", "is_primary", "notes"],
    "raw/artefacts.csv": ["artefact_id", "run_id", "artefact_type", "format", "path", "hash", "created_at", "depends_on_json"]
  },
  "dynamic_columns": {
    "tables/leaderboard.csv": "metric__<component_metric> columns are added per leaderboard",
    "tables/failure_map.csv": "stratum__<key> and metric__<metric_name> columns are data dependent",
    "tables/failures.csv": "stratum__<key> columns are data dependent"
  }
}

Contract 1.1.0 added the conditional raw/truths.csv companion-truth ledger while retaining the 1.0.0 columns in existing files. It stores primary and companion estimands on the same realisation, with is_primary identifying the primary truth. raw/records.csv retains only that primary truth. Conditional correction and stress tables carry paired-record identifiers.

The manifest snapshot is written whenever BenchmarkManifest.raw_yaml is available. This includes both YAML loading and manifest_from_mapping(); a directly constructed dataclass with raw_yaml=None does not produce that snapshot.

Persistence boundaries

The public store also writes run_summary.json, signal arrays under raw/values/<record_id>.npy, and raw/plugin_provenance.csv when plugin provenance rows exist. raw/records.csv points to each array through values_path. Preserve or relocate referenced files when moving a report; the CSVs alone are not a self-contained signal archive.

raw/estimates.csv retains points, endpoint fields, standard errors, validity, runtime, failure reasons, warnings and labelled bootstrap intervals. It does not persist the full EstimateResult.diagnostics mapping or bootstrap draw arrays. Use the returned Python objects when those fields are needed; a public CSV export cannot reconstruct the entire in-memory run.

The confirmation archive has separate point/draw storage, completion evidence, paired resampling and hash checks. Its files are not interchangeable with these public exports. Follow its protocol and audit instructions for paper reproduction.

Validation command and limits

lrdbench validate-output reports/public_small/<run_id>

The command checks required-file presence and minimum CSV headers, including headers of conditional CSVs that exist. It returns 0 when those checks pass, or 2 with an error=... line per violation. It does not determine whether a conditional file should exist, verify a stored contract version, inspect row values/counts, follow array paths, verify hashes, or establish scientific validity. A passing check is structural validation, not a completeness audit.

Artefact inventories

artefacts/artefact_index.csv inventories report artefacts known to the reporter, including metric tables, figures, environment metadata, HTML and the index itself. Its hash and dependency metadata are optional; their presence as columns does not mean all files have been hashed.

raw/artefacts.csv records the returned report bundle and any model artefacts appended by the runner after report generation. Consequently it can contain model files absent from the reporter index. Neither inventory enumerates every signal array or every raw CSV in the directory.

Maintaining the specification

Change the Python contract and tracked JSON together when changing a stable requirement; the existing equality test guards against divergence. Review the contract version and migration notes for compatibility changes. The docs include the JSON directly, so required paths and headers should not be copied into a second manually maintained schema table.

For workflow or serialization changes, run relevant runner/reporter tests, generate a smoke report and validate it, then build the documentation with python -m mkdocs build --strict. Check conditional outputs and persistence semantics separately from the minimum-header check.