Documentation

Tested-model pipeline

ConfigIQ's tested-model system turns measured inference results into configuration-specific performance-envelope classifiers. The pipeline and its source code live in this repository; configiq-tested-models is no longer the production pipeline.

Runtime contract

The pipeline produces one sanitized bundle containing:

registry.json                 Webapp-facing pair and classifier metadata
dashboard/<pair-id>.json     Sanitized dashboard data
triton/<pair-id>/             Triton FIL model repositories
  config.pbtxt
  1/xgboost.json

The ConfigIQ container reads registry.json from TESTED_MODELS_DIR, which is /tested-models by default. The deployment repository mounts the host bundle at that path and mounts its triton/ subdirectory at /models for Triton. The webapp endpoints are GET /api/tested-models, GET /api/tested-models/dashboard/<pair-id>, and POST /api/tested-models/classify. The human-facing pages are /tested-models and /tested-models/<pair-id>.

Registry schema v2

registry.json separates three concepts that must never be conflated:

  • validated — a pair has a trained classifier whose held-out test measurements clear the validation gate. Only validated pairs can emit a classifier verdict.
  • measured_no_envelope — the pair has published measurements and dashboard graphs, but no classifier passes the gate. The graph stays available; the sizing pages show "measured, no validated envelope" and emit no verdict.
  • approved_no_data — the model appears in approvedModels but has no pair in the registry. It is approved for measurement, not classifier-tested.

The validation gate requires at least 100 records, at least five concurrency levels, test ROC-AUC ≥ 0.70, and test F1 ≥ 0.30. A failing pair retains its measurements and graphs and records diagnostic reasons; it exposes no classifier artifacts. The gate is published in the registry and returned by the API so the UI and pipeline cannot drift.

Schema v1 registries remain readable during rollout. A v1 pair is treated as validated only when its recorded test AUC and F1 clear the metric thresholds; v1 carries no coverage summary, so it cannot claim the full gate.

Coverage and feature categories

A v2 pair's coverage object records records, concurrencyLevels, concurrencyLevelCount, the raw backendVersions for provenance, and the measured backends (runtime families such as vllm or sglang). The classify endpoint refuses to score a request whose backend family is not in that list: it returns no_measurements with a reason instead of applying an envelope trained on a different runtime. Bundles that predate backends are treated as unknown and stay permissive.

The classifier's version feature is a backend-scoped major.minor family, not a raw label. Training rows (vLLM-0.24.0) and runtime catalog versions (0.24.0) both normalize to vLLM-0.24; sglang-0.5.18-recipe-latency-oriented normalizes to sglang-0.5. Pooling keeps a category dense enough to validate while preventing families from different runtimes from colliding. Unrecognised or unresolved labels (current, arbitrary build strings) become unknown.

ConfigIQ's public/config.json names approved models in approvedModels. The pipeline accepts the retired testedModels name for older deployments, but the frontend config no longer exposes it. The config list governs measurement coverage; it is not evidence that a model has been classifier-tested. Tested status comes from registry.json alone.

The deployment-owned k2_horizon vLLM model is separate from the generated FIL bundle. It shares the Triton server but is not generated by these scripts. The deployment-owned qwen3_8b model is a fallback and is normally unloaded.

Data boundary

Ground-truth extraction runs only from an approved environment. The legacy API extractor receives its source endpoint through PERF_DATA_API_URL; the GitHub publisher reads the approved S3 aggregate with an OIDC read-only role. Neither source location is written to published artifacts. Extraction removes run IDs, UUIDs, MLflow identifiers, timestamps, raw runtime arguments, and other administrative fields before publication. The S3 importer includes only models in public/config.json's approvedModels list, and only rows whose version is a plain upstream vLLM release (vLLM-<major>.<minor>.<patch>). Productized Red Hat AI Inference Server builds and pre-release labels are dropped, and the import manifest records the kept and dropped counts.

The scheduled GitHub Actions workflow consumes the public redhat-performance/configiq-performance-upstream-data Hugging Face dataset. It uploads only the sanitized registry/Triton/dashboard bundle as a workflow artifact. Raw parquet files, source identifiers, and training work files are not published by that workflow.

By default the workflow follows the dataset's default branch. The download step resolves that mutable ref once to a full 40-character commit SHA, downloads only that SHA, and records it in dataset-manifest.json; registry.json records the same repository and revision under dataset so a deployed bundle can be traced back to immutable input. Pass a commit SHA, branch, or tag to pin a build explicitly (--revision on the script, or the revision input on the tested-models.yml workflow dispatch).

That dataset is the upstream-only subset: it retains rows whose serving stack is a plain upstream vLLM release and excludes productized Red Hat AI Inference Server builds. It grows as new measurements are published, so a model can be approved for testing without having a trained performance envelope yet. Do not read absence from this dataset as "not tested".

registry.json lists every measured pair in this build, with its validation status. It is a catalog of measured coverage, not the approved-model list. A model can be approved for testing with no pair, and a pair can be measured without having a validated envelope. Neither case is classifier-tested.

The workflow reads the repository variable TESTED_DATASET_REPO when it is set, and otherwise falls back to the dataset above. Set it to move the pipeline to a renamed or republished dataset without a code change.

The former TESTED_DATASET_REVISION repository variable is no longer read. Builds follow the dataset automatically; use the workflow dispatch revision input when a specific revision must be rebuilt.

Production workflow

1. Publish ground truth

The GitHub workflow .github/workflows/publish-performance-data.yml downloads the aggregate, prepares sanitized Parquet pairs, and compares their content with the public Hugging Face dataset. It runs at 00:00 Monday in America/New_York on the default branch, or manually through workflow_dispatch. Scheduled publishing requires the GitHub repository variable PERFORMANCE_DATA_PUBLISH_ENABLED=true; otherwise scheduled runs only prepare and compare. A manual run on main publishes only when its publish input is selected. Unchanged content is never uploaded.

When an upload succeeds, the publish workflow dispatches tested-models.yml with the new dataset commit SHA, so new ground truth retrains the classifiers without a manual revision bump. The build workflow also runs on its Monday 09:17 UTC fallback schedule and on relevant code changes. It appends a Markdown comparison of the new registry against the deployed registry (when https://configiq.dev/api/tested-models is reachable) to the workflow job summary, including added/removed pairs and validated-to-measured transitions. That report is advisory: it never fails the build or installs the bundle.

Preparation applies the upstream-only version filter and refuses to publish when the prepared set would delete pairs that exist only in the remote dataset. Run the complete extraction if pairs are missing; pass --allow-removal only when removing pairs is intentional. It also rejects distinct model/accelerator labels that normalize to the same Parquet pair ID before writing either file. The publisher checks upstream version and backend fields again before upload.

Hugging Face publishing uses a Trusted Publisher on the dataset repository, restricted to redhat-performance/configiq, branch main, and workflow publish-performance-data.yml. No persistent Hugging Face token is stored in GitHub. The AWS read-only role must trust the repository's OIDC subject for main. If TESTED_DATASET_REPO points at another dataset repository, configure an equivalent Trusted Publisher there; the workflow derives HF_OIDC_RESOURCE from that same variable so publishing and dispatched builds target one repository.

The earlier API extraction path remains available from an approved data environment:

From the approved data environment, extract and publish a sanitized dataset:

uv sync --group tested-models
export PERF_DATA_API_URL="https://<approved-data-source>"
uv run --group tested-models python scripts/tested_models/discover_data.py --accelerator H200
uv run --group tested-models python scripts/tested_models/extract_datasets.py
uv run --group tested-models python scripts/tested_models/publish_ground_truth.py

The publish workflow dispatches the tested-model build automatically with the new Hugging Face commit SHA, so no revision variable needs updating. To rebuild a specific published revision, dispatch tested-models.yml with its revision input. If the data is published to a repository other than the default one above, set the TESTED_DATASET_REPO repository variable as well. Do not commit the dataset or its source credentials.

Refresh the private InferenceX comparison dataset

The same Monday 00:00 America/New_York workflow has an independent refresh-inferencex job. It collects published InferenceX single-turn 8k/1k measurements for models in public/config.json's approvedModels list and updates redhat-performance/configiq-inferencex-comparison only when the measurements change. The publisher refuses to upload if that dataset is public. The InferenceX job does not use the S3 role.

Scheduled InferenceX publishing requires the repository variable INFERENCEX_PUBLISH_ENABLED=true. On a manual run from main, select the separate publish_inferencex input to refresh the private dataset. When it is the only publish input selected, the S3 job is skipped. The InferenceX job uses a write-enabled Hugging Face Trusted Publisher restricted to redhat-performance/configiq, refs/heads/main, and .github/workflows/publish-performance-data.yml; it does not store a long-lived write token in GitHub. The deployed comparison app uses its separate server-side read token to load the private dataset.

2. Build and validate the bundle

The repeatable build is .github/workflows/tested-models.yml. It runs the regression tests, downloads the current dataset revision, creates grouped leakage-safe splits, fits labels from training rows, trains XGBoost classifiers, exports Triton FIL repositories, builds dashboard payloads for every measured pair, and validates the registry. build_dashboard.py must run before build_registry.py, because the registry requires a dashboard payload for each pair it publishes.

Run the same sequence locally when debugging:

uv run --group tested-models pytest scripts/tested_models/tests -q
uv run --group tested-models python scripts/tested_models/download_ground_truth.py
uv run --group tested-models python scripts/tested_models/split_dataset.py
uv run --group tested-models python scripts/tested_models/label_data.py
uv run --group tested-models python scripts/tested_models/train_classifiers.py
uv run --group tested-models python scripts/tested_models/export_triton.py
uv run --group tested-models python scripts/tested_models/build_dashboard.py
uv run --group tested-models python scripts/tested_models/build_registry.py
uv run --group tested-models python scripts/tested_models/validate_artifacts.py

The generated bundle is under data/tested-models/stage6/registry/. It is a build artifact and is ignored by Git.

3. Install on hosts

Use the matching configiq-deploy checkout and its deploy-tested-models.sh installer. The installer stages and atomically replaces the host bundle at /var/lib/configiq/tested-models; it does not rebuild the application image. After installation, it explicitly loads the generated FIL models into Triton. A webapp restart is not required.

For a remote host:

./deploy-tested-models.sh \
  --host <host> \
  --ssh-key ~/.ssh/configiq-instance \
  --bundle data/tested-models/stage6/registry

Run the installer for each environment that should receive the new bundle. Verify GET /api/tested-models, Triton /v2/health/ready, and the expected FIL model states before considering the rollout complete.

Classifier guarantees

Feature order, categorical vocabularies, numeric medians, and decision thresholds are recorded in each model's metadata. Configuration groups are assigned as a unit so repeated measurements cannot cross train/validation/test splits. Throughput, latency, identifiers, runtime arguments, and other measurements are excluded from classifier inputs to avoid leakage.

split-summary.json records, per version family, the rows and configuration groups that landed in each split. A family that only appears in one split cannot be both learned and evaluated: the classifier is either blind to it at inference or cannot score it on held-out data. The training report copies that support into version_support, and records selection_basis (validation or train_fallback) plus an explicit validation_single_class / test_single_class warning when a split lacks both label classes. Those data shape caveats are published as diagnostics.warnings for both validated and measured pairs, while diagnostics.reasons stays a direct mirror of the quality gate so a caveat is never mistaken for a gate failure.

legacy_mvp.py remains only as a migration reference. It is not part of the production build and must not be used to publish artifacts.

The tested-models index at /tested-models lists validated and measured-only pairs, the active validation gate, and approved models without measurements. Filters accept ?model=<modelId> and ?system=<systemId>.

Each pair has a canonical graph at /tested-models/<pair-id> with #coverage, #boundary, and #metrics anchors. Legacy ?pair=<pair-id> links redirect to the canonical route. The sizing pages link problematic-input feedback to the exact model/system pair:

  • validated or measured-only pair: /tested-models/<pair-id>
  • approved or measured model without a pair for that system: /tested-models?model=<modelId>&system=<systemId>

The classifier API returns dashboard_href and boundary_href for the same locations. It never substitutes a different simulator or classifier; a pair without a validated envelope returns measured_no_envelope with no verdict.