Documentation
Tested-model pipeline
ConfigIQ's tested-model system turns measured inference results into
configuration-specific performance-envelope classifiers. The pipeline and its
source code live in this repository; configiq-tested-models is no longer the
production pipeline.
Runtime contract
The pipeline produces one sanitized bundle containing:
registry.json Webapp-facing pair and classifier metadata
dashboard/<pair-id>.json Sanitized dashboard data
triton/<pair-id>/ Triton FIL model repositories
config.pbtxt
1/xgboost.json
The ConfigIQ container reads registry.json from TESTED_MODELS_DIR, which is
/tested-models by default. The deployment repository mounts the host bundle
at that path and mounts its triton/ subdirectory at /models for Triton.
The webapp endpoints are GET /api/tested-models,
GET /api/tested-models/dashboard/<pair-id>, and
POST /api/tested-models/classify. The human-facing pages are /tested-models
and /tested-models/<pair-id>.
Registry schema v2
registry.json separates three concepts that must never be conflated:
- validated — a pair has a trained classifier whose held-out test measurements clear the validation gate. Only validated pairs can emit a classifier verdict.
- measured_no_envelope — the pair has published measurements and dashboard graphs, but no classifier passes the gate. The graph stays available; the sizing pages show "measured, no validated envelope" and emit no verdict.
- approved_no_data — the model appears in
approvedModelsbut has no pair in the registry. It is approved for measurement, not classifier-tested.
The validation gate requires at least 100 records, at least five concurrency levels, test ROC-AUC ≥ 0.70, and test F1 ≥ 0.30. A failing pair retains its measurements and graphs and records diagnostic reasons; it exposes no classifier artifacts. The gate is published in the registry and returned by the API so the UI and pipeline cannot drift.
Schema v1 registries remain readable during rollout. A v1 pair is treated as validated only when its recorded test AUC and F1 clear the metric thresholds; v1 carries no coverage summary, so it cannot claim the full gate.
Coverage and feature categories
A v2 pair's coverage object records records, concurrencyLevels,
concurrencyLevelCount, the raw backendVersions for provenance, and the
measured backends (runtime families such as vllm or sglang). The
classify endpoint refuses to score a request whose backend family is not in
that list: it returns no_measurements with a reason instead of applying an
envelope trained on a different runtime. Bundles that predate backends are
treated as unknown and stay permissive.
The classifier's version feature is a backend-scoped major.minor family, not
a raw label. Training rows (vLLM-0.24.0) and runtime catalog versions
(0.24.0) both normalize to vLLM-0.24;
sglang-0.5.18-recipe-latency-oriented normalizes to sglang-0.5. Pooling
keeps a category dense enough to validate while preventing families from
different runtimes from colliding. Unrecognised or unresolved labels
(current, arbitrary build strings) become unknown.
ConfigIQ's public/config.json names approved models in approvedModels. The
pipeline accepts the retired testedModels name for older deployments, but
the frontend config no longer exposes it. The config list governs measurement
coverage; it is not evidence that a model has been classifier-tested. Tested
status comes from registry.json alone.
The deployment-owned k2_horizon vLLM model is separate from the generated
FIL bundle. It shares the Triton server but is not generated by these scripts.
The deployment-owned qwen3_8b model is a fallback and is normally unloaded.
Data boundary
Ground-truth extraction runs only from an approved environment. The legacy API
extractor receives its source endpoint through PERF_DATA_API_URL; the GitHub
publisher reads the approved S3 aggregate with an OIDC read-only role. Neither
source location is written to published artifacts. Extraction removes run IDs,
UUIDs, MLflow identifiers, timestamps, raw runtime arguments, and other
administrative fields before publication. The S3 importer includes only models
in public/config.json's approvedModels list, and only rows whose version is a
plain upstream vLLM release (vLLM-<major>.<minor>.<patch>). Productized Red
Hat AI Inference Server builds and pre-release labels are dropped, and the
import manifest records the kept and dropped counts.
The scheduled GitHub Actions workflow consumes the public
redhat-performance/configiq-performance-upstream-data Hugging Face dataset.
It uploads only the sanitized registry/Triton/dashboard bundle as a workflow
artifact. Raw parquet files, source identifiers, and training work files are
not published by that workflow.
By default the workflow follows the dataset's default branch. The download step
resolves that mutable ref once to a full 40-character commit SHA, downloads only
that SHA, and records it in dataset-manifest.json; registry.json records the
same repository and revision under dataset so a deployed bundle can be traced
back to immutable input. Pass a commit SHA, branch, or tag to pin a build
explicitly (--revision on the script, or the revision input on the
tested-models.yml workflow dispatch).
That dataset is the upstream-only subset: it retains rows whose serving stack is a plain upstream vLLM release and excludes productized Red Hat AI Inference Server builds. It grows as new measurements are published, so a model can be approved for testing without having a trained performance envelope yet. Do not read absence from this dataset as "not tested".
registry.json lists every measured pair in this build, with its validation
status. It is a catalog of measured coverage, not the approved-model list. A
model can be approved for testing with no pair, and a pair can be measured
without having a validated envelope. Neither case is classifier-tested.
The workflow reads the repository variable TESTED_DATASET_REPO when it is set,
and otherwise falls back to the dataset above. Set it to move the pipeline to a
renamed or republished dataset without a code change.
The former TESTED_DATASET_REVISION repository variable is no longer read.
Builds follow the dataset automatically; use the workflow dispatch revision
input when a specific revision must be rebuilt.
Production workflow
1. Publish ground truth
The GitHub workflow .github/workflows/publish-performance-data.yml downloads
the aggregate, prepares sanitized Parquet pairs, and compares their content
with the public Hugging Face dataset. It runs at 00:00 Monday in
America/New_York on the default branch, or manually through
workflow_dispatch. Scheduled publishing requires the GitHub repository
variable PERFORMANCE_DATA_PUBLISH_ENABLED=true; otherwise scheduled runs
only prepare and compare. A manual run on main publishes only when its
publish input is selected. Unchanged content is never uploaded.
When an upload succeeds, the publish workflow dispatches tested-models.yml
with the new dataset commit SHA, so new ground truth retrains the classifiers
without a manual revision bump. The build workflow also runs on its Monday
09:17 UTC fallback schedule and on relevant code changes. It appends a
Markdown comparison of the new registry against the deployed registry (when
https://configiq.dev/api/tested-models is reachable) to the workflow job
summary, including added/removed pairs and validated-to-measured transitions.
That report is advisory: it never fails the build or installs the bundle.
Preparation applies the upstream-only version filter and refuses to publish when
the prepared set would delete pairs that exist only in the remote dataset. Run
the complete extraction if pairs are missing; pass --allow-removal only when
removing pairs is intentional. It also rejects distinct model/accelerator labels
that normalize to the same Parquet pair ID before writing either file. The
publisher checks upstream version and backend fields again before upload.
Hugging Face publishing uses a Trusted Publisher on the dataset repository,
restricted to redhat-performance/configiq, branch main, and workflow
publish-performance-data.yml. No persistent Hugging Face token is stored in
GitHub. The AWS read-only role must trust the repository's OIDC subject for
main. If TESTED_DATASET_REPO points at another dataset repository, configure
an equivalent Trusted Publisher there; the workflow derives HF_OIDC_RESOURCE
from that same variable so publishing and dispatched builds target one
repository.
The earlier API extraction path remains available from an approved data environment:
From the approved data environment, extract and publish a sanitized dataset:
uv sync --group tested-models
export PERF_DATA_API_URL="https://<approved-data-source>"
uv run --group tested-models python scripts/tested_models/discover_data.py --accelerator H200
uv run --group tested-models python scripts/tested_models/extract_datasets.py
uv run --group tested-models python scripts/tested_models/publish_ground_truth.py
The publish workflow dispatches the tested-model build automatically with the
new Hugging Face commit SHA, so no revision variable needs updating. To rebuild
a specific published revision, dispatch tested-models.yml with its revision
input. If the data is published to a repository other than the default one
above, set the TESTED_DATASET_REPO repository variable as well. Do not commit
the dataset or its source credentials.
Refresh the private InferenceX comparison dataset
The same Monday 00:00 America/New_York workflow has an independent
refresh-inferencex job. It collects published InferenceX single-turn 8k/1k
measurements for models in public/config.json's approvedModels list and
updates redhat-performance/configiq-inferencex-comparison only when the
measurements change. The publisher refuses to upload if that dataset is public.
The InferenceX job does not use the S3 role.
Scheduled InferenceX publishing requires the repository variable
INFERENCEX_PUBLISH_ENABLED=true. On a manual run from main, select the
separate publish_inferencex input to refresh the private dataset. When it is
the only publish input selected, the S3 job is skipped. The InferenceX job
uses a write-enabled Hugging Face Trusted Publisher restricted to
redhat-performance/configiq, refs/heads/main, and
.github/workflows/publish-performance-data.yml; it does not store a long-lived
write token in GitHub. The deployed comparison app uses its separate
server-side read token to load the private dataset.
2. Build and validate the bundle
The repeatable build is .github/workflows/tested-models.yml. It runs the
regression tests, downloads the current dataset revision, creates grouped
leakage-safe splits, fits labels from training rows, trains XGBoost classifiers,
exports Triton FIL repositories, builds dashboard payloads for every measured
pair, and validates the registry. build_dashboard.py must run before
build_registry.py, because the registry requires a dashboard payload for each
pair it publishes.
Run the same sequence locally when debugging:
uv run --group tested-models pytest scripts/tested_models/tests -q
uv run --group tested-models python scripts/tested_models/download_ground_truth.py
uv run --group tested-models python scripts/tested_models/split_dataset.py
uv run --group tested-models python scripts/tested_models/label_data.py
uv run --group tested-models python scripts/tested_models/train_classifiers.py
uv run --group tested-models python scripts/tested_models/export_triton.py
uv run --group tested-models python scripts/tested_models/build_dashboard.py
uv run --group tested-models python scripts/tested_models/build_registry.py
uv run --group tested-models python scripts/tested_models/validate_artifacts.py
The generated bundle is under
data/tested-models/stage6/registry/. It is a build artifact and is ignored
by Git.
3. Install on hosts
Use the matching configiq-deploy checkout and its
deploy-tested-models.sh installer. The installer stages and atomically
replaces the host bundle at /var/lib/configiq/tested-models; it does not
rebuild the application image. After installation, it explicitly loads the
generated FIL models into Triton. A webapp restart is not required.
For a remote host:
./deploy-tested-models.sh \
--host <host> \
--ssh-key ~/.ssh/configiq-instance \
--bundle data/tested-models/stage6/registry
Run the installer for each environment that should receive the new bundle.
Verify GET /api/tested-models, Triton /v2/health/ready, and the expected
FIL model states before considering the rollout complete.
Classifier guarantees
Feature order, categorical vocabularies, numeric medians, and decision thresholds are recorded in each model's metadata. Configuration groups are assigned as a unit so repeated measurements cannot cross train/validation/test splits. Throughput, latency, identifiers, runtime arguments, and other measurements are excluded from classifier inputs to avoid leakage.
split-summary.json records, per version family, the rows and configuration
groups that landed in each split. A family that only appears in one split
cannot be both learned and evaluated: the classifier is either blind to it at
inference or cannot score it on held-out data. The training report copies that
support into version_support, and records selection_basis
(validation or train_fallback) plus an explicit validation_single_class /
test_single_class warning when a split lacks both label classes. Those data
shape caveats are published as diagnostics.warnings for both validated and
measured pairs, while diagnostics.reasons stays a direct mirror of the
quality gate so a caveat is never mistaken for a gate failure.
legacy_mvp.py remains only as a migration reference. It is not part of the
production build and must not be used to publish artifacts.
Status and deep links
The tested-models index at /tested-models lists validated and measured-only
pairs, the active validation gate, and approved models without measurements.
Filters accept ?model=<modelId> and ?system=<systemId>.
Each pair has a canonical graph at /tested-models/<pair-id> with #coverage,
#boundary, and #metrics anchors. Legacy ?pair=<pair-id> links redirect to
the canonical route. The sizing pages link problematic-input feedback to the
exact model/system pair:
- validated or measured-only pair:
/tested-models/<pair-id> - approved or measured model without a pair for that system:
/tested-models?model=<modelId>&system=<systemId>
The classifier API returns dashboard_href and boundary_href for the same
locations. It never substitutes a different simulator or classifier; a pair
without a validated envelope returns measured_no_envelope with no verdict.