Tao Analyze Detection Kpi

TAO Analyze Detection KPI

Use this skill to run TAO Data Services KPI analysis for object detection. The skill compares inference annotations against ground truth over one or more KPI sources and writes a per-class metrics CSV. It does not run inference; an upstream step must produce the inference annotations first.

The container entrypoint is:

analytics kpi_analyze -e /absolute/path/to/kpi_analyze.yaml

Inputs

The user provides either a finished spec or the paths to fill into the template.

Required spec fields:

Common optional fields. The Default column is what TAO DS uses when the field is absent; assets/default_kpi_analyze.yaml already carries the recommended value for each, so filling the template needs none of them changed:

For a Grounding DINO loop, inference_ann_path is the labels/ directory TAO inference writes under {results_dir}/inference/labels/, and input_format is KITTI.

The default template is assets/default_kpi_analyze.yaml.

Quick Start

Run from the tao-skill-bank repo root.

Write the spec into the results directory. The run does not retain it, so a completed run otherwise cannot tell you which settings produced kpi_calc.csv. Keeping them together makes the result reproducible from the run alone.

RESULTS_DIR=/absolute/path/for/this/run          # results_dir in the spec
SPEC="$RESULTS_DIR/kpi_analyze.yaml"        # spec lives beside its outputs
RUN_ROOT=/absolute/path/that/contains/images/annotations/and/results

python3 skills/data/tao-analyze-detection-kpi/scripts/verify_kpi_analyze_spec.py \
  --spec "$SPEC"

DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"

docker run --rm --gpus all --shm-size=8g --network=host \
  -v "$RUN_ROOT:$RUN_ROOT" \
  -w "$RUN_ROOT" \
  "$DS_IMAGE" \
  analytics kpi_analyze -e "$SPEC"

Pass --gpus all even though the analysis itself is CPU-only. The TAO launcher calls nvidia-smi -L unconditionally before dispatching any subtask, so a container started without GPU access dies with FileNotFoundError: 'nvidia-smi' before kpi_analyze runs.

Do not pass --user $(id -u):$(id -g); some TAO DS images call getpass.getuser() at startup and fail when the UID is not in /etc/passwd.

Generate A Spec

If the user provides paths instead of a ready spec, copy the template and fill in the nulls. Every tuning value it already carries is the one this stage wants — change one only deliberately.

cp skills/data/tao-analyze-detection-kpi/assets/default_kpi_analyze.yaml "$SPEC"

Fill data.kpi_sources (one entry per source), data.mapping and results_dir, all as absolute paths, then validate:

python3 skills/data/tao-analyze-detection-kpi/scripts/verify_kpi_analyze_spec.py --spec "$SPEC"

data:
  input_format: KITTI
  kpi_sources:
  - image_dir: /absolute/path/kpi/images               # no trailing slash
    ground_truth_ann_path: /absolute/path/kpi/labels
    inference_ann_path: /absolute/path/results/inference/labels
  mapping: /absolute/path/mapping.yaml
results_dir: /absolute/path/results/analyze_kpi

The template is the only place a default value lives, so nothing can disagree with it. verify reports the three settings that change what the numbers mean — conf_threshold, num_recall_points, ignore_sqwidth — so the spec that ran is recoverable from its output.

Preflight

1. Verify Docker access:

docker info > /dev/null

2. Resolve and pull the data-services image if needed:

DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
docker image inspect "$DS_IMAGE" > /dev/null || docker pull "$DS_IMAGE"

3. Validate the spec:

python3 skills/data/tao-analyze-detection-kpi/scripts/verify_kpi_analyze_spec.py --spec "$SPEC"

4. Confirm RUN_ROOT contains the spec, every image_dir, both annotation paths per source, the mapping file, and the results directory. Mount RUN_ROOT to the same absolute path inside Docker.

Outputs

kpi_calc.csv columns: Sequence Name, TP, FP, FN, TN, Pr, Re, Acc, AP — one row per sequence per class. A per-class result table and the aggregate mAP are also printed to stdout; capture the log if the caller needs the mAP value, since it is not written to the CSV.

Pitfalls

Ground truth may be 15- or 16-field KITTI. The parser names 15 columns for ground truth and 16 for predictions, but reads with index_col=False, so a trailing conf_score on a GT file is truncated rather than shifted. Feeding GT straight from tooling that writes a score column is fine — verified byte-identical results either way. The ParserWarning about "length of header or names does not match length of data" describes exactly that truncation and is not a sign of corruption.

input_format is uppercase here. analytics kpi_analyze accepts only KITTI or COCO. This differs from gap_analysis object_detection, which takes lowercase kitti / coco. Passing lowercase to this action fails to construct the data object.

Sequence Name is derived from the path, not configured. It is image_dir.split('/')[-2] — the second-to-last component of image_dir. A trailing slash or a flat image directory shifts which component is picked, so two sources can collide under one name. Lay out image_dir so that component is the sequence identifier you want.

data.mapping values are LISTS of aliases, not strings. This is the single most destructive thing to get wrong: the file is a YAML list of single-key dicts whose value is a list of source names that fold into that canonical class.

- bicycle:
    - Bicycle
    - Motorcycle
    - bicycle
    - twowheeler
- car:
    - car
    - Heavy Truck
    - Vehicle

construct_category_map stores the value verbatim (cat_map[k] = v), so writing - car: car — a bare string — yields a value that downstream code iterates character by character. Class matching then fails for every box, and the run still exits 0: the result is TP=0, FN=0, every prediction counted a false positive, and mAP: 0.0, with no error and no warning.

The tell is a perfect-looking run with all-zero metrics. Sanity-check by scoring a ground-truth set against a copy of itself — with a correct mapping that returns TP = every box and mAP: 1.0; anything else means the mapping, not the model.

data.mapping is required. The Hydra schema marks it mandatory even though the underlying category-map builder can derive classes from the label directory when it is absent. Supply the YAML.

Two different conf_threshold defaults. The dataclass default is 0.5, the shipped spec template uses 0.0. Whichever you rely on, set it explicitly — an unset value silently changes which predictions are scored.

is_internal: true is destructive to the report. It drops every class except person and appends a Summary row. Leave it false unless you specifically want the internal person-only KPI.

Troubleshooting

<key> not found in kpi_sources: every source entry needs all three of image_dir, ground_truth_ann_path, inference_ann_path.

Paths not found inside Docker: use a RUN_ROOT mount where host and container paths are identical, and confirm the images and annotation directories are under that mount.

Empty or all-zero metrics: usually conf_threshold above the model's score range, or an input_format that does not match the annotations on disk.

wandb errors or hangs: set visualize.platform: local to write a PR-curve plot instead of logging to wandb.