AutoGineer
Generated banner for Tao Train Fast Foundation Stereo
Generated logo for Tao Train Fast Foundation Stereo

Tao Train Fast Foundation Stereo

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Tool integration
  • Data and AI
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Real-time stereo depth estimation using FastFoundationStereo (FFS), the distilled bp2 commercial variant of FoundationStereo. Predicts disparity maps from stereo image pairs with ~10× lower latency than full FoundationStereo. Use when training, evaluating, exporting, or running inference for a TAO FastFoundationStereo (FFS) model. Trigger phrases include "train fast stereo", "real-time stereo disparity", "FastFoundationStereo", "distilled stereo depth".
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-train-fast-foundation-stereo skill enables users to work with the FastFoundationStereo (FFS) model, a distilled variant of FoundationStereo designed for stereo depth estimation. It integrates with the TAO depth_net CLI to support the full model lifecycle, including training, evaluation, export, and inference. The skill provides specific configuration requirements for model architecture parameters and dataset handling to ensure compatibility with the bp2 commercial checkpoint.

What it does

  • Supports training and fine-tuning of FastFoundationStereo models.
  • Facilitates model evaluation using standard stereo depth metrics.
  • Enables export of models to TensorRT engines for deployment.
  • Provides inference capabilities for generating disparity maps from stereo image pairs.
  • Includes AutoML support for optimizing training hyperparameters.
  • Manages configuration templates for consistent model deployment.

When to use it

  • When you need to train or fine-tune a FastFoundationStereo model on custom stereo datasets.
  • When evaluating the performance of a FastFoundationStereo model using ground truth disparity data.
  • When deploying a pre-trained bp2 commercial checkpoint for real-time inference.
  • When converting a FastFoundationStereo model to a TensorRT engine for optimized execution.

Advantages

  • Provides a unified CLI interface for the entire model development lifecycle.
  • Includes specific configuration overrides to ensure compatibility with bp2 commercial checkpoints.
  • Supports both raw deployment and fine-tuning workflows.
  • Integrates with TAO AutoML for hyperparameter optimization.

Usage examples

“Train a FastFoundationStereo model using my custom dataset.”
The agent will guide you through setting up the spec file with the required model width overrides and launching the training job via the depth_net CLI.
“Export my FastFoundationStereo model to a TensorRT engine.”
The agent will assist in configuring the export action and generating the optimized TensorRT engine for deployment.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/tao-train-fast-foundation-stereo/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-train-fast-foundation-stereo.zip -d .agents/skills

Documentation by NVIDIA Corporation

Depth Net Fast Stereo

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Real-time stereo depth estimation using FastFoundationStereo (FFS) — the bp2 commercial distilled variant of FoundationStereo. Predicts disparity maps from rectified stereo image pairs with per-layer pruned widths for real-time inference.

The mono / stereo / fast-stereo skills share the unified TAO depth_net CLI; FFS is selected via model.model_type: FastFoundationStereo. FFS differs from FoundationStereo only in pruned per-layer widths and a serialized forward path; everything else (entrypoint, action verbs, dataset classes, deploy chain) is identical to depth-net-stereo.

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, TensorRT inference), read references/tao-deploy-fast-foundation-stereo.md first. The deploy spec template lives at references/spec_template_deploy.yaml.

When to Use

Use this skill to train, evaluate, export, or run inference for a TAO FastFoundationStereo model. Two supported use cases:

FFS raw-deploy and bp2-finetune flows require a pre-trained bp2 commercial checkpoint (model_best_bp2_serialize.pth). The default PyT image does not guarantee that this file is present on disk, so treat the checkpoint path as a required user/registry artifact. If no bp2 checkpoint is available, scratch training is still usable for workflow validation, but the resulting metrics are not representative of the bp2 model.

  1. Raw deploy — use the bp2 ckpt as-is. Skip train; run inference / evaluate / export / gen_trt_engine directly with the bp2 file as the action's checkpoint.

  2. Finetune on user data — set train.pretrained_model_path to the bp2 file, train on user data, then verify + deploy on the resulting ckpt. The full 7-action sequence (train → evaluate pyt → inference pyt → export → gen_trt_engine → inference deploy → evaluate deploy) is supported.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

FFS shares the depth_net_stereo schema but its bp2 architecture widths are fixed invariants. For default AutoML, search only train.optim.lr and train.optim.lr_decay unless the user explicitly requests a wider search. Do not include FFS architecture fields such as model.volume_dim, model.hidden_dims, or other bp2 width settings in the default search space. Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.

Workflow

Prerequisites — data accessibility

Your dataset (left + right images + GT disparity for train / evaluate, left + right only for inference) must be reachable from inside the container:

  • SDK runner: place files at the S3 paths the runner resolves (S3_TRAIN / S3_EVAL placeholders shown in spec overrides).

  • Direct docker run (e.g. local testing): mount the host dataset root read-only at the same in-container path:

docker run ... -v <host_data_root>:<host_data_root>:ro <container> ...

The same accessibility requirement applies to the <output_dir> written by all actions, and to the bp2 checkpoint path.

Step 1 — Annotation file

Per-line annotation file referenced by data_sources[*].data_file. Schema is identical to depth-net-stereo:

ColumnsFormatUse
2<left> <right>Stereo inference (no GT)
3<left> <right> <disparity>Stereo with GT
4<left> <right> <disparity> <occlusion_mask>Stereo with GT and occlusion mask

Generate via depth_net convert if needed; see the depth-net-stereo skill for convert_spec.yaml template.

Step 2 — Pair model_type and dataset_name based on your data

Use model_type: FastFoundationStereo for FFS. The dataset_name choice mirrors the stereo skill — pick the dataset-specific class when your layout matches a registered one, otherwise GenericDataset.

Data categorymodel_typedataset_name
MiddleburyFastFoundationStereoMiddlebury
KITTIFastFoundationStereoKitti
ETH3DFastFoundationStereoEth3d
FSD syntheticFastFoundationStereoFSD
IsaacReal syntheticFastFoundationStereoIsaacRealDataset
Crestereo syntheticFastFoundationStereoCrestereo
Other / non-canonicalFastFoundationStereoGenericDataset

For inference with 2-column annotations (left + right, no GT), use dataset_name: GenericDataset regardless of layout.

Step 3 — Set the bp2 distilled width overrides

FFS requires 15 model-section width override fields whose values match the bp2 commercial checkpoint exactly. Omitting any field falls back to TAO defaults that do not match the bp2 ckpt and produce shape-mismatch errors at forward time. See references/setup-and-run.md for the full copy-as-is model: block and notes. The spec templates at references/spec_template_*.yaml carry this block as the canonical source.

Step 4 — Write spec yaml from spec overrides

Copy the action block from references/spec-overrides.md. Replace:

  • model.model_type: FastFoundationStereo (already set)

  • dataset.<...>.data_sources[*].dataset_name from Step 2

  • dataset.<...>.data_sources[*].data_file with the path from Step 1

  • For raw deploy use cases (no train): set <action>.checkpoint to the bp2 file path

  • For finetune use cases: set train.pretrained_model_path to the bp2 file path

For chained train → next-action checkpoint path resolution and shape-consistency notes, see references/setup-and-run.md. SDK-runner deploys resolve handoff automatically via parent_job_id — see references/parent-model-inference.md.

Step 5 — Run

Create writable home/cache directories inside the mounted output path before using --user, then launch docker run ... depth_net <action> -e <spec.yaml>. See references/setup-and-run.md for the full mkdir + docker run command, the --user rationale, and the local bind-mount __pycache__ tip.

Step 6 — Verify

Check container exit code 0 and a populated status.json kpi block. For train inspect per-step train_loss directly (the entrypoint reports Execution status: PASS even when loss is NaN); for evaluate rely on epe / bp1 / bp2 / bp3 / d1 / rmse; for inference check artifacts under results_dir. The pyt-vs-deploy KPI namespace difference and the expected deploy drift are detailed in references/setup-and-run.md.

7-action deploy flow
train (optional)            → finetuned ckpt
evaluate (pyt)              → PyT eager EPE / bp on val GT
inference (pyt)             → PyT eager disparity samples (visual sanity)
export                      → static fp32 ONNX (recommended at 480×736 or 320×736)
gen_trt_engine             → fp16 TRT engine on static ONNX path
inference (deploy)         → TRT disparity samples
evaluate (deploy)          → TRT EPE / bp drift vs PyT eager fp32

Skip train for raw-bp2 deploy. The remaining 6 actions (or the 4 deploy-only verbs starting from export) cover both use cases.

Training Requirements

  • Valid dataset_name values for stereo data_sources (case-insensitive): FSD, IsaacRealDataset, Crestereo, Middlebury, Eth3d, Kitti, GenericDataset

  • Monitoring metric: val/loss

Per-Action Dataset Requirements
ActionSpec KeySourceFilesList?
evaluatedataset.test_dataset.data_sourceseval_datasetdata_file: annotations.txt + dataset_nameYes
inferencedataset.infer_dataset.data_sourcesinference_datasetdata_file: annotations.txt + dataset_nameYes
traindataset.train_dataset.data_sourcestrain_datasetsdata_file: annotations.txt + dataset_nameYes
traindataset.val_dataset.data_sourceseval_datasetdata_file: annotations.txt + dataset_nameYes
Typical Spec Overrides

Data source overrides are mandatory for every action. Each data_sources entry is a dict with two mandatory fields: data_file and dataset_name. The model.* width fields are also mandatory — see Step 3. See references/spec-overrides.md for the FFS_MODEL_BLOCK and per-action (train / evaluate / inference / export) Python override dicts.

Eval Dataset

Optional. Val dataset configured via dataset.val_dataset.data_sources (each entry needs data_file and dataset_name).

Important Parameters

Key knobs include model.model_type (FastFoundationStereo), model.encoder (vitl), model.max_disparity (set 192 explicitly — schema default 416 causes severe drift), model.mixed_precision (false), model.gwc_feature_normalize (true), model.volume_dim (28), model.valid_iters (8), and per-split batch_size / workers / crop_size / data_sources. Full parameter reference, evaluation metrics, multi-GPU / multi-node spec keys, export / TRT defaults, the export use-case matrix, and hardware guidance are in references/important-parameters.md.

Error Patterns

For shape mismatch, gwc_feature_normalize schema errors, max_disparity drift, negative disparity, depth_net_stereo: not found, the pyt-evaluate crop_size asymmetry, the Failed to import SAM3 warning, and the dynamic-engine stride-incompatible silent failure, see references/error-patterns.md.

Spec Param / Parent Model Inference

Model-specific inference mappings (per-action spec field → inference function) for train / evaluate / inference / export / gen_trt_engine, plus parent_job_id / parent_model resolution and raw-bp2 explicit-checkpoint handling, are in references/parent-model-inference.md. Generated runners should read that section and apply the mappings with SDK helpers before create_job().

Deployment

Files

references/important-parameters.md

Important Parameters

  • model.model_type: Must be FastFoundationStereo for this skill.

  • model.encoder: ViT backbone size; bp2 ckpt was trained with vitl. Other sizes will fail to load the bp2 weights.

  • model.hidden_dims: bp2 uses [128] (single-GRU). Do not use the full-FS default [128, 128, 128] — shape-mismatch on the GRU head.

  • model.n_gru_layers: bp2 uses 1. Pair with hidden_dims: [128].

  • model.max_disparity: bp2 commercial uses 192. The TAO Core schema default for this field is 416 — if the spec yaml's model: block does not explicitly set max_disparity: 192, OmegaConf falls back to the schema default and the cost volume is built with 2× the correct number of disparity levels (~104 vs the bp2-trained 48 at 1/4 scale). The model still loads and runs, but per-pixel disparity drifts severely from upstream because the cost-volume softmax peak shifts out of the trained regime. Always set model.max_disparity: 192 explicitly in the spec for FFS-bp2 deploy — do not rely on the schema default. The setting on dataset.max_disparity is a separate dataset-side knob and does not propagate to the model.

  • model.mixed_precision: Recommend false for FFS-bp2 train and pyt eval. The bp2 commercial ckpt was distilled upstream with bf16 amp, but the FS trainer in TAO does not support bf16 (only fp32 and fp16). Using mixed_precision: false (= fp32 forward) gives the cleanest pyt-vs-deploy parity check.

  • model.gwc_feature_normalize: Must be true for FFS-bp2. The bp2 model was trained with normalized group-wise correlation cost volume, and the model code without this flag produces broken disparity (negative values, large drift from upstream baseline). Required for both pyt and deploy paths.

  • model.train_iters: GRU refinement iterations during training. Default 22.

  • model.valid_iters: GRU refinement iterations during inference / eval. bp2 ckpt was distilled targeting 8; values higher than 8 do not improve quality.

  • model.volume_dim: Cost volume Conv output channels. Schema default 32 (full-FS); FFS bp2 ckpt requires 28 — must override explicitly. Changing breaks bp2 ckpt key-shape match.

  • model.low_memory: Memory optimization level. Range 0-4. Higher = less memory, slower.

  • dataset.dataset_name: Top-level dataset family identifier (StereoDataset).

  • dataset.{train,val,test,infer}_dataset.batch_size: Per-split batch size. Use 1 for variable-aspect datasets (Middlebury / KITTI / ETH3D) and during eval / TRT comparison; larger batch sizes are fine for fixed-shape synthetic data.

  • dataset.{train,val,test,infer}_dataset.workers: Per-split DataLoader worker count.

  • dataset.{train,val,test,infer}_dataset.augmentation.crop_size: Per-split crop. Match export.input_height / export.input_width and the deploy-side evaluate crop_size for end-to-end shape consistency.

  • dataset.{train,val,test,infer}_dataset.data_sources: List of {data_file, dataset_name} dicts.

  • train.optim.lr: Learning rate. Default 1e-4 (AdamW). For bp2 finetune, prefer 1e-5 (matches upstream).

  • train.precision: Training precision. Options: fp32 (recommended for FFS-bp2), fp16. (bf16 is not supported by the FS trainer.)

  • train.distributed_strategy: Distribution strategy. Options: ddp, fsdp.

  • inference.save_raw_pfm: Pyt inference action only — when true, the per-image disparity is dumped as a raw .pfm next to the colorized .png. Deploy inference (TRT engine path) emits only the colorized .png under predicted_depth/<scene>_im0.png; the save_raw_pfm knob is not consumed there. Use the pyt inference path if raw .pfm output is required.

Evaluation Metrics

StereoDepthEvaluator emits a fixed metric set; only the disparity-domain metrics are meaningful:

MetricMeaningUse
epemean End-Point-Error in pixelsprimary stereo metric
bp1 / bp2 / bp3fraction of pixels with EPE > 1 / 2 / 3 pxquality thresholds
d1KITTI-style outlier rate (EPE > 3 px AND > 5% of GT disparity)KITTI-comparable headline
rmseRMSE on disparity valuessensitivity to large errors

The same evaluator also emits abs_rel, sq_rel, rmse_log — these are formulated for monocular metric depth and produce non-meaningful values on disparity. Ignore them for stereo evaluation.

Multi-GPU / Multi-Node

Launch method: Lightning-managed (single python process, Lightning spawns workers). Same DDP / FSDP behavior as depth-net-stereo.

Spec KeyDescriptionDefault
train.num_gpusNumber of GPUs1
train.gpu_idsGPU device indices[0]
train.num_nodesNumber of nodes1
train.distributed_strategyddp or fsdpddp

Multi-node requires WORLD_SIZE, NODE_RANK, MASTER_ADDR, MASTER_PORT env vars.

Export / TRT Defaults

  • TRT data types: FP32, FP16.

  • Recommended TRT precision for FFS-bp2: fp16 on the static-shape ONNX path (lowest drift). Dynamic-shape path supports both fp32 (default; static-fp32 parity) and fp16 (latency-critical multi-resolution; higher drift than static fp16, may NaN under some checkpoint states — fall back to fp32 if observed). See references/tao-deploy-fast-foundation-stereo.md deployment matrix.

  • export always emits a fp32 ONNX regardless of model.mixed_precision. The fp16 vs fp32 selection happens at the gen_trt_engine step via gen_trt_engine.tensorrt.data_type.

  • For static-shape FFS at 480×736: export.batch_size: 1, export.opset_version: 17, export.on_cpu: False.

  • export.batch_size: positive int (default 1) — static batch dimension; -1 enables a dynamic batch axis on the ONNX input.

  • export.dynamic_hw: bool (default false) — true enables dynamic H/W axes on the ONNX input. FFS only. FS / mono models ignore this flag with a warning and fall back to static H/W (their DINOv2 backbone constant-folds positional embeddings into the trace, so dynamic H/W at runtime would produce a wrong-shape pos-embed mismatching the actual patch tokens — silent crash). FFS uses EdgeNeXt only and is safe.

Export use-case matrix

export.batch_size and export.dynamic_hw are independent. The four combinations:

Use casebatch_sizedynamic_hwResulting ONNX
Fixed-batch fixed-resolution (most common, production fp16)1 (positive)falsestatic [1, 3, H, W]
Variable-batch fixed-resolution-1falsedynamic batch only
Variable-resolution single-batch (FFS only)1 (positive)truedynamic H/W only
Variable-resolution + variable-batch (FFS only)-1trueboth batch and H/W dynamic

For FS / mono models, dynamic_hw: true is automatically ignored with a warning and the engine falls back to static H/W. Only FastFoundationStereo supports dynamic H/W due to its EdgeNeXt-only encoder.

Hardware

  • Minimum 1 GPU, 24 GB+ VRAM per GPU recommended (A6000 / A100). FFS is ~10× lower-memory than full FoundationStereo at the same input shape, but cost-volume convolution still dominates peak VRAM during training.

  • For inference / deploy on edge: A2 / Orin-class GPUs handle FFS at 480×736 fp16 within real-time budget.

  • model.low_memory > 0 for constrained GPUs at training time.

  • fp32 recommended for training (bf16 unsupported by FS trainer).

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs a command-line tool, Needs Node.js

The author says: Requires docker + nvidia-container-toolkit.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-train-fast-foundation-stereo · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-train-fast-foundation-stereo.zip