Tao Train Nvpanoptix3d
NVIDIAOfficialCreated by NVIDIA Corporation
- Skill
- Tool integration
- Data and AI
- Apache-2.0
- Medium risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
NVPanoptix3D is a specialized tool for performing panoptic 3D scene reconstruction from posed RGB images. It leverages a VGGT backbone with a Mask2Former-style head to generate semantic, instance, and panoptic masks alongside 3D occupancy completion. This skill provides the necessary interface to train, evaluate, perform inference, and export models within the TAO Toolkit environment, supporting complex 3D vision workflows.
What it does
- Performs 3D panoptic segmentation including semantic and instance masks.
- Executes 3D occupancy completion for scene reconstruction.
- Supports training, evaluation, and inference workflows using posed RGB images.
- Enables model export to ONNX format for deployment.
- Integrates with TAO Toolkit AutoML for optimized model training.
- Handles staged initialization using 2D and 3D checkpoints.
When to use it
- When you need to reconstruct 3D scenes from 2D RGB image data.
- When performing panoptic segmentation tasks in 3D environments.
- When you require occupancy completion for spatial awareness in vision models.
- When integrating 3D scene understanding into a TAO-based machine learning pipeline.
Advantages
- Provides a unified pipeline for complex 3D reconstruction tasks.
- Supports AutoML for automated hyperparameter optimization.
- Includes robust evaluation metrics like PRQ, RSQ, and RRQ.
- Compatible with standard datasets like Front3D and Matterport.
Usage examples
- “Train an NVPanoptix3D model using my custom dataset.”
- The agent initiates a training job using the specified train.yaml spec, utilizing the configured GPU resources and dataset paths.
- “Evaluate my trained NVPanoptix3D model.”
- The agent runs the evaluation entrypoint against the provided checkpoint and evaluation dataset, outputting performance metrics.
- “Run inference on a folder of RGB images.”
- The agent processes the images in the specified directory and generates panoptic segmentation results based on the provided model checkpoint.
- “Export my NVPanoptix3D model to ONNX.”
- The agent executes the export workflow to produce a 2D ONNX model file suitable for deployment.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/tao-train-nvpanoptix3d/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip tao-train-nvpanoptix3d.zip -d .agents/skills
Documentation by NVIDIA Corporation
NVPanoptix3D
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the
tao-setupskill first (host preflight, credentials, cross-skill discovery).
NVPanoptix3D for panoptic 3D scene reconstruction from posed RGB images. Produces 3D panoptic segmentation (semantic, instance, and panoptic masks) with occupancy completion. Built on VGGT backbone with Mask2Former-style head and 3D frustum reconstruction.
Uses 2D and 3D stage checkpoints. Set train.checkpoint_2d and train.checkpoint_3d for staged initialization.
Quick Start (docker run)
Docker-native launch — no TAO SDK and no Python on the host. Use the local Docker/platform skill instead when it gives a stricter environment-specific command (non-root UID mapping, cache redirects, remote daemons).
TAO_PYT_IMAGE_DEFAULT=nvcr.io/nvidia/tao/tao-toolkit:7.3.0-pyt # versions-key: images.tao_toolkit.pyt
TAO_PYT_IMAGE="${TAO_PYT_IMAGE:-$TAO_PYT_IMAGE_DEFAULT}"
RUN_ROOT="${RUN_ROOT:-$PWD}"
DOCKER_COMMON=(
--rm --gpus all --shm-size=8g
--shm-size=8g
--ulimit memlock=-1
--ulimit stack=67108864
-v "$RUN_ROOT/data:/data:ro"
-v "$RUN_ROOT/specs:/specs:ro"
-v "$RUN_ROOT/results:/results"
)Train:
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d train -e /specs/train.yamlEvaluate:
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d evaluate -e /specs/evaluate.yamlInference:
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d inference -e /specs/inference.yamlExport:
docker run "${DOCKER_COMMON[@]}" "$TAO_PYT_IMAGE" \
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d export -e /specs/export.yamlEvery action takes its spec with -e; results_dir is set in the spec or
overridden on the command line. Mount any pretrained-weights directory the spec
references, and keep every in-container path consistent across actions.
Dataclass Schemas
Generated TAO Core schemas are packaged in schemas/<action>.schema.json, with schemas/manifest.json listing available actions. Each generated schema also emits references/spec_template_<action>.yaml from the schema top-level default field. AutoML enablement is declared at the model layer in references/skill_info.yaml via automl_enabled. Runnable AutoML for an action requires schemas/<action>.schema.json and references/spec_template_<action>.yaml to exist and parse. Use the packaged selected-action schema for automl_default_parameters, automl_disabled_parameters, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect ~/tao-core at runtime; maintainers regenerate schemas/templates before packaging the skill bank.
Train Action Policy
This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.
For AutoML, use PRQ as the optimization metric with direction=maximize.
NVPanoptix3D train validation and evaluate jobs both emit PRQ, RSQ, and
RRQ in status.json, so use PRQ consistently for the baseline, every
recommendation, and final best-checkpoint evaluation. The model may use
train.optim.monitor_name: train_loss internally for checkpointing, but
minimal jobs do not reliably export a numeric train_loss to the TAO status
channel; do not use it as the AutoML selection metric. Multi-fidelity
promotions must obtain a fresh PRQ after the resumed epoch and must still
resume from the explicit epoch/step checkpoint, produce a real checkpoint,
and pass evaluate/inference.
Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.
Training Requirements
Dataset type: nvpanoptix3d
Formats: front3d, matterport
Monitoring metric:
PRQAutoML direction: maximize
Validation and evaluate status KPIs are
PRQ,RSQ, andRRQ. UsePRQfor AutoML selection; do not usetrain_lossorval_lossunless a different workflow proves that exact scalar is externally emitted for every trial and obtains explicit approval to use a proxy objective.
Per-Action Dataset Requirements
| Action | Spec Key | Source | Files | List? |
|---|---|---|---|---|
| evaluate | dataset.frustum_mask_path | eval_dataset | meta/frustum_mask.npz | No |
| evaluate | dataset.label_map | eval_dataset | meta/colormap.json | No |
| evaluate | dataset.val.json_path | eval_dataset | meta/val.json | No |
| evaluate | dataset.val.base_dir | eval_dataset | No | |
| evaluate | dataset.test.json_path | inference_dataset | meta/test.json | No |
| evaluate | dataset.test.base_dir | inference_dataset | No | |
| inference | dataset.frustum_mask_path | inference_dataset | meta/frustum_mask.npz | No |
| inference | dataset.label_map | inference_dataset | meta/colormap.json | No |
| inference | inference.images_dir | inference_dataset | flat folder of .jpg/.png RGB images | No |
| train | dataset.frustum_mask_path | train_datasets | meta/frustum_mask.npz | No |
| train | dataset.label_map | train_datasets | meta/colormap.json | No |
| train | dataset.train.json_path | train_datasets | meta/train.json | No |
| train | dataset.train.base_dir | train_datasets | No | |
| train | dataset.val.json_path | eval_dataset | meta/val.json | No |
| train | dataset.val.base_dir | eval_dataset | No | |
| train | dataset.test.json_path | inference_dataset | meta/test.json | No |
| train | dataset.test.base_dir | inference_dataset | No |
Typical Spec Overrides
Data source overrides are mandatory for every action — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in spec_overrides.
For packaged S3 folders that store scene data as data/images.tar.gz, the
skill metadata requests extraction into the parent data/ directory because
the TAO loader expects base_dir/data/<scene_id>/....
S3_TRAIN = "s3://bucket/data/train"
S3_EVAL = "s3://bucket/data/eval"train (mandatory data sources):
{
"train.num_epochs": 10,
"train.checkpoint_interval": 10,
"train.validation_interval": 10,
"train.num_gpus": 1,
"dataset.enable_3d": True,
"dataset.contiguous_id": True,
"model.sem_seg_head.num_classes": 13,
"dataset.frustum_mask_path": f"{S3_TRAIN}/meta/frustum_mask.npz",
"dataset.label_map": f"{S3_TRAIN}/meta/colormap.json",
"dataset.train.json_path": f"{S3_TRAIN}/meta/train.json",
"dataset.train.base_dir": f"{S3_TRAIN}",
"dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
"dataset.val.base_dir": f"{S3_EVAL}",
"dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
"dataset.test.base_dir": f"{S3_EVAL}",
}evaluate (mandatory data sources):
{
"evaluate.checkpoint": "<selected train/AutoML checkpoint>",
"dataset.enable_3d": True,
"dataset.contiguous_id": True,
"dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
"dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
"dataset.val.json_path": f"{S3_EVAL}/meta/val.json",
"dataset.val.base_dir": f"{S3_EVAL}",
"dataset.test.json_path": f"{S3_EVAL}/meta/test.json",
"dataset.test.base_dir": f"{S3_EVAL}",
}inference (mandatory data sources):
{
"inference.checkpoint": "<selected train/AutoML checkpoint>",
"dataset.enable_3d": True,
"dataset.frustum_mask_path": f"{S3_EVAL}/meta/frustum_mask.npz",
"dataset.label_map": f"{S3_EVAL}/meta/colormap.json",
"inference.images_dir": "/path/to/flat_rgb_images",
}Eval Dataset
Optional. Val/test splits configured via dataset.val and dataset.test paths.
Important Parameters
model.sem_seg_head.num_classes: Number of semantic classes. Default 13.
model.mode: Prediction mode. Options: panoptic, instance, semantic. Default panoptic.
model.backbone_type: Backbone. Default vggt (only option in schema).
model.mask_former.num_object_queries: Object queries. Default 100.
model.mask_former.dec_layers: Decoder layers. Default 10.
model.frustum3d.truncation: 3D frustum truncation. Default 3.
model.frustum3d.panoptic_weight: Panoptic loss weight. Default 25.
model.frustum3d.completion_weights: Completion loss weights. Default [50, 25, 10].
dataset.name: Dataset name. Options: front3d, matterport, synthetic_hospital, synthetic_warehouse.
dataset.contiguous_id: Set
Truewhen the label-map JSON already suppliestrainIdvalues for its category IDs; leaving the defaultFalsecan synthesize placeholder categories withouttrainIdand fail during metadata construction.dataset.downsample_factor: Image downsample factor. Default 1 (Front3D), 2 (Matterport).
dataset.target_size: Target image size. Default [320, 240].
dataset.depth_min: Min depth. Default 0.4 meters.
dataset.depth_max: Max depth. Default 6.0 meters.
train.lr: Learning rate. Default 2e-4. backbone_multiplier=0.1.
train.lr_scheduler: Options: MultiStep, Warmuppoly. Milestones [88, 96].
train.precision: Only
fp32is supported by the current train code.train.distributed_strategy: Options: ddp, fsdp. activation_checkpoint=True by default.
train.clip_grad_norm: Gradient clipping norm. Default 0.1.
export.onnx_file_2d: ONNX path for 2D model component.
export.max_voxels: Max voxels for engine input. Default 700000.
inference.mode: Options: semantic, instance, panoptic.
Multi-GPU / Multi-Node
Launch method: Lightning-managed (single python process, Lightning spawns workers).
| Spec Key | Description | Default |
|---|---|---|
train.num_gpus | Number of GPUs | 1 |
train.gpu_ids | GPU device indices | [0] |
train.num_nodes | Number of nodes | 1 |
train.distributed_strategy | ddp only | ddp |
fsdpis NOT supported for NVPanoptix3D (code only handlesddp)ddpwith activation checkpointing (enabled by default):find_unused_parameters=Falseddpwithout:find_unused_parameters=TrueFAN backbones with 3D enabled auto-enable
sync_batchnorm
Multi-node env vars (set by orchestrator): WORLD_SIZE, NODE_RANK, MASTER_ADDR, MASTER_PORT, NUM_GPU_PER_NODE.
Export / TRT Defaults
Exports the 2D ONNX model to
export.onnx_file_2d. The current export entrypoint callsexport_2d_model;export.onnx_file_3dis present in the schema but not produced by this toolkit image.TRT data types: FP32, FP16 only
max_voxels: 700000 (engine input tensor limit)
Hardware
Minimum 2 GPU(s), recommended 4 GPU(s). 40GB+ (A100 recommended) VRAM per GPU. 3D reconstruction is very memory intensive. Use train.precision: fp32; the current training entrypoint rejects fp16. activation_checkpoint enabled by default. FSDP for multi-node. AutoML is enabled at the model layer; preserve this GPU/VRAM guidance when routing train through AutoML.
Error Patterns
nvpanoptix3d: not found in the PyTorch image: Use the packaged module
entrypoint command:
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d <action> -e <spec>.
The 7.0 PyTorch image contains the NVPanoptix3D package but does not expose a
nvpanoptix3d console script.
Missing frustum mask: Ensure meta/frustum_mask.npz is present in the dataset directory.
Downsample factor mismatch: Use downsample_factor=2 for Matterport3D, 1 for Front3D / synthetic datasets.
3D occupancy OOM: Reduce frustum_dims or grid_dimensions if running out of GPU memory during 3D reconstruction.
fp16 precision rejected: The schema advertises fp16, but the current
training entrypoint raises ValueError: Only fp32 precision is supported. Use
train.precision: fp32 for train and resume/retrain.
Inference dataloader length is zero: inference.images_dir is scanned only
for top-level .jpg and .png files. If the S3 test archive extracts to
scene subdirectories, create or point to a flat folder of real RGB images before
running inference.
3D ONNX missing after export: The current export entrypoint only calls the
2D ONNX exporter and writes export.onnx_file_2d. Do not require
export.onnx_file_3d unless the toolkit image adds a 3D exporter.
Resume stops at the epoch boundary: A one-epoch smoke run writes an
end-of-epoch checkpoint such as model_epoch_000_step_00020.pth. Resuming with
train.num_epochs set only one epoch beyond the original run can restore the
checkpoint and stop without producing a new epoch checkpoint. When validating
actual retraining from an epoch-boundary checkpoint, set train.num_epochs at
least two epochs beyond the source smoke run and raise train.optim.max_steps
accordingly. For example, resuming from model_epoch_000_step_00020.pth needs
train.num_epochs: 3 and enough max steps to produce a new exact epoch/step
checkpoint such as model_epoch_001_step_00040.pth before handing the model to
evaluate, inference, or export.
Spec Param / Parent Model Inference
Model-specific inference mappings belong in this MD file, not in config.json. Generated runners should read this section and apply the mappings with SDK helpers before create_job(). This mirrors the old microservices infer_params.py flow.
Model-specific handoff mappings:
| Action | Spec Field | Inference Function | Meaning |
|---|---|---|---|
| evaluate | encryption_key | key | encryption key |
| evaluate | evaluate.checkpoint | parent_model | model file inferred from the parent job results folder |
| evaluate | results_dir | output_dir | current job results directory |
| export | encryption_key | key | encryption key |
| export | export.checkpoint | parent_model | model file inferred from the parent job results folder |
| export | export.onnx_file_2d | create_onnx_file_2d | output 2D ONNX path |
| export | results_dir | output_dir | current job results directory |
| inference | encryption_key | key | encryption key |
| inference | inference.checkpoint | parent_model | model file inferred from the parent job results folder |
| inference | results_dir | output_dir | current job results directory |
| train | encryption_key | key | encryption key |
| train | results_dir | output_dir | current job results directory |
| train | train.checkpoint_2d | parent_model_or_ptm | parent model if available, otherwise PTM |
| train | train.checkpoint_3d | ptm | pretrained model |
| train | train.resume_training_checkpoint_path | resume_model | model file inferred from the current job results folder |
For parent_model or parent_model_folder, pass the upstream train/export/AutoML child job id as parent_job_id. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to config.json and do not patch generated runner scripts to guess checkpoint paths.
Files
- ATTRIBUTION.mdAdded by AutoGineer936 B
- BENCHMARK.md7.0 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md16.6 KiB
- config/skillspector-baseline.yaml1.0 KiB
- evals/evals.json824 B
- references/skill_info.yaml6.3 KiB
- references/spec_template_evaluate.yaml3.6 KiB
- references/spec_template_export.yaml3.8 KiB
- references/spec_template_inference.yaml3.6 KiB
- references/spec_template_train.yaml3.5 KiB
- schemas/evaluate.schema.json58.3 KiB
- schemas/export.schema.json59.9 KiB
- schemas/inference.schema.json58.7 KiB
- schemas/manifest.json10.7 KiB
- schemas/train.schema.json55.7 KiB
- skill-card.md4.4 KiB
- skill.oms.sig7.1 KiB
schemas/evaluate.schema.json
{
"automl_default_parameters": [
"train.optim.lr"
],
"automl_disabled_parameters": [
"train.cudnn",
"dataset.augmentation.crop_size",
"model.frustum3d.completion_weights",
"train.gpu_ids",
"model.sem_seg_head",
"wandb.tags",
"model.backbone",
"dataset.depth_size",
"train.optim.milestones",
"evaluate",
"inference",
"train",
"dataset.augmentation",
"dataset.augmentation.train_crop_size",
"gen_trt_engine",
"gen_trt_engine.tensorrt.layers_precision",
"model.sem_seg_head.in_features",
"dataset.target_size",
"model.frustum3d",
"dataset",
"gen_trt_engine.tensorrt",
"dataset.reduced_target_size",
"model.sem_seg_head.deformable_transformer_encoder_in_features",
"dataset.train",
"model",
"train.freeze",
"evaluate.gpu_ids",
"dataset.augmentation.train_min_size",
"model.mask_former",
"train.optim",
"dataset.val",
"export",
"dataset.truncation_range",
"wandb",
"model.projection",
"dataset.occ_truncation_lvl",
"inference.gpu_ids",
"dataset.test"
],
"default": {
"dataset": {
"augmentation": {
"color_aug_ssd": false,
"crop_size": [
240,
240
],
"enable_crop": false,
"gen_aug_weight": 0.0,
"random_flip": "",
"random_flip_prob": 0.5,
"single_category_max_area": 1.0,
"size_divisibility": -1.0,
"test_max_size": 960,
"test_min_size": 240,
"train_crop_size": [
240,
240
],
"train_max_size": 768,
"train_min_size": [
448
]
},
"contiguous_id": false,
"depth_bound": false,
"depth_max": 6.0,
"depth_min": 0.4,
"depth_scale": 25.0,
"depth_size": [
120,
160
],
"downsample_factor": 1,
"enable_3d": false,
"enable_mp_occ": true,
"frustum_mask_path": "meta/frustum_mask.npz",
"ignore_label": 255,
"img_format": "RGB",
"iso_value": 1.0,
"label_map": "",
"min_instance_pixels": 200,
"name": "front3d",
"num_thing_classes": 9,
"occ_truncation_lvl": [
8.0,
6.0
],
"pin_memory": true,
"reduced_target_size": [
160,
120
],
"target_size": [
320,
240
],
"test": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"train": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"truncation_range": [
0.0,
12.0
],
"val": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"workers": 8
},
"encryption_key": "",
"evaluate": {
"batch_size": -1,
"checkpoint": "",
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1,
"results_dir": "",
"trt_engine": ""
},
"model": {
"backbone": {
"backbone_type": "vggt",
"pretrained_model_path": ""
},
"frustum3d": {
"completion_weights": [
50.0,
25.0,
10.0
],
"frustum_dims": 256,
"grid_dimensions": 256,
"iso_recon_value": 2.0,
"panoptic_weight": 25.0,
"signed_channel": 3,
"surface_weight": 5.0,
"truncation": 3.0,
"unet_features": 16,
"unet_output_channels": 16,
"use_multi_scale": false
},
"mask_former": {
"class_weight": 2.0,
"dec_layers": 10,
"deep_supervision": true,
"depth_weight": 5.0,
"dice_weight": 5.0,
"dim_feedforward": 2048,
"dropout": 0.0,
"hidden_dim": 256,
"importance_sample_ratio": 0.75,
"mask_weight": 5.0,
"mp_occ_weight": 5.0,
"nheads": 8,
"no_object_weight": 0.1,
"num_object_queries": 100,
"oversample_ratio": 3.0,
"pre_norm": false,
"size_divisibility": 32,
"train_num_points": 12544,
"transformer_dim_feedforward": 1024
},
"mode": "panoptic",
"object_mask_threshold": 0.4,
"overlap_threshold": 0.5,
"projection": {
"depth_feature_dim": 256,
"sign_channel": true,
"voxel_size": 0.03
},
"sem_seg_head": {
"common_stride": 4,
"convs_dim": 256,
"deformable_transformer_encoder_in_features": [
"res3",
"res4",
"res5"
],
"depth_dim": 256,
"ignore_value": 255,
"in_features": [
"res2",
"res3",
"res4",
"res5"
],
"mask_dim": 256,
"norm": "GN",
"num_classes": 13,
"transformer_enc_layers": 6
},
"test_topk_per_image": 100
},
"model_name": "",
"results_dir": "",
"train": {
"activation_checkpoint": true,
"checkpoint_2d": "",
"checkpoint_3d": "",
"checkpoint_interval": 1,
"checkpoint_interval_unit": "epoch",
"clip_grad_norm": 0.1,
"clip_grad_type": "full",
"cudnn": {
"benchmark": false,
"deterministic": true
},
"distributed_strategy": "ddp",
"freeze": [],
"gpu_ids": [
0
],
"is_dry_run": false,
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1,
"optim": {
"backbone_multiplier": 0.1,
"gamma": 0.1,
"lr": 0.0002,
"lr_scheduler": "MultiStep",
"max_steps": 160000,
"milestones": [
88,
96
],
"momentum": 0.9,
"monitor_name": "val_loss",
"type": "AdamW",
"warmup_factor": 1.0,
"warmup_iters": 0,
"weight_decay": 0.05
},
"precision": "fp32",
"results_dir": "",
"resume_training_checkpoint_path": "",
"seed": 1234,
"val_check_interval": 5,
"validation_interval": 1,
"verbose": false
},
"wandb": {
"enable": true,
"entity": "",
"group": "",
"name": "TAO Toolkit Training",
"project": "TAO Toolkit",
"reinit": false,
"run_id": "",
"save_code": false,
"sync_tensorboard": false,
"tags": [
"tao-toolkit"
]
}
},
"popular": {
"evaluate": {
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1
},
"gen_trt_engine": {
"batch_size": -1,
"gpu_id": 0,
"tensorrt": {
"max_batch_size": 1,
"min_batch_size": 1,
"opt_batch_size": 1
}
},
"inference": {
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1
},
"model": {
"mask_former": {
"class_weight": 2.0,
"dice_weight": 5.0,
"hidden_dim": 256,
"importance_sample_ratio": 0.75,
"mask_weight": 5.0,
"nheads": 8,
"num_object_queries": 100
},
"sem_seg_head": {
"convs_dim": 256,
"depth_dim": 256,
"mask_dim": 256,
"transformer_enc_layers": 6
}
},
"train": {
"checkpoint_interval": 1,
"gpu_ids": [
0
],
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1,
"validation_interval": 1
}
},
"properties": {
"automl_disabled_parameters": [
"wandb",
"model",
"dataset",
"train",
"inference",
"evaluate",
"export",
"gen_trt_engine"
],
"dataset": {
"automl_disabled_parameters": [
"dataset.train",
"dataset.val",
"dataset.test",
"dataset.augmentation",
"dataset.target_size",
"dataset.reduced_target_size",
"dataset.depth_size",
"dataset.occ_truncation_lvl",
"dataset.truncation_range"
],
"automl_enabled": false,
"default": {
"augmentation": {
"color_aug_ssd": false,
"crop_size": [
240,
240
],
"enable_crop": false,
"gen_aug_weight": 0.0,
"random_flip": "",
"random_flip_prob": 0.5,
"single_category_max_area": 1.0,
"size_divisibility": -1.0,
"test_max_size": 960,
"test_min_size": 240,
"train_crop_size": [
240,
240
],
"train_max_size": 768,
"train_min_size": [
448
]
},
"contiguous_id": false,
"depth_bound": false,
"depth_max": 6.0,
"depth_min": 0.4,
"depth_scale": 25.0,
"depth_size": [
120,
160
],
"downsample_factor": 1,
"enable_3d": false,
"enable_mp_occ": true,
"frustum_mask_path": "meta/frustum_mask.npz",
"ignore_label": 255,
"img_format": "RGB",
"iso_value": 1.0,
"label_map": "",
"min_instance_pixels": 200,
"name": "front3d",
"num_thing_classes": 9,
"occ_truncation_lvl": [
8.0,
6.0
],
"pin_memory": true,
"reduced_target_size": [
160,
120
],
"target_size": [
320,
240
],
"test": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"train": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"truncation_range": [
0.0,
12.0
],
"val": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"workers": 8
},
"description": "Configurable parameters to construct the dataset for the NVPanoptix3D experiment.",
"properties": {
"augmentation": {
"automl_disabled_parameters": [
"dataset.augmentation.train_min_size",
"dataset.augmentation.train_crop_size",
"dataset.augmentation.crop_size"
],
"automl_enabled": false,
"default": {
"color_aug_ssd": false,
"crop_size": [
240,
240
],
"enable_crop": false,
"gen_aug_weight": 0.0,
"random_flip": "",
"random_flip_prob": 0.5,
"single_category_max_area": 1.0,
"size_divisibility": -1.0,
"test_max_size": 960,
"test_min_size": 240,
"train_crop_size": [
240,
240
],
"train_max_size": 768,
"train_min_size": [
448
]
},
"description": "Configuration parameters for data augmentation",
"properties": {
"color_aug_ssd": {
"default": false,
"description": "Color augmentation.",
"title": "color augmentation",
"type": "bool"
},
"crop_size": {
"automl_enabled": false,
"default": [
240,
240
],
"description": "Size to crop input image.",
"title": "input image size crop",
"type": "list"
},
"enable_crop": {
"default": false,
"description": "Enable cropping for input image.",
"title": "enable cropping",
"type": "bool"
},
"gen_aug_weight": {
"default": 0.0,
"description": "Weight for generated augmentation, 0.0 will disable generated augmentation.",
"maximum": 1.0,
"minimum": 0.0,
"title": "weight for generated augmentation",
"type": "float"
},
"random_flip": {
"default": "",
"description": "Flip horizontal/vertical.",
"title": "flip horizontal/vertical",
"type": "string"
},
"random_flip_prob": {
"default": 0.5,
"description": "Flip probability.",
"maximum": 1.0,
"minimum": 0.0,
"title": "flip probability",
"type": "float"
},
"single_category_max_area": {
"default": 1.0,
"description": "Maximum ratio of crop area that can be occupied by a single semantic category.",
"maximum": 1.0,
"minimum": 0.0,
"title": "maximum ratio of crop area",
"type": "float"
},
"size_divisibility": {
"default": -1.0,
"description": "Size divisibility to pad.",
"title": "size divisibility to pad",
"type": "float"
},
"test_max_size": {
"default": 960,
"description": "The maximum resize size for test.",
"maximum": 960,
"minimum": 32,
"title": "test max size",
"type": "int"
},
"test_min_size": {
"default": 240,
"description": "The minimum resize size for test data.",
"maximum": 960,
"minimum": 32,
"title": "test min size",
"type": "int"
},
"train_crop_size": {
"automl_enabled": false,
"default": [
240,
240
],
"description": "The random crop size for training data in [H, W].",
"title": "train crop size",
"type": "list"
},
"train_max_size": {
"default": 768,
"description": "The maximum random crop size for training data.",
"maximum": 960,
"minimum": 32,
"title": "train max size",
"type": "int"
},
"train_min_size": {
"automl_enabled": false,
"default": [
448
],
"description": "A list of sizes to perform random resize.",
"title": "train min size",
"type": "list"
}
},
"type": "collection"
},
"contiguous_id": {
"default": false,
"description": "Flag to enable contiguous ids for labels.",
"title": "contiguous id",
"type": "bool"
},
"depth_bound": {
"default": false,
"description": "Enable depth truncation in bounds.",
"title": "enable depth truncation",
"type": "bool"
},
"depth_max": {
"default": 6.0,
"description": "Max depth value.",
"title": "max depth value",
"type": "float"
},
"depth_min": {
"default": 0.4,
"description": "Min depth value.",
"title": "min depth value",
"type": "float"
},
"depth_scale": {
"default": 25.0,
"description": "Depth scale.",
"title": "depth scale",
"type": "float"
},
"depth_size": {
"automl_enabled": false,
"default": [
120,
160
],
"description": "Input depth size to resize.",
"title": "input depth size to resize",
"type": "list"
},
"downsample_factor": {
"default": 1,
"description": "Downsample factor(1: Synthetic & Front3D, 2: Matterport3D).",
"title": "downsample factor",
"type": "int"
},
"enable_3d": {
"default": false,
"description": "Enable 3d for training.",
"title": "enable 3d",
"type": "bool"
},
"enable_mp_occ": {
"default": true,
"description": "Enable multi-plane occupancy.",
"title": "enable multi-plane occupancy",
"type": "bool"
},
"frustum_mask_path": {
"default": "meta/frustum_mask.npz",
"description": "Relative frustum mask path.",
"title": "relative frustum mask path",
"type": "string"
},
"ignore_label": {
"default": 255,
"description": "Ignore label value.",
"title": "ignore label value",
"type": "int"
},
"img_format": {
"default": "RGB",
"description": "Image format.",
"title": "image format",
"type": "string"
},
"iso_value": {
"default": 1.0,
"description": "ISO value to reconstruct mesh from TUDF volume.",
"title": "ISO value",
"type": "float"
},
"label_map": {
"default": "",
"description": "A path to label map file.",
"title": "label map path",
"type": "string"
},
"min_instance_pixels": {
"default": 200,
"description": "Minimum number of pixels required for an instance to be considered valid.",
"title": "minimum number of pixels",
"type": "int"
},
"name": {
"default": "front3d",
"description": "Dataset name.",
"enum": [
"front3d",
"matterport",
"synthetic_hospital",
"synthetic_warehouse"
],
"title": "dataset name",
"type": "categorical"
},
"num_thing_classes": {
"default": 9,
"description": "Number of thing classes.",
"title": "number of thing classes",
"type": "int"
},
"occ_truncation_lvl": {
"automl_enabled": false,
"default": [
8.0,
6.0
],
"description": "Value to create occuppancy volume from TUDF volume.",
"title": "occ truncation level",
"type": "list"
},
"pin_memory": {
"default": true,
"description": "Flag to allocate pagelocked memory for faster of data between the CPU and GPU.",
"title": "pin memory",
"type": "bool"
},
"reduced_target_size": {
"automl_enabled": false,
"default": [
160,
120
],
"description": "Image size to process at 3D stage.",
"title": "image size to process at 3D stage",
"type": "list"
},
"target_size": {
"automl_enabled": false,
"default": [
320,
240
],
"description": "Input image size to resize.",
"title": "input image size to resize",
"type": "list"
},
"test": {
"automl_enabled": false,
"default": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"description": "Configurable parameters to construct the test dataset.",
"properties": {
"base_dir": {
"default": "",
"description": "Root directory of the dataset",
"title": "dataset root",
"type": "string"
},
"batch_size": {
"default": 1,
"description": "Batch size",
"math_cond": ">0",
"minimum": 1,
"title": "batch size",
"type": "int"
},
"json_path": {
"default": "",
"description": "JSON file in JSON format for image/mask pair.",
"title": "annotation file path",
"type": "string"
},
"num_workers": {
"default": 1,
"description": "Number of workers in the dataloader.",
"minimum": 0,
"title": "num workers",
"type": "int"
}
},
"type": "collection"
},
"train": {
"automl_enabled": false,
"default": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"description": "Configurable parameters to construct the train dataset.",
"properties": {
"base_dir": {
"default": "",
"description": "Root directory of the dataset",
"title": "dataset root",
"type": "string"
},
"batch_size": {
"default": 1,
"description": "Batch size",
"math_cond": ">0",
"minimum": 1,
"title": "batch size",
"type": "int"
},
"json_path": {
"default": "",
"description": "JSON file in JSON format for image/mask pair.",
"title": "annotation file path",
"type": "string"
},
"num_workers": {
"default": 1,
"description": "Number of workers in the dataloader.",
"minimum": 0,
"title": "num workers",
"type": "int"
}
},
"type": "collection"
},
"truncation_range": {
"automl_enabled": false,
"default": [
0.0,
12.0
],
"description": "truncation range for TUDF volume.",
"title": "TUDF truncation range",
"type": "list"
},
"val": {
"automl_enabled": false,
"default": {
"base_dir": "",
"batch_size": 1,
"json_path": "",
"num_workers": 1
},
"description": "Configurable parameters to construct the validation dataset.",
"properties": {
"base_dir": {
"default": "",
"description": "Root directory of the dataset",
"title": "dataset root",
"type": "string"
},
"batch_size": {
"default": 1,
"description": "Batch size",
"math_cond": ">0",
"minimum": 1,
"title": "batch size",
"type": "int"
},
"json_path": {
"default": "",
"description": "JSON file in JSON format for image/mask pair.",
"title": "annotation file path",
"type": "string"
},
"num_workers": {
"default": 1,
"description": "Number of workers in the dataloader.",
"minimum": 0,
"title": "num workers",
"type": "int"
}
},
"type": "collection"
},
"workers": {
"default": 8,
"description": "The number of parallel workers processing data",
"minimum": 1,
"title": "num workers",
"type": "int"
}
},
"type": "collection"
},
"encryption_key": {
"default": "",
"description": "Key for encrypting model checkpoints",
"title": "Encryption key",
"type": "string"
},
"evaluate": {
"automl_disabled_parameters": [
"evaluate.gpu_ids"
],
"automl_enabled": false,
"default": {
"batch_size": -1,
"checkpoint": "",
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1,
"results_dir": "",
"trt_engine": ""
},
"description": "Configurable parameters to construct the evaluator for the NVPanoptix3D experiment.",
"popular": [
"num_gpus",
"num_nodes",
"gpu_ids"
],
"properties": {
"batch_size": {
"default": -1,
"description": "The batch size of the input Tensor. This is important if batch_size > 1 for large dataset.",
"minimum": -1,
"title": "batch size",
"type": "int"
},
"checkpoint": {
"default": "",
"description": "Path to the checkpoint used for evaluation.",
"title": "Checkpoint path",
"type": "string"
},
"gpu_ids": {
"automl_enabled": false,
"default": [
0
],
"description": "\n List of GPU IDs to run the evaluation on. The length of this list\n must be equal to the number of gpus in evaluate.num_gpus.",
"popular": true,
"title": "GPU IDs",
"type": "list"
},
"num_gpus": {
"default": 1,
"description": "The number of GPUs to run the evaluation job.",
"minimum": 1,
"popular": true,
"title": "Number of GPUs",
"type": "int"
},
"num_nodes": {
"default": 1,
"description": "Number of nodes to run the evaluation on. If > 1, then multi-node is enabled.",
"minimum": 1,
"popular": true,
"title": "Number of nodes",
"type": "int"
},
"results_dir": {
"default": "",
"description": "\n Path to where all the assets generated from a task are stored.\n ",
"title": "Results directory",
"type": "string"
},
"trt_engine": {
"default": "",
"description": "Path to the TensorRT engine to be used for evaluation.\n This only works with :code:`tao-deploy`.",
"title": "TensorRT Engine",
"type": "string"
}
},
"type": "collection"
},
"model": {
"automl_disabled_parameters": [
"model.backbone",
"model.sem_seg_head",
"model.mask_former",
"model.frustum3d",
"model.projection"
],
"automl_enabled": false,
"default": {
"backbone": {
"backbone_type": "vggt",
"pretrained_model_path": ""
},
"frustum3d": {
"completion_weights": [
50.0,
25.0,
10.0
],
"frustum_dims": 256,
"grid_dimensions": 256,
"iso_recon_value": 2.0,
"panoptic_weight": 25.0,
"signed_channel": 3,
"surface_weight": 5.0,
"truncation": 3.0,
"unet_features": 16,
"unet_output_channels": 16,
"use_multi_scale": false
},
"mask_former": {
"class_weight": 2.0,
"dec_layers": 10,
"deep_supervision": true,
"depth_weight": 5.0,
"dice_weight": 5.0,
"dim_feedforward": 2048,
"dropout": 0.0,
"hidden_dim": 256,
"importance_sample_ratio": 0.75,
"mask_weight": 5.0,
"mp_occ_weight": 5.0,
"nheads": 8,
"no_object_weight": 0.1,
"num_object_queries": 100,
"oversample_ratio": 3.0,
"pre_norm": false,
"size_divisibility": 32,
"train_num_points": 12544,
"transformer_dim_feedforward": 1024
},
"mode": "panoptic",
"object_mask_threshold": 0.4,
"overlap_threshold": 0.5,
"projection": {
"depth_feature_dim": 256,
"sign_channel": true,
"voxel_size": 0.03
},
"sem_seg_head": {
"common_stride": 4,
"convs_dim": 256,
"deformable_transformer_encoder_in_features": [
"res3",
"res4",
"res5"
],
"depth_dim": 256,
"ignore_value": 255,
"in_features": [
"res2",
"res3",
"res4",
"res5"
],
"mask_dim": 256,
"norm": "GN",
"num_classes": 13,
"transformer_enc_layers": 6
},
"test_topk_per_image": 100
},
"description": "Configurable parameters to construct the model for the NVPanoptix3D experiment.",
"popular": [
"mask_former",
"sem_seg_head"
],
"properties": {
"backbone": {
"automl_enabled": false,
"default": {
"backbone_type": "vggt",
"pretrained_model_path": ""
},
"description": "Configuration hyper parameters for the NVPanoptix3D Backbone.",
"properties": {
"backbone_type": {
"default": "vggt",
"description": "Type of backbone to use. Available backbone: vggt.",
"enum": [
"vggt"
],
"title": "backbone name",
"type": "categorical"
},
"pretrained_model_path": {
"default": "",
"description": "Path to a pretrained backbone file.",
"title": "pretrained backbone path",
"type": "string"
}
},
"title": "backbone",
"type": "collection"
},
"frustum3d": {
"automl_disabled_parameters": [
"model.frustum3d.completion_weights"
],
"automl_enabled": false,
"default": {
"completion_weights": [
50.0,
25.0,
10.0
],
"frustum_dims": 256,
"grid_dimensions": 256,
"iso_recon_value": 2.0,
"panoptic_weight": 25.0,
"signed_channel": 3,
"surface_weight": 5.0,
"truncation": 3.0,
"unet_features": 16,
"unet_output_channels": 16,
"use_multi_scale": false
},
"description": "Configuration hyper parameters for the Frustum3D model.",
"properties": {
"completion_weights": {
"automl_enabled": false,
"default": [
50.0,
25.0,
10.0
],
"description": "The weights of the completion loss.",
"title": "completion weights",
"type": "list"
},
"frustum_dims": {
"default": 256,
"description": "The number of frustum dimensions.",
"title": "frustum dimensions",
"type": "int"
},
"grid_dimensions": {
"default": 256,
"description": "The number of grid dimensions.",
"title": "grid dimensions",
"type": "int"
},
"iso_recon_value": {
"default": 2.0,
"description": "The iso recon value.",
"title": "iso recon value",
"type": "float"
},
"panoptic_weight": {
"default": 25.0,
"description": "The weight of the panoptic loss.",
"title": "panoptic weight",
"type": "float"
},
"signed_channel": {
"default": 3,
"description": "The number of signed channel.",
"title": "signed channel",
"type": "int"
},
"surface_weight": {
"default": 5.0,
"description": "The weight of the surface loss.",
"title": "surface weight",
"type": "float"
},
"truncation": {
"default": 3.0,
"description": "The truncation value.",
"title": "truncation",
"type": "float"
},
"unet_features": {
"default": 16,
"description": "The number of features of the UNet.",
"title": "unet features",
"type": "int"
},
"unet_output_channels": {
"default": 16,
"description": "The number of output channels of the UNet.",
"title": "unet output channels",
"type": "int"
},
"use_multi_scale": {
"default": false,
"description": "Whether to use multi-scale.",
"title": "use multi-scale",
"type": "bool"
}
},
"title": "frustum3d",
"type": "collection"
},
"mask_former": {
"automl_enabled": false,
"default": {
"class_weight": 2.0,
"dec_layers": 10,
"deep_supervision": true,
"depth_weight": 5.0,
"dice_weight": 5.0,
"dim_feedforward": 2048,
"dropout": 0.0,
"hidden_dim": 256,
"importance_sample_ratio": 0.75,
"mask_weight": 5.0,
"mp_occ_weight": 5.0,
"nheads": 8,
"no_object_weight": 0.1,
"num_object_queries": 100,
"oversample_ratio": 3.0,
"pre_norm": false,
"size_divisibility": 32,
"train_num_points": 12544,
"transformer_dim_feedforward": 1024
},
"description": "Configuration hyper parameters for the Mask2Former model.",
"popular": [
"hidden_dim",
"importance_sample_ratio",
"class_weight",
"dice_weight",
"mask_weight",
"nheads",
"num_object_queries"
],
"properties": {
"class_weight": {
"default": 2.0,
"description": "The relative weight of the classification error in the matching cost.",
"maximum": Infinity,
"minimum": 0.0,
"popular": true,
"title": "Class loss coefficient",
"type": "float"
},
"dec_layers": {
"default": 10,
"description": "Numer of decoder layers in the transformer",
"minimum": 1,
"title": "decoder layers",
"type": "int"
},
"deep_supervision": {
"default": true,
"description": "Flag to enable deep supervision.",
"title": "deep supervision",
"type": "bool"
},
"depth_weight": {
"default": 5.0,
"description": "The relative weight of the depth loss in the matching cost.",
"maximum": Infinity,
"minimum": 0.0,
"title": "depth loss coefficient",
"type": "float"
},
"dice_weight": {
"default": 5.0,
"description": "The relative weight of the focal loss of the binary mask in the matching cost.",
"maximum": Infinity,
"minimum": 0.0,
"popular": true,
"title": "focal loss coefficient",
"type": "float"
},
"dim_feedforward": {
"default": 2048,
"description": "Dimension of the feedforward network",
"minimum": 1,
"title": "dim feedforward",
"type": "int"
},
"dropout": {
"default": 0.0,
"description": "The probability to drop out.",
"maximum": 1.0,
"minimum": 0.0,
"title": "drop out ratio",
"type": "float"
},
"hidden_dim": {
"default": 256,
"description": "Dimension of the hidden units.",
"popular": true,
"type": "int"
},
"importance_sample_ratio": {
"default": 0.75,
"description": "Ratio of points that are sampled via important sampling.",
"popular": true,
"title": "importance sampling ratio",
"type": "float"
},
"mask_weight": {
"default": 5.0,
"description": "The relative weight of the dice loss of the binary mask in the matching cost",
"maximum": Infinity,
"minimum": 0.0,
"popular": true,
"title": "mask loss coefficient",
"type": "float"
},
"mp_occ_weight": {
"default": 5.0,
"description": "The relative weight of the mp occ loss in the matching cost.",
"maximum": Infinity,
"minimum": 0.0,
"title": "mp occ loss coefficient",
"type": "float"
},
"nheads": {
"default": 8,
"description": "Number of heads",
"popular": true,
"title": "nheads",
"type": "int"
},
"no_object_weight": {
"default": 0.1,
"description": "The relative classification weight applied to the no-object category.",
"title": "no object coefficient",
"type": "float"
},
"num_object_queries": {
"default": 100,
"description": "The number of queries",
"maximum": Infinity,
"minimum": 1,
"popular": true,
"title": "number of queries",
"type": "int"
},
"oversample_ratio": {
"default": 3.0,
"description": "Oversampling parameter.",
"title": "oversampling ratio",
"type": "float"
},
"pre_norm": {
"default": false,
"description": "Flag to add layer norm in the encoder or not.",
"title": "Pre norm",
"type": "bool"
},
"size_divisibility": {
"default": 32,
"description": "Size divisibility.",
"title": "size divisibility",
"type": "int"
},
"train_num_points": {
"default": 12544,
"description": "The number of points P to sample.",
"title": "number of points",
"type": "int"
},
"transformer_dim_feedforward": {
"default": 1024,
"description": "Dimension of the feedforward network in the transformer",
"minimum": 1,
"title": "transformer dim feedforward",
"type": "int"
}
},
"title": "mask2former",
"type": "collection"
},
"mode": {
"default": "panoptic",
"description": "Segmentation mode.",
"enum": [
"panoptic",
"instance",
"semantic"
],
"title": "segmentation mode",
"type": "categorical"
},
"object_mask_threshold": {
"default": 0.4,
"description": "The value of the threshold to be used when filtering out the object mask.",
"title": "object mask threshold",
"type": "float"
},
"overlap_threshold": {
"default": 0.5,
"description": "The value of the threshold to be used when evaluating overlap.",
"title": "overlap threshold",
"type": "float"
},
"projection": {
"automl_enabled": false,
"default": {
"depth_feature_dim": 256,
"sign_channel": true,
"voxel_size": 0.03
},
"description": "Configuration hyper parameters for the Projection model.",
"properties": {
"depth_feature_dim": {
"default": 256,
"description": "The dimension of the depth feature.",
"title": "depth feature dim",
"type": "int"
},
"sign_channel": {
"default": true,
"description": "Whether to use signed channel.",
"title": "sign channel",
"type": "bool"
},
"voxel_size": {
"default": 0.03,
"description": "The size of the voxel.",
"title": "voxel size",
"type": "float"
}
},
"title": "projection",
"type": "collection"
},
"sem_seg_head": {
"automl_disabled_parameters": [
"model.sem_seg_head.deformable_transformer_encoder_in_features",
"model.sem_seg_head.in_features"
],
"automl_enabled": false,
"default": {
"common_stride": 4,
"convs_dim": 256,
"deformable_transformer_encoder_in_features": [
"res3",
"res4",
"res5"
],
"depth_dim": 256,
"ignore_value": 255,
"in_features": [
"res2",
"res3",
"res4",
"res5"
],
"mask_dim": 256,
"norm": "GN",
"num_classes": 13,
"transformer_enc_layers": 6
},
"description": "Configuration hyper parameters for the Mask2Former Semantic Segmentation Head.",
"popular": [
"transformer_enc_layers",
"convs_dim",
"mask_dim",
"depth_dim"
],
"properties": {
"common_stride": {
"default": 4,
"description": "Common stride.",
"minimum": 2,
"title": "Common stride",
"type": "int"
},
"convs_dim": {
"default": 256,
"description": "Convolutional layer dimension.",
"minimum": 1,
"popular": true,
"title": "conv layer dim.",
"type": "int"
},
"deformable_transformer_encoder_in_features": {
"automl_enabled": false,
"default": [
"res3",
"res4",
"res5"
],
"description": "List of feature names for deformable transformer encoder input.",
"title": "transformer encoder in_features",
"type": "list"
},
"depth_dim": {
"default": 256,
"description": "Depth head dimension.",
"minimum": 1,
"popular": true,
"title": "depth head dim.",
"type": "int"
},
"ignore_value": {
"default": 255,
"description": "Ignore value.",
"maximum": 255,
"minimum": 0,
"title": "ignore value",
"type": "int"
},
"in_features": {
"automl_enabled": false,
"default": [
"res2",
"res3",
"res4",
"res5"
],
"description": "List of input feature names.",
"title": "transformer encoder in_features",
"type": "list"
},
"mask_dim": {
"default": 256,
"description": "Mask head dimension.",
"minimum": 1,
"popular": true,
"title": "mask head dim.",
"type": "int"
},
"norm": {
"default": "GN",
"description": "Norm layer type.",
"title": "norm type",
"type": "string"
},
"num_classes": {
"default": 13,
"description": "Number of classes.",
"minimum": 1,
"title": "number of classes.",
"type": "int"
},
"transformer_enc_layers": {
"default": 6,
"description": "Number of transformer encoder layers.",
"minimum": 1,
"popular": true,
"title": "Number of transformer encoder layers.",
"type": "int"
}
},
"title": "segmentation head configs",
"type": "collection"
},
"test_topk_per_image": {
"default": 100,
"description": "Keep topk instances per image for instance segmentation.",
"title": "top k per image",
"type": "int"
}
},
"type": "collection"
},
"model_name": {
"default": "",
"description": "Name of model if invoking task via :code:`model_agnostic`",
"title": "Model name",
"type": "string"
},
"results_dir": {
"default": "",
"description": "\n Path to where all the assets generated from a task are stored.\n ",
"title": "Results directory",
"type": "string"
},
"train": {
"automl_disabled_parameters": [
"train.gpu_ids",
"train.cudnn",
"train.freeze",
"train.optim"
],
"automl_enabled": false,
"default": {
"activation_checkpoint": true,
"checkpoint_2d": "",
"checkpoint_3d": "",
"checkpoint_interval": 1,
"checkpoint_interval_unit": "epoch",
"clip_grad_norm": 0.1,
"clip_grad_type": "full",
"cudnn": {
"benchmark": false,
"deterministic": true
},
"distributed_strategy": "ddp",
"freeze": [],
"gpu_ids": [
0
],
"is_dry_run": false,
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1,
"optim": {
"backbone_multiplier": 0.1,
"gamma": 0.1,
"lr": 0.0002,
"lr_scheduler": "MultiStep",
"max_steps": 160000,
"milestones": [
88,
96
],
"momentum": 0.9,
"monitor_name": "val_loss",
"type": "AdamW",
"warmup_factor": 1.0,
"warmup_iters": 0,
"weight_decay": 0.05
},
"precision": "fp32",
"results_dir": "",
"resume_training_checkpoint_path": "",
"seed": 1234,
"val_check_interval": 5,
"validation_interval": 1,
"verbose": false
},
"description": "Configurable parameters to construct the trainer for the NVPanoptix3D experiment.",
"popular": [
"validation_interval",
"checkpoint_interval",
"num_gpus",
"num_nodes",
"num_epochs",
"gpu_ids"
],
"properties": {
"activation_checkpoint": {
"default": true,
"description": "\n A True value instructs train to recompute in backward pass to save GPU memory,\n rather than storing activations.",
"title": "enable activation checkpointing",
"type": "bool"
},
"checkpoint_2d": {
"default": "",
"description": "Path to 2D stage checkpoint to initialize the 3D stage training.",
"title": "2D stage checkpoint path",
"type": "string"
},
"checkpoint_3d": {
"default": "",
"description": "Path to 3D stage checkpoint to initialize the 3D stage training.",
"title": "3D stage checkpoint path",
"type": "string"
},
"checkpoint_interval": {
"default": 1,
"description": "The interval (in epochs) at which a checkpoint will be saved. Helps resume training.",
"minimum": 1,
"popular": true,
"title": "Checkpoint interval",
"type": "int"
},
"checkpoint_interval_unit": {
"default": "epoch",
"description": "The unit of the checkpoint interval.",
"enum": [
"epoch",
"step"
],
"title": "Checkpoint interval unit",
"type": "categorical"
},
"clip_grad_norm": {
"default": 0.1,
"description": "Amount to clip the gradient by L2 Norm.",
"math_cond": "> 0.0",
"title": "clip gradient norm",
"type": "float"
},
"clip_grad_type": {
"default": "full",
"description": "Gradient clip type.",
"title": "clip gradient type",
"type": "string"
},
"cudnn": {
"automl_enabled": false,
"default": {
"benchmark": false,
"deterministic": true
},
"properties": {
"benchmark": {
"default": false,
"type": "bool"
},
"deterministic": {
"default": true,
"type": "bool"
}
},
"type": "collection"
},
"distributed_strategy": {
"default": "ddp",
"description": "\n The multi-GPU training strategy.\n DDP (Distributed Data Parallel) and Fully Sharded DDP are supported.",
"enum": [
"ddp",
"fsdp"
],
"title": "distributed_strategy",
"type": "categorical"
},
"freeze": {
"automl_enabled": false,
"default": [],
"description": "\n List of layer names to freeze.\n Example: [\"backbone\", \"transformer.encoder\", \"input_proj\"].",
"title": "freeze",
"type": "list"
},
"gpu_ids": {
"automl_enabled": false,
"default": [
0
],
"description": "\n List of GPU IDs to run the training on. The length of this list\n must be equal to the number of gpus in train.num_gpus.",
"popular": true,
"title": "GPU IDs",
"type": "list"
},
"is_dry_run": {
"default": false,
"description": "Whether to run the trainer in Dry Run mode.",
"title": "Is dry run",
"type": "bool"
},
"iters_per_epoch": {
"description": "Number of iteration per epoch.",
"title": "iteration per epoch",
"type": "int"
},
"num_epochs": {
"default": 10,
"description": "Number of epochs to run the training.",
"maximum": Infinity,
"minimum": 1,
"popular": true,
"title": "Number of epochs",
"type": "int"
},
"num_gpus": {
"default": 1,
"description": "The number of GPUs to run the train job.",
"minimum": 1,
"popular": true,
"title": "Number of GPUs",
"type": "int"
},
"num_nodes": {
"default": 1,
"description": "Number of nodes to run the training on. If > 1, then multi-node is enabled.",
"minimum": 1,
"popular": true,
"title": "Number of nodes",
"type": "int"
},
"optim": {
"automl_default_parameters": [
"train.optim.lr"
],
"automl_disabled_parameters": [
"train.optim.milestones"
],
"automl_enabled": false,
"default": {
"backbone_multiplier": 0.1,
"gamma": 0.1,
"lr": 0.0002,
"lr_scheduler": "MultiStep",
"max_steps": 160000,
"milestones": [
88,
96
],
"momentum": 0.9,
"monitor_name": "val_loss",
"type": "AdamW",
"warmup_factor": 1.0,
"warmup_iters": 0,
"weight_decay": 0.05
},
"description": "Hyper parameters to configure the optimizer.",
"properties": {
"backbone_multiplier": {
"default": 0.1,
"description": "A multiplier for backbone learning rate.",
"math_cond": "> 0.0",
"maximum": 1.0,
"minimum": 0.01,
"title": "backbone learning rate multiplier",
"type": "float"
},
"gamma": {
"default": 0.1,
"description": "Multiplicative factor of learning rate decay.",
"math_cond": "> 0.0",
"title": "gamma",
"type": "float"
},
"lr": {
"automl_enabled": true,
"default": 0.0002,
"description": "The initial learning rate for training the model.",
"math_cond": "> 0.0",
"maximum": 0.01,
"minimum": 1e-06,
"title": "learning rate",
"type": "float"
},
"lr_scheduler": {
"default": "MultiStep",
"description": "The learning scheduler:\n * MultiStep : Decrease the lr by lr_decay from lr_steps\n * Warmuppoly : Poly learning rate schedule.",
"enum": [
"MultiStep",
"Warmuppoly"
],
"title": "learning rate scheduler",
"type": "categorical"
},
"max_steps": {
"default": 160000,
"description": "The maximum number of steps to train the model.",
"math_cond": "> 0",
"title": "max steps",
"type": "int"
},
"milestones": {
"automl_enabled": false,
"default": [
88,
96
],
"description": "learning rate decay epochs.",
"title": "learning rate decay epochs.",
"type": "list"
},
"momentum": {
"default": 0.9,
"description": "The momentum for the AdamW optimizer.",
"math_cond": "> 0.0",
"maximum": 0.999,
"minimum": 0.5,
"title": "momentum - AdamW",
"type": "float"
},
"monitor_name": {
"default": "val_loss",
"description": "The metric value to be monitored for the :code:`AutoReduce` Scheduler.",
"enum": [
"val_loss",
"train_loss"
],
"title": "monitor_name",
"type": "categorical"
},
"type": {
"default": "AdamW",
"description": "Type of optimizer used to train the network.",
"enum": [
"AdamW"
],
"type": "categorical"
},
"warmup_factor": {
"default": 1.0,
"description": "The warmup factor for the learning rate scheduler.",
"math_cond": "> 0.0",
"title": "warmup factor",
"type": "float"
},
"warmup_iters": {
"default": 0,
"description": "The number of warmup iterations.",
"math_cond": "> 0",
"title": "warmup iters",
"type": "int"
},
"weight_decay": {
"default": 0.05,
"description": "The weight decay coefficient.",
"math_cond": "> 0.0",
"maximum": 0.5,
"minimum": 0.0001,
"title": "weight decay",
"type": "float"
}
},
"title": "optimizer",
"type": "collection"
},
"precision": {
"default": "fp32",
"description": "Precision to run the training on.",
"enum": [
"fp16",
"fp32"
],
"title": "precision",
"type": "categorical"
},
"results_dir": {
"default": "",
"description": "The folder to save the experiment.",
"title": "results directory",
"type": "string"
},
"resume_training_checkpoint_path": {
"default": "",
"description": "Path to the checkpoint to resume training from.",
"title": "Resume checkpoint path",
"type": "string"
},
"seed": {
"default": 1234,
"description": "The seed for the initializer in PyTorch. If < 0, disable fixed seed.",
"maximum": Infinity,
"minimum": -1,
"title": "Seed for randomization",
"type": "int"
},
"val_check_interval": {
"default": 5,
"description": "The number of iterations between validation checks.",
"math_cond": "> 0",
"title": "val check interval",
"type": "int"
},
"validation_interval": {
"default": 1,
"description": "\n The interval (in epochs) at which a evaluation\n will be triggered on the validation dataset.",
"minimum": 1,
"popular": true,
"title": "Validation interval",
"type": "int"
},
"verbose": {
"default": false,
"description": "Flag to enable printing of detailed learning rate scaling from the optimizer.",
"title": "enable verbose logs",
"type": "bool"
}
},
"type": "collection"
},
"wandb": {
"automl_disabled_parameters": [
"wandb.tags"
],
"automl_enabled": false,
"default": {
"enable": true,
"entity": "",
"group": "",
"name": "TAO Toolkit Training",
"project": "TAO Toolkit",
"reinit": false,
"run_id": "",
"save_code": false,
"sync_tensorboard": false,
"tags": [
"tao-toolkit"
]
},
"properties": {
"enable": {
"default": true,
"type": "bool"
},
"entity": {
"default": "",
"type": "string"
},
"group": {
"default": "",
"type": "string"
},
"name": {
"default": "TAO Toolkit Training",
"type": "string"
},
"project": {
"default": "TAO Toolkit",
"type": "string"
},
"reinit": {
"default": false,
"type": "bool"
},
"run_id": {
"default": "",
"type": "string"
},
"save_code": {
"default": false,
"type": "bool"
},
"sync_tensorboard": {
"default": false,
"type": "bool"
},
"tags": {
"automl_enabled": false,
"default": [
"tao-toolkit"
],
"type": "list"
}
},
"type": "collection"
}
},
"type": "object",
"x_tao_schema": {
"action": "evaluate",
"core_module": "nvpanoptix3d",
"model": "nvpanoptix3d",
"network_arch": "nvpanoptix3d",
"schema_action": "evaluate",
"schema_version": 1,
"source": "tao-core dataclass config"
}
}
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Declares allowed tools, Needs a command-line tool, Needs Python, Needs Node.js
The author says: Requires docker + nvidia-container-toolkit.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Corporation
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/tao-train-nvpanoptix3d · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.
Related skills
Pytorch Lightning
K-Dense· Community
A framework for organizing PyTorch code to streamline neural network training, scaling, and experiment tracking.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Aeon
K-Dense· Community
A scikit-learn compatible Python toolkit for comprehensive time series machine learning tasks.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Umap Learn
K-Dense· Community
Integrate UMAP-learn for non-linear dimensionality reduction, visualization, and clustering preprocessing.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Scikit Learn
K-Dense· Community
Comprehensive guidance for classical machine learning workflows using the scikit-learn Python library.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Shap
K-Dense· Community
Explain and audit machine-learning model predictions using the SHAP framework for feature attribution and visualization.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Pufferlib
K-Dense· Community
Version-aware guidance for PufferLib reinforcement learning environments, vectorization, policies, and training.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-train-nvpanoptix3d.zip