AutoGineer
Generated banner for Tao Finetune Cosmos Reason
Generated logo for Tao Finetune Cosmos Reason

Tao Finetune Cosmos Reason

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Data and AI
  • Apache-2.0
  • Medium risk
  • Has scripts
Download ZIP

From the author

Shared Cosmos3 frontend that explicitly routes Cosmos Framework and Cosmos-RL, validates runtime model/video-dataset/SLURM inputs, consumes an SQSH or packaged backend image, optionally plans explicit clean source builds, prepares checkpoints, validates the first update in-process, and returns token-weighted losses and task-aware accuracy.
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-finetune-cosmos-reason skill provides a structured, unified interface for managing the lifecycle of Cosmos3 multimodal model training and evaluation. It handles complex routing between Cosmos Framework and Cosmos-RL backends, enforces strict input validation for datasets and model checkpoints, and manages the orchestration of training jobs on platforms like SLURM. The skill ensures reproducibility by sealing plans, validating runtime environments, and enforcing rigorous data and model contracts, while automating necessary conversions for Nano-tier architectures.

What it does

  • Routes training requests to the appropriate Cosmos Framework or Cosmos-RL backend.
  • Validates runtime inputs, including model paths, video datasets, and SLURM configurations.
  • Automates the conversion of Nano-tier checkpoints to required formats.
  • Executes training jobs with integrated validation of gradients and loss metrics.
  • Manages evaluation workflows, inheriting training configurations for consistency.
  • Ensures reproducibility through sealed plans and immutable reference resolution.

When to use it

  • When training or fine-tuning Cosmos3-Nano or Cosmos3-Edge models.
  • When you need to perform task-aware video reasoning or conversation-style training.
  • When orchestrating training jobs on SLURM clusters with specific container requirements.
  • When you require a reproducible, validated pipeline for model evaluation and metric extraction.

Advantages

  • Enforces strict reproducibility through sealed, immutable plan artifacts.
  • Simplifies backend selection by automatically routing based on the task type.
  • Provides robust validation of inputs and environment configurations before job submission.
  • Ensures consistency between training and evaluation phases by inheriting configurations.
  • Supports both dense and PEFT training modes with standardized parameter contracts.

Usage examples

“Train a Cosmos3-Nano model using the Cosmos-RL backend on my video conversation dataset.”
The agent will validate the dataset structure, resolve the base model, prepare the Nano checkpoint, and generate a sealed plan for the training job.
“Evaluate my fine-tuned Cosmos3-Edge model on the validation set.”
The agent will inherit the training configuration, run the evaluation workflow, and extract final metrics such as token-weighted loss and task accuracy.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/tao-finetune-cosmos-reason/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-finetune-cosmos-reason.zip -d .agents/skills

Documentation by NVIDIA Corporation

Cosmos3 TAO training

Keep one shared model-facing frontend. Backend image fields and contract paths live under backend_contracts in references/skill_info.yaml; image literals are stamped from versions.yaml, while referenced backend YAMLs define native runtime schemas. Never translate between them.

Mandatory runtime intake

Before planning training, collect all of the following. Do not infer a path from history, another user, a prior job, an image, or a developer checkout.

  • base_model_path_or_uri. For a Hugging Face model ID or URL, accept an optional friendly base_model_revision such as a branch or tag. If omitted, resolve main; do not ask the user for a commit SHA. Resolve the selected ref read-only through the Hub API to its immutable commit and seal the model ID, requested ref, and resolved SHA in the plan. A complete local snapshot needs no revision and is sealed by its file fingerprints.

  • For Cosmos3-Nano, an explicit input-checkpoint model_type: qwen3_vl or cosmos3_omni. If the user did not supply it, ask once before planning; never infer the choice from config.json, a model ID, a path name, or a previous run. Explain the two choices in plain language: qwen3_vl uses a compatible Hugging Face checkpoint directly, while cosmos3_omni requires an immutable conversion to exact Qwen3-VL safetensors before training. Record the answer as base_model_format. Cosmos3-Edge is inferred as cosmos3_edge from the resolved model ID and does not present this Nano-only choice.

  • Do not expose Omni preparation implementation fields during normal intake. For Nano, use Qwen/Qwen3-VL-8B-Instruct, resolve both Hub models to immutable commits, and run the selected backend's packaged converter. Framework uses cosmos_framework.scripts.convert_model_to_vlm_safetensors; Cosmos-RL uses cosmos_rl.model_preparation.vlm_safetensors with its pinned isolated converter runtime. Validate an explicitly supplied prepared_checkpoint_path or donor, but never ask for one by default.

  • Accept hf_model://nvidia/Cosmos3-Nano directly. If a gated/private model cannot be resolved, ask the user only to set HF_TOKEN in the session environment; never ask them to discover a SHA or provide the token value in chat.

  • explicit video sampling mode: either uniform nframes or fps. FPS mode may also set min_frames and max_frames; both modes may set clip-time, resize, and pixel-budget fields supported by the selected backend.

  • training/validation annotation paths and media roots for conversation-style or task-aware video supervision, plus optional task selection.

  • explicit backend for a comparison; cosmos-framework or cosmos-rl.

  • training_mode; dense or peft. PEFT also requires rank, alpha, dropout, target modules, bias, RS-LoRA, modules-to-save, and adapter precision.

  • user-owned results_dir, checkpoint_dir, cache_dir, and, for SLURM, sqsh_cache_dir, ssh_key_path, mounts, and scheduler settings.

  • Runtime order: compute-readable sqsh_path, explicit image, then the selected backend image in references/skill_info.yaml. On SLURM reuse or convert it once under sqsh_cache_dir. Never compare an SQSH filename with an image tag or request source provenance/SHA.

  • Repository paths, commits/trees, branch, base image, build context, and timestamp are advanced inputs required only for explicit source-build. See references/cosmos-backend-operations.md; never infer a build from runtime selection.

The planner preserves each original path and reports an accessible realpath. Missing required paths fail. A missing supplied SQSH fails; an omitted SQSH selects the packaged image. No historical fallback path or image is allowed.

Nano checkpoint model-type choice

Treat the selected base_model_format as a user decision and verify that it matches the supplied local checkpoint's config.json.model_type. A mismatch fails; it is not permission to relabel or rewrite the source checkpoint.

For qwen3_vl, fingerprint and use a complete compatible Hugging Face Nano checkpoint directly. For cosmos3_omni, tell the user in the launch review that Cosmos-RL will prepare its compatible checkpoint automatically, name the planned output path, and preserve the source checkpoint unchanged. Do not ask the user for an architecture donor, preparation image, or preparation SQSH. The backend-owned preparation step emits a verified qwen3_vl checkpoint under the selected platform's user-owned checkpoint_dir:

  • Docker: the local Docker host's checkpoint_dir.

  • SLURM: the compute-node-verified shared checkpoint_dir, covered by an explicit container mount. Run conversion through the SLURM/Pyxis contract; do not write the converted checkpoint to controller-local storage.

Use the packaged Nano architecture mapping and selected backend runtime unless an advanced override was explicitly supplied. Fingerprint the source, architecture mapping, converted config/tokenizer/processor/index/shards, and conversion provenance. Before conversion, bind the sealed plan's backend-native training fields and VLM_SAFETENSORS_PATH to the planned converted checkpoint inside the selected container. Validate that exact output before training without mutating the sealed plan. The original Omni/Hugging Face path remains provenance only and must never remain as the runtime model path. Reuse is allowed only when the exact target has complete matching conversion provenance and passes the same validation.

Public Cosmos3-Edge checkpoint contract

Accept the public model at its resolved immutable revision or a local snapshot; never request a second checkpoint. Apply the model-aware runtime defaults from references/skill_info.yaml and references/cosmos-framework-backend.yaml, preserving separate model and processor-profile fingerprints and each default/override origin.

Backend selection

Run scripts/cosmos_workflow.py resolve first.

RequestAutomatic selection
Cosmos3-Nano plain trainCosmos-RL (compatibility default)
Cosmos3-Nano AutoML/HPOCosmos-RL
Nano Framework-DCP exportCosmos Framework
Nano evaluate/inference/microservice with no explicit backendCosmos-RL
Nano quantizeCosmos-RL
Cosmos3-Edge train/export/evaluate/inference/microserviceCosmos Framework

An explicit supported backend wins, so users can select Cosmos Framework for Nano training without changing model ownership. Comparative runs reject auto, so both sides of an experiment are deliberately forced. Framework-trained checkpoints use the native exact-key exporter, then the repository-backed TAO evaluation adapter. That does not make Framework a Cosmos-RL version.

Evaluation intake and inheritance

Run scripts/evaluation_workflow.py for every evaluate action and follow references/cosmos-reason-evaluate.md completely. A parent training plan owns all inheritable model, dataset, prompt, preprocessing, precision, checkpoint, and scoring fields; never ask the user to repeat them. Ask once only for the helper's required_user_inputs, execute its automated_actions, and launch only a checksum-valid ready=true plan. Cosmos-RL policy checkpoints require the emitted cosmos_rl_checkpoint_pre_action; Framework DCP inputs require their emitted export pre-action. The ready plan includes a validated spec_bundle.execution; pass it unchanged to the selected platform. Cosmos owns runtime attestation and evaluator configuration; the platform owns launch.

Framework checkpoint pre-action

Before Framework evaluate, inference, or microservice actions, run scripts/framework_checkpoint_action.py plan and its emitted prepare and verify steps. Follow references/cosmos-backend-operations.md; never ask the user to export DCP manually. On SLURM stage only the helper dependency set declared by workflow_contract.action_helper_dependencies; the platform verifies the closed bundle. Use only the verified action_model_path, reuse only matching complete exports, and record the pre-action, manifest, fingerprints, and independent child result.

Required gates

Execute these stages in order and persist their outputs.

  1. Resolve model/backend/action and load the selected backend contract.

  2. Check credentials by presence only. Never read or persist credential values. Require a token only for the operation that needs it.

  3. Validate tools, storage, paths, and runtime selection in the mandatory intake order; existing-SQSH and packaged-image modes skip source gates.

  4. Only for explicit source-build, validate/build clean sources and verify /opt/tao/image-provenance.json. Never mount host source into training.

  5. Enforce the explicit Nano checkpoint model-type choice. If it is cosmos3_omni, show the conversion and platform-owned output path in the launch review, then prepare the model through the shared TAO integration entrypoint packaged in the selected clean backend image after approval. Resolve URI/model-ID refs to immutable Hub commits automatically. Validate exact tensor/config keys and fingerprint model, tokenizer, processor, weight index, every shard, and provenance. Assign the verified converted path—not the original source path—to training.

  6. Validate inputs, counts, duplicates, overlap, tasks, and identities. A no-byte-hashing request selects both fast fingerprint flags; metadata stays hashed while weight/media payloads use path+size. Verify the resolved inputs again from an allocated compute node. When SLURM storage is not mounted on the launch host, let cosmos_workflow.py stream its checked-in cosmos_common.py inspector to a login host over SSH. It runs from stdin, preserves remote realpath values, and creates no remote script or source overlay. Do not require local Lustre, sbatch, or srun on an SSH-based launch host. Run this expensive input inspection exactly once with the plan verb and pass a local --plan-artifact <path> so the resolved request and inspection results are sealed for the remaining launch verbs.

  7. Resolve the video runtime from the structural dataset contract and enforce every profile gate in references/cosmos-reproducibility-gates.md. Cosmos-RL auto selects source-baked pynv-device-rgbp for video_conversation and system-pyav for task_aware_video_reasoning; Framework selects torchcodec-cuda-on-demand. All defaults use bounded rank-local, on-demand memory with no disk prewarm. Repeated-media validation may use the backend-native grouped sharder and validation-only caches only while preserving records, explicit batch, weighting, prompts, and preprocessing. Throughput settings that relax batch-1 parity require user authorization.

  8. Generate backend-native TOML, environment, topology, preflight commands, parity data, runtime profile, and metadata. Full specs contain no sample limit. Reuse one sealed --plan-artifact across read-only preflight, post-review materialize, and render-slurm; do not repeat original inputs. Materialize atomically in the verified compute frame, derive container paths only from explicit mounts, and never copy source patches to the cluster. Diagnostic subsets are explicit opt-ins, never launch prerequisites.

  9. On SLURM reuse the supplied/derived SQSH or convert the selected image once before GPU submit. SQSH SHA and source provenance are not runtime gates. When Omni preparation is required, inspect the SQSH filesystem and reject it unless it contains the shared TAO launcher, the native Framework converter, and /opt/tao/framework-converter-runtime.json. The runtime artifact must report validation_mode=imported_converter_module, proving the isolated converter's transitive dependency graph imported during the image build; file presence alone is not sufficient. Verify Pyxis/Enroot, mounts, Python/packages, decoder, GPU, CUDA, NCCL, and storage in the training allocation.

  10. Launch the requested training job directly; do not submit a separate smoke job unless the user explicitly requests one. For Cosmos-RL VLM training, require packaged source that emits a padding-aware attention_mask and runs the visual-gradient contract after the first backward pass and before the first optimizer update. Persist total/trainable/frozen counts and gradient norms for the vision encoder, visual projector, language model, and language head. Fail immediately when a trainable visual component has no gradient, a non-finite norm, or a zero norm. Report an explicitly frozen visual component as not applicable rather than as a failure.

  11. Materialize the full spec once and verify its SHA256 in the compute frame before rendering the job from the same plan artifact. Monitor scheduler and structured TAO state to a terminal result, and preserve the child exit code independently of scheduler state. Require child exit zero, structured SUCCESS, finite global train/validation loss, checkpoint completion, and a final evaluator metric before reporting completion.

  12. Resolve evaluation with scripts/evaluation_workflow.py. Inherit exact fine-tuning artifacts, collect only its remaining user inputs, run its backend-owned automated checkpoint pre-actions, and require ready=true. On Cosmos-RL, verify the selected HF export with cosmos_rl_checkpoint_action.py, rerun resolution with its manifest, and submit the emitted spec-bundle through the chosen platform; never select the native policy directory, copy the lifecycle into an application-owned launcher, or improvise result/status variables. Evaluate the selected checkpoint with identical prompt, preprocessing, generation, normalization, and task scoring. Extract final metrics with scripts/extract_cosmos_metrics.py.

Dataset contracts

Resolve datasets by structure, not by project, benchmark, directory, or file name. The supported families are:

  • video_conversation: a JSON array with media and at least two ShareGPT, LLaVA, or OpenAI-style conversation turns;

  • task_aware_video_reasoning: one or more item-envelope or array annotation files with media, task identity, and conversation/response targets.

Default dataset_family to auto, inspect every annotation, and require train and validation to resolve to the same family. Capture record count, unique media count, media reuse, extensions, byte-size distribution, task/metric metadata, and any declared width, height, FPS, and duration. Select processor, cache and resource profiles from those characteristics and the model tier. Never branch on a customer dataset name.

Tasks declaring accuracy participate in deterministic accuracy; common binary and multiple-choice task types are recognized. Generative tasks report their declared metrics and are excluded from aggregate accuracy with a reason. Aggregate accuracy is example-weighted over records with an accuracy definition.

Dense and PEFT contracts

Dense SFT has no LoRA block and reports trainable, frozen, and total counts. PEFT preserves rank, alpha, dropout, targets, bias, RS-LoRA, saved modules, precision, and trainable count across backends; mismatched semantics block a pair.

For fair comparisons, force the same logical model, train/validation records, media, prompt, frames, sequence length, precision, seed, epochs, effective global batch, optimizer, learning rate, schedule, warmup, weight decay, clipping, loss masking, validation/checkpoint cadence, evaluated checkpoint, and generation/normalization settings. Classify differences as equivalent syntax, unavoidable implementation difference, or invalid mismatch; an invalid mismatch blocks the full pair.

Metrics and completion

The required primary metrics are:

  • complete-run globally reduced token-weighted training loss, with numerator and valid-label denominator;

  • final-validation globally reduced token-weighted loss, with numerator and valid-label denominator;

  • the repository evaluator's final validation metric and any supporting values it emits. Do not add a second post-evaluation gate.

Do not average console lines or rank means. A step loss is not the average training loss. A validation heartbeat is not final validation loss. A generative exact match is not accuracy unless the task defines it.

The native Framework callback and native Cosmos-RL logger own early failure, checkpoint, progress, metric, and terminal events. Do not stage a status bridge or patch status at container startup. COMPLETED from SLURM is failure when the child exit code is nonzero or the terminal TAO state is not successful.

SLURM invariants

Generated jobs use Bash, SQSH via Pyxis, no requeue by default, one launcher task per node, runtime-supplied logs, and the training child exit code. A single-node exclusive allocation preserves the requested CPU count in the SBATCH contract but passes every CPU actually granted by SLURM to the training step and records requested, allocated, and step counts. Before the training child, the same allocation runs the planner-owned packaged-runtime gate as the training job's first Pyxis step; a failed gate writes the independent child exit artifact and blocks training. Generated jobs set SLURM_EXPORT_ENV=ALL, and the platform consumer still submits with sbatch --export=ALL. Every Pyxis step also receives the planner-owned runtime variable names through --container-env, so a baked image ENV value cannot override the selected decoder, frame-transfer, cache, or worker profile. Framework topology is shard=gpus_per_node, replica=nodes. Cosmos-RL uses one controller on node zero and its policy-worker topology. Asynchronous distributed checkpointing is rejected for multi-node runs.

Every job metadata record must validate against schemas/cosmos-job-metadata.schema.json and contain paths, fingerprints, runtime identity, config, resources, states, outputs, and timing without credentials. Source provenance exists only for source-build; SQSH SHA is optional and never a runtime gate.

Source-affecting recovery

If a run exposes a code or image defect, stop the affected path, change the owning repository, add a test, commit it, rebuild both image and SQSH from a clean checkout, then restart every affected training job from its clean sealed plan. Never edit a running container, patch an existing image, reuse an old SQSH after a source change, or rely on a temporary launch script as the implementation.

Use references/cosmos-reproducibility-gates.md as the source-owner/test map.

For infrastructure retries, the launch skill classifies the failure and opens the new --retry-of record; SLURM supplies validated node inventory and exclusions. Run cosmos_workflow.py retry-plan with the new record's <action-root>/config/train.toml; it rebases all writable paths and reseals the Cosmos request. Render that plan; never patch SBATCH.

Files

references/skill_info.yaml

network_arch: cosmos-reason
huggingface_model_ids:
- nvidia/Cosmos3-Nano
- nvidia/Cosmos3-Edge
aliases:
- Cosmos Reason 3
- Cosmos3 Nano Reasoner
- Cosmos3-Nano
- Cosmos3-Edge
- cosmos-rl
- cosmos-framework
automl_enabled: true
backend_contracts:
  cosmos-framework:
    path: references/cosmos-framework-backend.yaml
    container_image: nvcr.io/nvidia/tao/tao-toolkit:7.2.0-cosmos-framework  # versions-key: images.tao_toolkit.cosmos_framework
  cosmos-rl:
    path: references/cosmos-rl-backend.yaml
    container_image: nvcr.io/nvidia/tao/tao-toolkit:7.2.0-cosmos-rl  # versions-key: images.tao_toolkit.cosmos_rl
backend_resolver: scripts/cosmos_workflow.py
backend_selection:
  policy: explicit backend wins; otherwise select by model, action, and workflow and report the rationale
  defaults:
    train: cosmos-rl
    export: cosmos-framework
    evaluate: cosmos-rl
    inference: cosmos-rl
    inference_microservice: cosmos-rl
    quantize: cosmos-rl
  rules:
  - match: Cosmos3-Edge train
    backend: cosmos-framework
  - match: Cosmos3-Edge export, evaluate, inference, or inference_microservice
    backend: cosmos-framework
  - match: Cosmos3-Nano AutoML or HPO train
    backend: cosmos-rl
  - match: Cosmos3-Nano plain SFT train
    backend: cosmos-rl
  - match: Cosmos3-Nano evaluate, inference, inference_microservice, or quantize
    backend: cosmos-rl
  ambiguity_behavior: preserve the Cosmos-RL compatibility default and emit backend and backend_selection_reason in the launch review
required_credentials: []
conditional_credentials:
  HF_TOKEN: required only when fetching a gated Hugging Face checkpoint; not required for a complete local checkpoint
  NGC_KEY: required only when pulling or converting an authenticated nvcr.io image; not required for an existing local image or SQSH
data_format: llava
gpu_spec_key: policy.parallelism.dp_shard_size
node_spec_key: policy.parallelism.dp_replicate_size
runtime_requirements:
  gpu_host:
    min_driver_version: '580'
    min_cuda_version: '13.0'
    min_container_toolkit_version: '1.19.1'
workflow_contract:
  planner: scripts/cosmos_workflow.py
  nano_checkpoint_intake:
    explicit_user_choice_required: true
    choices:
      qwen3_vl: direct_compatible_huggingface_checkpoint
      cosmos3_omni: backend_packaged_exact_key_conversion_to_qwen3_vl_before_training
    inference_from_checkpoint_or_history: forbidden
    conversion_storage:
      docker: user_owned_checkpoint_dir_on_docker_host
      slurm: compute_verified_shared_checkpoint_dir
    runtime_model_path: verified_direct_or_converted_qwen3_vl_path
    source_checkpoint_mutation: forbidden
    nano_architecture_mapping:
      model_id: Qwen/Qwen3-VL-8B-Instruct
      user_input_required: false
      revision_resolution: automatic_immutable_hub_commit
    preparation_runtime:
      source: selected_backend_image_and_sqsh
      separate_user_input: forbidden
      entrypoints:
        cosmos-framework: cosmos_framework.scripts.convert_model_to_vlm_safetensors
        cosmos-rl: cosmos_rl.model_preparation.vlm_safetensors
      sqsh_required_entries:
        cosmos-framework:
          - cosmos_framework/scripts/convert_model_to_vlm_safetensors.py
        cosmos-rl:
          - cosmos_rl/model_preparation/vlm_safetensors.py
          - cosmos_framework/scripts/convert_model_to_vlm_safetensors.py
          - opt/tao/framework-converter-runtime.json
    huggingface_revision_resolution:
      user_commit_sha_required: false
      default_requested_ref: main
      optional_user_ref: branch_tag_or_commit
      resolved_value: immutable_hub_commit_sha
      local_snapshot_identity: content_fingerprint
      private_model_blocker: request_HF_TOKEN_presence_only
  framework_checkpoint_action: scripts/framework_checkpoint_action.py
  action_helper_dependencies:
    framework_checkpoint:
      entrypoint: scripts/framework_checkpoint_action.py
      files:
      - scripts/framework_checkpoint_action.py
      - scripts/cosmos_common.py
    cosmos_rl_checkpoint:
      entrypoint: scripts/cosmos_rl_checkpoint_action.py
      files:
      - scripts/cosmos_rl_checkpoint_action.py
      - scripts/cosmos_common.py
    staging_contract: tao-artifacts.spec_bundle.execution.supporting_files
    staging_owner: selected_platform
  metric_extractor: scripts/extract_cosmos_metrics.py
  diagnostic_subset:
    policy: opt-in only; never submit a separate smoke job unless the user explicitly requests it
    train_samples: max(16, total_gpus)
    validation_samples: max(8, total_gpus)
  first_update_gate:
    execution: inside the requested Cosmos-RL training job after backward and before the first optimizer update
    requires:
    - padding-aware attention_mask emitted by the HF VLM collator
    - component total and trainable parameter counts
    - finite nonzero gradient norm for each trainable visual component
    status_output: vision encoder, visual projector, language model, and language-head counts and gradient norms
    failure_behavior: terminate the training child before the first optimizer update
  parity_fingerprint:
  - train record and media fingerprints
  - validation record and media fingerprints
  - original and resolved runtime paths
  - system prompt
  - checkpoint identity and format
  - frame count
  - optimizer, scheduler, batch size, epochs, and random seed
  - backend and container digest
  public_model_profiles:
    cosmos3_edge:
      checkpoint_input: public Hugging Face model ID with automatically resolved immutable commit or complete local snapshot
      alternate_workload_checkpoint: false
      checkpoint_mutation: false
      runtime_processor_defaults:
        frames: 6
        frame_width: 1280
        frame_height: 720
        max_video_pixels: 5529600
        sequence_length: 16000
        attention_implementation: flash_attention_2
      provenance: base checkpoint and runtime processor profile have independent fingerprints
  dataset_resolution:
    default: auto
    families: [video_conversation, task_aware_video_reasoning]
    selection_inputs:
    - annotation structure and metadata
    - record and unique-media counts
    - media reuse ratio and byte-size distribution
    - declared resolution, FPS, and duration
    - model tier and explicit user overrides
    forbidden_routing_inputs: [dataset project name, benchmark name, directory basename, historical path]
  cosmos_rl_video_runtime:
    selector: --rl-video-profile
    choices: [auto, system-pyav, pynv-device-rgbp]
    auto:
      video_conversation: pynv-device-rgbp
      task_aware_video_reasoning: system-pyav
    pynv-device-rgbp:
      decoder: pynvvideocodec
      transfer: device_rgbp_dlpack
      dataloader: {workers: 1, prefetch_factor: 2, persistent_workers: true}
      sft_batch_threads: 4
      processed_video_cache: rank_local_memory_populated_on_demand
      decoder_session_cache: rank_local_native_populated_on_demand
      worker_decoder_cache_forwarding: TAO_PYNV_DECODER_CACHE_SIZE_required
      processed_video_cache_capacity_default: inspected_unique_media_capacity
      decoder_session_cache_capacity_default: inspected_unique_media_capacity
      prewarm: false
      disk_cache: false
    validation:
      selection_inputs: [dataset_family, validation_record_count, validation_unique_media_count]
      auto_repeated_media_condition: video_conversation_and_record_count_greater_than_unique_media_count
      shard_strategy: media_grouped
      padded_multiset: preserved_exactly
      video_feature_cache: rank_local_gpu_embeddings_populated_on_demand
      feature_cache_capacity_default: min_inspected_unique_media_capacity_512
      feature_cache_environment: TAO_VALIDATION_VIDEO_FEATURE_CACHE_SIZE
      exact_validation_batch: preserved_from_user_contract
      maximum_throughput_overrides:
        validation_batch_size: 16
        cache_frontload_batch_size: 16
        cache_frontload_unique_per_batch: 8
      prewarm: false
      disk_cache: false
      compatibility_artifact: optional_and_never_a_direct_launch_gate
      capability_fallback: tao_system_pyav_sparse
      capability_fallback_scope: nvdec_incompatible_stream_only
    system-pyav:
      decoder: torchvision
      implementation: sparse_software_system_PyAV
      dataloader: {workers: 0, prefetch_factor: omitted}
      sft_batch_threads: 1
      prewarm: false
      disk_cache: false
  cosmos_framework_video_runtime:
    profile: torchcodec-cuda-on-demand
    decoder: torchcodec
    device: cuda
    device_binding: explicit_local_rank
    frame_transfer: cuda_uint8_to_host_pil
    dataloader:
      workers: 1
      prefetch_factor:
        video_conversation: 4
        task_aware_video_reasoning: 2
      multiprocessing_context: spawn
      persistent_workers: true
      pin_memory: true_for_video_conversation
    sft_process_threads: 8
    decoder_threads: 1
    decoded_frame_cache: rank_local_memory_populated_on_demand
    capacity_default: inspected_unique_media_count
    nano_per_forward_batch_default: effective_global_batch_divided_by_total_gpus
    nano_gradient_accumulation_default: 1
    batcher: resume_safe_contiguous
    actual_device_attestation: first_successful_decode_per_rank
    actual_device_requirement: resolved_device_equals_cuda_local_rank
    prewarm: false
    disk_cache: false
    validation:
      selection_inputs: [dataset_family, validation_record_count, validation_unique_media_count]
      auto_repeated_media_condition: video_conversation_and_record_count_greater_than_unique_media_count
      shard_strategy: media_grouped
      padded_multiset: preserved_exactly
      finite_stream: true
      video_feature_cache: rank_local_gpu_embeddings_populated_on_demand
      feature_cache_capacity_default: min_inspected_unique_media_capacity_512
      feature_cache_environment: TAO_FRAMEWORK_VALIDATION_VIDEO_FEATURE_CACHE_SIZE
      processed_video_cache: worker_local_host_tensors_populated_on_demand
      processed_video_cache_capacity_default: inspected_unique_media_capacity
      processed_video_cache_environment: TAO_FRAMEWORK_VALIDATION_PROCESSED_VIDEO_CACHE_SIZE
      exact_validation_batch: preserved_from_user_contract
      maximum_throughput_frontload:
        selection_condition: repeated_media_and_user_selected_validation_batch_at_least_2
        unique_per_batch_default: min_8_and_half_validation_batch
        environment: TAO_FRAMEWORK_VALIDATION_CACHE_FRONTLOAD_UNIQUE_PER_BATCH
      deferred_metric_scalar_transfer: validation_end
      prewarm: false
      disk_cache: false
  backend_action_commands:
    cosmos-framework:
      export: python -m cosmos_framework.scripts.export_vlm_dcp
      evaluate: cosmos-framework-evaluate --config {config_path}
      inference: cosmos-framework-inference <arguments>
      inference_microservice: cosmos-framework-inference-microservice <arguments>
      required_pre_action: scripts/framework_checkpoint_action.py prepare
    cosmos-rl:
      evaluate: cosmos-rl-evaluate --config {config_path}
      inference: cosmos-rl-inference <arguments>
      inference_microservice: cosmos-rl-inference-microservice <arguments>
      quantize: cosmos-rl-quantize <arguments>
data_sources:
  train:
    custom.train_dataset.annotation_path:
      source: train_datasets
      multiple_sources: false
      path: '{train_dataset_annotation}'
    custom.train_dataset.media_path:
      source: train_datasets
      multiple_sources: false
      path_from_format:
        llava:
        - images.tar.gz
        - videos.tar.gz
        '*': images.tar.gz
    custom.val_dataset.annotation_path:
      source: eval_dataset
      multiple_sources: false
      path: '{eval_dataset_annotation}'
    custom.val_dataset.media_path:
      source: eval_dataset
      multiple_sources: false
      path_from_format:
        llava:
        - images.tar.gz
        - videos.tar.gz
        '*': images.tar.gz
  evaluate:
    dataset.annotation_path:
      source: eval_dataset
      multiple_sources: false
      path: '{eval_dataset_annotation}'
    dataset.media_dir:
      source: eval_dataset
      multiple_sources: false
      path_from_format:
        llava:
        - images.tar.gz
        - videos.tar.gz
        '*': videos.tar.gz
  inference:
    media:
      source: inference_dataset
      multiple_sources: false
      path_from_format:
        llava:
        - videos.tar.gz
        - images.tar.gz
        '*': videos.tar.gz
  quantize:
    dataset.annotation_path:
      source: calibration_dataset
      multiple_sources: false
      path: '{calibration_dataset_annotation}'
    dataset.media_dir:
      source: calibration_dataset
      multiple_sources: false
      path_from_format:
        llava:
        - videos.tar.gz
        - images.tar.gz
        '*': videos.tar.gz
spec_params:
  evaluate:
    model.model_name: parent_model_folder
    results_dir: output_dir
  inference:
    model_path: parent_model_folder
    results_dir: output_dir
  quantize:
    model.model_path: parent_model_folder
    results_dir: output_dir
  train:
    train.output_dir: output_dir
    train.resume: resume_model
actions:
  evaluate:
    command: cosmos-rl-evaluate --config {config_path}
    config_format: toml
    mode: config
    execution_contract:
      source: scripts/evaluation_workflow.py output spec_bundle.execution
      platform_owned: [helper_staging, container_mounts, distributed_launcher, timeout, child_exit_preservation]
      model_owned: [runtime_attestation, evaluator_command, metric_extraction, structured_success]
    automl_default_parameters:
    - dataset.system_prompt
    - vision.num_frames
    - generation.max_tokens
    - generation.temperature
    - generation.repetition_penalty
    - generation.presence_penalty
    - generation.frequency_penalty
    inputs:
      dataset.annotation_path:
        type: file
      dataset.media_dir:
        type: folder
        selective:
          annotation: dataset.annotation_path
          format: json
          keys:
          - video
      model.model_name:
        type: folder
    outputs:
      results_dir:
        type: folder
    upload_excludes:
    - inputs/
  inference:
    command: |
      python - {config_path} <<'PY'
      import subprocess
      import sys
      import tomllib

      cfg = tomllib.load(open(sys.argv[1], "rb"))
      args = [
          "cosmos-rl-inference",
          "--model_path", cfg["model_path"],
          "--torch_dtype", cfg.get("torch_dtype", "auto"),
          "--device_map", cfg.get("device_map", "auto"),
          "--type", cfg.get("type", "video"),
          "--media", cfg["media"],
          "--prompt", cfg["prompt"],
          "--max_new_tokens", str(cfg.get("max_new_tokens", 4096)),
          "--results_dir", cfg["results_dir"],
          "--enable_lora", str(cfg.get("enable_lora", False)).lower(),
      ]
      if cfg.get("num_gpus") not in (None, ""):
          args.extend(["--num_gpus", str(cfg["num_gpus"])])
      if cfg.get("fps") not in (None, ""):
          args.extend(["--fps", str(cfg["fps"])])
      if cfg.get("num_frames") not in (None, ""):
          args.extend(["--num_frames", str(cfg["num_frames"])])
      if cfg.get("total_pixels") not in (None, ""):
          args.extend(["--total_pixels", str(cfg["total_pixels"])])
      if cfg.get("base_model_path"):
          args.extend(["--base_model_path", cfg["base_model_path"]])
      raise SystemExit(subprocess.call(args))
      PY
    config_format: toml
    mode: config
    inputs:
      model_path:
        type: folder
      media:
        type: folder
    outputs:
      results_dir:
        type: folder
    upload_excludes:
    - inputs/
  quantize:
    command: |
      python - {config_path} <<'PY'
      import sys
      import tomllib
      import types

      cfg = tomllib.load(open(sys.argv[1], "rb"))
      model = cfg.get("model", dict())
      dataset = cfg.get("dataset", dict())
      quantize = cfg.get("quantize", dict())

      args = [
          "--model_path", model["model_path"],
          "--results_dir", cfg["results_dir"],
          "--num_calibration_samples", str(quantize.get("num_calibration_samples", 512)),
          "--max_sequence_length", str(quantize.get("max_sequence_length", 2048)),
          "--quantization_scheme", quantize.get("quantization_scheme", "FP8_DYNAMIC"),
          "--kv_precision", quantize.get("kv_precision", "bf16"),
          "--smoothing_strength", str(quantize.get("smoothing_strength", 0.8)),
          "--skip_test_generation", str(quantize.get("skip_test_generation", False)).lower(),
          "--enable_lora", str(model.get("enable_lora", False)).lower(),
      ]
      if model.get("base_model_path"):
          args.extend(["--base_model_path", model["base_model_path"]])
      if dataset.get("dataset_id"):
          args.extend(["--dataset_id", dataset["dataset_id"]])
      if dataset.get("dataset_split"):
          args.extend(["--dataset_split", dataset["dataset_split"]])
      if dataset.get("annotation_path"):
          args.extend(["--annotation_path", dataset["annotation_path"]])
      if dataset.get("media_dir"):
          args.extend(["--media_dir", dataset["media_dir"]])

      sys.path.insert(0, "/opt/quantize_deps")
      import compressed_tensors.utils.match as match
      if not hasattr(match, "_match_name") and hasattr(match, "match_name"):
          match._match_name = match.match_name
      from compressed_tensors.config import CompressionFormat
      format_mod = types.ModuleType("compressed_tensors.config.format")

      def _get_quant_compression_format(input_activations=None, weights=None):
          qtype = getattr(weights, "type", None) or getattr(input_activations, "type", None)
          if str(qtype).lower() == "float":
              return CompressionFormat.float_quantized
          return CompressionFormat.int_quantized

      format_mod._get_quant_compression_format = _get_quant_compression_format
      sys.modules["compressed_tensors.config.format"] = format_mod
      sys.argv = ["cosmos-rl-quantize"] + args
      from cosmos_rl.quantization.quantize import main
      raise SystemExit(main())
      PY
    config_format: toml
    mode: config
    inputs:
      model.model_path:
        type: folder
      dataset.annotation_path:
        type: file
      dataset.media_dir:
        type: folder
    outputs:
      results_dir:
        type: folder
    upload_excludes:
    - inputs/
  train:
    command: |
      hook="$(/opt/venv/cosmos_rl/bin/python -c 'import cosmos_rl; from pathlib import Path; print(Path(cosmos_rl.__file__).parent / "tools" / "custom_hooks" / "tao_sft_example.py")')"
      test -f "$hook"
      cosmos-rl --config {config_path} "$hook"
    config_format: toml
    mode: config
    inputs:
      custom.train_dataset.annotation_path:
        type: file
      custom.train_dataset.media_path:
        type: folder
      custom.val_dataset.annotation_path:
        type: file
      custom.val_dataset.media_path:
        type: folder
    outputs:
      train.output_dir:
        type: folder
    upload_excludes:
    - inputs/
spec_shorthand_keys:
  dp_shard_size: policy.parallelism.dp_shard_size
  dp_replicate_size: policy.parallelism.dp_replicate_size
  num_epochs: train.epoch
  batch_size: train.train_batch_per_replica
  learning_rate: train.optm_lr
  mini_batch: train.train_policy.mini_batch
  weight_decay: train.optm_weight_decay
  warmup_epochs: train.optm_warmup_epochs

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python, Needs Node.js

The author says: Requires Python 3.11+ with PyYAML and a supported container execution platform; SLURM runs additionally require SSH, sbatch/srun, Pyxis/Enroot, and shared storage.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-finetune-cosmos-reason · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-finetune-cosmos-reason.zip