Physical AI Infrastructure Setup And Resilient Scaling
NVIDIAOfficialCreated by NVIDIA Physical AI
- Skill
- Workflow
- Cloud and DevOps
- Apache-2.0
- Medium risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a comprehensive framework for deploying, scaling, and validating the infrastructure required for NVIDIA Physical AI synthetic data generation (SDG) workflows. It manages the lifecycle of Kubernetes clusters (MicroK8s or Azure AKS), OSMO orchestration, and inference backends (NIM Operator, NVCF, or Azure AI Foundry). The skill enforces strict preflight checks, reproducible configuration management, and systematic recovery procedures to ensure reliable workload execution.
What it does
- Provisions and configures Kubernetes clusters on MicroK8s or Azure AKS.
- Deploys and manages OSMO orchestration layers for workload submission.
- Sets up inference endpoints using NIM Operator, NVCF, or Azure AI Foundry.
- Executes mandatory preflight checks to validate infrastructure readiness.
- Provides structured recovery paths for failed workflow states or infrastructure issues.
- Ensures secure credential handling and environment-aware configuration.
When to use it
- When setting up a new Physical AI SDG environment from scratch.
- When scaling existing infrastructure to meet new workload demands.
- When validating the health and readiness of cluster, inference, or OSMO components.
- When troubleshooting and recovering from infrastructure or workflow deployment failures.
Advantages
- Standardizes infrastructure deployment across different cloud and local environments.
- Reduces configuration drift by using checked-in scripts and reproducible patterns.
- Improves reliability through mandatory preflight and verification gates.
- Provides a unified interface for complex multi-component stacks.
Usage examples
- “Set up resilient Physical AI infrastructure for VDA on Azure AKS with NIM Operator.”
- The agent will initiate the setup flow for Azure AKS, configure the necessary OSMO and NIM Operator components, run preflight checks, and prepare the environment for VDA workload submission.
- “My MicroK8s cluster for SDG is failing to schedule jobs; help me validate and recover.”
- The agent will guide you through the verification gates for the MicroK8s cluster, inspect OSMO pod status, check storage credentials, and identify the root cause of the scheduling failure.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/physical-ai-infrastructure-setup-and-resilient-scaling/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip physical-ai-infrastructure-setup-and-resilient-scaling.zip -d .agents/skills
Documentation by NVIDIA Physical AI
Physical AI Infrastructure Setup And Resilient Scaling
Canonical skill for the Physical AI infrastructure stack. Use it to compose cluster, inference, OSMO, and workload stages into a reproducible Physical AI SDG environment, then keep the environment observable and recoverable.
Operating Rules
Read only the component references needed for the selected target. Do not load every component by default.
Keep the repo as the durable artifact. Fix checked-in config or scripts, then rerun. Do not recover a failed install with untracked one-off changes.
Run mutating cluster, OSMO, Helm, Terraform, or Azure operations through checked-in scripts when a script exists. Read-only diagnostics are allowed.
Stop at the first red gate. Fix the lowest owning layer in this order: config, script, then skill guidance.
Derive values from the environment when possible. Ask only for values that cannot be inferred, such as API keys, target choice, or quota tradeoffs.
Store secrets in
${REPO_ROOT}/.env. Cluster-derived values such as storage, database, Redis, and endpoint names come from Terraform outputs or platform queries, not.env.Preflight means no deployed state: no cluster API, Terraform outputs, Helm releases, OSMO pools, or workflow state. Those belong to deploy/verify gates.
Never print, echo, or paste raw keys into commands, YAML, logs, or transcripts. Prefer credential handles, Kubernetes
secretKeyRef, and runtime-only secret injection. Scan raw transcript exports withscripts/scan_transcript_secrets.pybefore sharing.Use absolute paths. Derive repo root with
git rev-parse --show-toplevel.
Component References
Each component lives inside this skill so the stack has one canonical trigger. Load the component reference only when the selected target needs that slice.
| Concern | Load | Assets |
|---|---|---|
| Stage matrix and old driver notes | components/driver/reference.md | None |
| MicroK8s cluster | components/cluster-microk8s/reference.md | components/cluster-microk8s/scripts/, components/cluster-microk8s/runtimeclass-nvidia-runc.yaml |
| Azure AKS cluster | components/cluster-azure/reference.md | components/cluster-azure/scripts/, components/cluster-azure/terraform/ |
| NIM Operator inference | components/inference-nim-operator/reference.md | components/inference-nim-operator/scripts/, components/inference-nim-operator/nims/ |
| NVCF inference | components/inference-nvcf/reference.md | components/inference-nvcf/scripts/ |
| Azure AI Foundry inference | components/inference-azure/reference.md | components/inference-azure/scripts/ |
| MicroK8s OSMO | components/osmo-k8s/reference.md | components/osmo-k8s/scripts/, upstream OSMO deploy scripts |
| Azure OSMO | components/osmo-azure/reference.md | components/osmo-azure/scripts/, upstream OSMO deploy scripts plus Azure TF outputs |
| Azure access setup | components/azure-access/reference.md | None |
| OSMO CLI and workflow operations | components/osmo-cli/reference.md | components/osmo-cli/scripts/, components/osmo-cli/references/, components/osmo-cli/agents/, components/osmo-cli/tests/ |
| OpenClaw Azure device login | components/openclaw-azure-login/reference.md | None |
OSMO CLI Support Files
The OSMO CLI component has second-level support files because its command and workflow surface is large. Load these directly only for the stated case.
| File | Read when |
|---|---|
components/osmo-cli/agents/workflow-expert.md | Spawning a workflow-generation or workflow-failure subagent. |
components/osmo-cli/agents/logs-reader.md | Spawning a log summarization subagent for OSMO workflow failures. |
components/osmo-cli/references/cli-commands.md | Exact OSMO CLI flags, payloads, or command syntax are needed. |
components/osmo-cli/references/workflow-spec.md | Workflow YAML schema, credentials, outputs, or provider fields are needed. |
components/osmo-cli/references/workflow-patterns.md | Multi-task, data dependency, Jinja, serial, or parallel workflow design is needed. |
components/osmo-cli/references/advanced-patterns.md | Checkpointing, retry/exit behavior, or node exclusion is needed. |
components/osmo-cli/tests/orchestrator-runtime-failure.md | Validating or debugging the OSMO orchestration review pattern. |
Target Selection
Pick exactly one option per stage. Stage 2 follows stage 1.
Kubernetes:
MicroK8sorAzureOSMO:
MicroK8s OSMOwhen Kubernetes is MicroK8s,Azure OSMOwhen Kubernetes is AzureInference:
NIM Operator,NVCF,Azure AI Foundry, orNoneWorkload: Video Data Augmentation, Defect Image Generation, NuRec Carline Adaptation, NRE, NCore, Asset Harvester, or custom workflow YAML
Reject invalid combinations before provisioning:
| Cluster | NIM Operator | NVCF | Azure AI Foundry |
|---|---|---|---|
| MicroK8s | yes | yes | no, Foundry requires Azure identities |
| Azure | yes | yes | yes |
For OpenClaw or any chat-only environment that cannot open a browser, read
components/openclaw-azure-login/reference.md before Azure prerequisites.
For any Azure target, read components/azure-access/reference.md before Azure
component preflights.
Setup Flow
Confirm target choices and workload compute requirements.
Load the selected component references.
Resolve prerequisites up front, including API keys, Azure access, caller CIDR, GPU quota, storage class, and OSMO login requirements.
Run
scripts/preflight.shfor every selected infrastructure component plus any OSMO CLI/workload preflight before provisioning; build the implementation plan from the results and stop on red preflight.Deploy Kubernetes first. Nothing else starts until the cluster gate is green.
Deploy OSMO and inference after Kubernetes. These can proceed in parallel once the cluster exists, but workload submission waits for both selected gates.
Submit the workload only after OSMO, storage credentials, compute pool, and selected inference endpoints are verified. For VDA, this includes
preflight_credentials.sh,pre_submit_guard.pywith resolved--setvalues, non-empty model-cache prefixes, and workflow-namespace endpoint smoke checks.Monitor through completion. On failed workflow state, inspect events and logs from
components/osmo-cli/reference.md; do not resubmit blindly.
Inference Discovery
Avoid over-deploying expensive endpoints.
Scan the chosen workflow spec and default values for endpoint references:
*.osmo-nims.svc.cluster.local,api.nvcf.nvidia.com/*,*.inference.ai.azure.com, or*.cognitiveservices.azure.com.Map each reference to the selected backend:
NIM Operator: service name must match a directory under
components/inference-nim-operator/nims/.NVCF: function URL or function ID must be supplied by the environment.
Azure AI Foundry: endpoint name must be deployed through
components/inference-azure/scripts/install.sh.
If the workflow needs a capability the selected backend lacks, stop and report the mismatch. Do not silently substitute another model.
Verification Gates
Each stage has its own Verify section in the component reference. These gates are mandatory:
| Stage | Gate |
|---|---|
| Kubernetes | Cluster API reachable, nodes Ready, GPU capacity advertised for GPU paths, and CPU+NVCF paths have runtimeclass/nvidia mapped to runc. |
| Inference | Every endpoint referenced by the workload is reachable. NIM readiness uses /v1/health/ready; NVCF and Foundry still need task-specific authenticated checks. |
| OSMO | OSMO pods Ready, pool ONLINE, port-forward watchdogs alive, storage credentials configured, and verify-hello workflow COMPLETED. |
| Workload | Selected workload pre-submit guards pass before submit. osmo workflow query <id> reports COMPLETED and every task is green. Failed terminal states require events and logs before retry. |
Resilient Scaling
Size the cluster from workload needs before provisioning. For Azure, check CPU and GPU quota for the selected VM families before
terraform apply.For NIM Operator, deploy only the NIMServices referenced by the workload. Each service pins GPU and model-cache storage for the lifetime of the cluster.
Keep OSMO storage URL schemes aligned with the active backend. Local MicroK8s uses MinIO, Azure uses Blob-backed configuration.
Treat Pending, Unknown, ImagePullBackOff, unbound PVCs, or 0 Ready replicas as layer failures. Investigate scheduling, storage, image credentials, and adjacent platform state before retrying the same command.
For long deploys or workflow watches, provide heartbeat updates with current state, elapsed time, last useful observation, and next check.
Workload Routing
Video Data Augmentation: use
skills/physical-ai-video-data-augmentation/SKILL.md.Defect Image Generation: use
skills/physical-ai-defect-image-generation/SKILL.md.NuRec carline adaptation: use
skills/carline-adaptation/SKILL.md.NRE, NCore, and Asset Harvester live in the canonical NuRec catalog listed in
skills/INDEX.md.Custom workload: submit the provided workflow YAML through OSMO after checking resource requests, image credentials, data credentials, and inference URLs.
Evaluation Prompts And Results
Positive trigger: "Set up resilient Physical AI infrastructure for VDA on Azure AKS with NIM Operator." Expected: use this skill.
Negative trigger: "Summarize recent OSMO workflow logs for this workflow ID." Expected: do not use this infrastructure setup skill unless the request also involves setup, scaling, validation, or recovery of the infrastructure stack.
Latest static review: 2026-05-26, description keywords match the expected routes above.
Files
- ATTRIBUTION.mdAdded by AutoGineer1.0 KiB
- BENCHMARK.md6.3 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md10.9 KiB
- components/azure-access/reference.md3.0 KiB
- components/cluster-azure/.gitignore103 B
- components/cluster-azure/reference.md5.9 KiB
- components/cluster-azure/scripts/helmfile.yaml354 B
- components/cluster-azure/scripts/main.tf25.4 KiB
- components/cluster-azure/scripts/outputs.tf1.6 KiB
- components/cluster-azure/scripts/preflight.shScript7.9 KiB
- components/cluster-azure/scripts/setup.shScript4.1 KiB
- components/cluster-azure/scripts/storage-class-nfs.yaml2.3 KiB
- components/cluster-azure/scripts/system_node_capacity_test.shScript915 B
- components/cluster-azure/scripts/terraform.tfvars.example568 B
- components/cluster-azure/scripts/variables.tf2.6 KiB
- components/cluster-azure/scripts/versions.tf588 B
- components/cluster-azure/terraform/main.tf8.0 KiB
- components/cluster-azure/terraform/modules/README.md705 B
- components/cluster-azure/terraform/outputs.tf3.9 KiB
- components/cluster-azure/terraform/prerequisites/az-sub-init.shScript2.2 KiB
- components/cluster-azure/terraform/prerequisites/install-terraform.shScript1.8 KiB
- components/cluster-azure/terraform/prerequisites/register-azure-providers.shScript6.8 KiB
- components/cluster-azure/terraform/prerequisites/robotics-azure-resource-providers.txt336 B
- components/cluster-azure/terraform/terraform.tfvars.example7.6 KiB
- components/cluster-azure/terraform/variables.tf9.1 KiB
- components/cluster-azure/terraform/versions.tf779 B
- components/cluster-microk8s/reference.md1.8 KiB
- components/cluster-microk8s/runtimeclass-nvidia-runc.yaml223 B
- components/cluster-microk8s/scripts/preflight.shScript4.5 KiB
- components/driver/reference.md8.6 KiB
- components/inference-azure/reference.md2.6 KiB
- components/inference-azure/scripts/install.shScript5.9 KiB
- components/inference-azure/scripts/preflight.shScript5.5 KiB
- components/inference-nim-operator/nims/cosmos-predict/nimservice.yaml2.4 KiB
- components/inference-nim-operator/nims/cosmos-reason/nimservice.yaml1.1 KiB
- components/inference-nim-operator/nims/cosmos-transfer/nimservice.yaml1.1 KiB
- components/inference-nim-operator/nims/qwen-image-edit-nvpcb-ovsl2sl/nimservice.yaml2.5 KiB
- components/inference-nim-operator/nims/qwen-image-edit/nimservice.yaml1.2 KiB
- components/inference-nim-operator/nims/qwen25-14b/hf-download-job.yaml2.0 KiB
- components/inference-nim-operator/nims/qwen25-14b/nimservice.yaml1.9 KiB
- components/inference-nim-operator/nims/qwen25-14b/pvc.yaml892 B
- components/inference-nim-operator/nims/qwen3-235b/hf-download-job.yaml1.7 KiB
- components/inference-nim-operator/nims/qwen3-235b/nimservice.yaml1.7 KiB
- components/inference-nim-operator/nims/qwen3-235b/pvc.yaml704 B
- components/inference-nim-operator/nims/qwen3-vl/hf-download-job.yaml2.6 KiB
- components/inference-nim-operator/nims/qwen3-vl/nimservice.yaml4.7 KiB
- components/inference-nim-operator/nims/qwen3-vl/pvc.yaml964 B
- components/inference-nim-operator/reference.md5.5 KiB
- components/inference-nim-operator/scripts/install.shScript11.8 KiB
- components/inference-nim-operator/scripts/preflight.shScript3.0 KiB
- components/inference-nvcf/reference.md3.2 KiB
- components/inference-nvcf/scripts/preflight.shScript2.7 KiB
- components/openclaw-azure-login/reference.md2.8 KiB
- components/osmo-azure/reference.md4.4 KiB
- components/osmo-azure/scripts/preflight.shScript5.9 KiB
- components/osmo-cli/agents/logs-reader.md3.5 KiB
- components/osmo-cli/agents/workflow-expert.md4.0 KiB
- components/osmo-cli/reference.md42.1 KiB
- components/osmo-cli/references/advanced-patterns.md2.4 KiB
- components/osmo-cli/references/cli-commands.md14.6 KiB
- components/osmo-cli/references/workflow-patterns.md8.0 KiB
- components/osmo-cli/references/workflow-spec.md8.5 KiB
- components/osmo-cli/scripts/preflight.shScript2.4 KiB
- components/osmo-cli/tests/orchestrator-runtime-failure.md11.0 KiB
- components/osmo-k8s/reference.md2.5 KiB
- components/osmo-k8s/scripts/preflight.shScript2.8 KiB
- evals/evals.json2.9 KiB
- skill-card.md3.4 KiB
- skill.oms.sig21.3 KiB
components/osmo-cli/references/cli-commands.md
OSMO CLI Command Reference
Complete reference for all osmo subcommands and their flags.
Table of Contents
osmo login/osmo logoutosmo workflowosmo poolosmo resourceosmo datasetosmo data(Direct Storage)osmo apposmo profileosmo user(Admin)osmo token(Personal Access Tokens)osmo credentialosmo bucketosmo taskConfigs (Admin)
osmo standalone(Local Docker Execution)osmo docker-compose(Local Parallel Execution)Global Flags
osmo login / osmo logout
osmo login <url> [--method code|password|token|dev]
# Device code flow (default) — opens browser
osmo login https://osmo.example.com
# Password flow
osmo login https://osmo.example.com --method password --username user --password pass
osmo login https://osmo.example.com --method password --username user --password-file /path
# Refresh token flow
osmo login https://osmo.example.com --method token --token <refresh_token>
osmo login https://osmo.example.com --method token --token-file /path
# Dev mode (no JWT, header-only auth)
osmo login https://osmo.example.com --method dev --username devuser
osmo logout # clears stored credentials from login.yamlAdditional flags: --device-endpoint <url> (override device flow endpoint).
osmo workflow
submit
osmo workflow submit <file_or_workflow_id> [flags]| Flag | Description |
|---|---|
--pool / -p | Target pool (required unless default set) |
--set key=value [k2=v2] | Override Jinja template variables (auto-casts numbers) |
--set-string key=value | Override as string (no type casting) |
--set-env key=ENV_VAR | Set variable from environment variable |
--dry-run | Validate and expand templates without submitting |
--priority HIGH|NORMAL|LOW | Workflow priority |
--rsync local:remote | Rsync local path into task |
--format-type json|text | Output format |
If the first argument is not a file path, it's treated as a workflow ID for resubmission
(in that case --dry-run, --set are not allowed).
validate
osmo workflow validate <file.yaml> --pool <pool> [--set key=val]Server-side validation only (no submission).
restart
osmo workflow restart <workflow_id> [--pool <pool>] [--format-type json|text]list
osmo workflow list [flags]| Flag | Description |
|---|---|
--count / -c | Results per page (default 20) |
--offset / -f | Pagination offset |
--name / -n | Filter by name |
--status | Filter: PENDING, SCHEDULING, RUNNING, COMPLETED, FAILED, CANCELED, etc. |
--pool / -p | Filter by pool(s) |
--user / -u | Filter by user |
--all-users / -a | Show all users (mutually exclusive with --user) |
--order / -o | asc or desc |
--submitted-after | Date filter (YYYY-MM-DD) |
--submitted-before | Date filter (YYYY-MM-DD) |
--tags | Filter by admin tags |
--priority | Filter by priority |
--app / -P | Filter by app name or app:version |
--format-type / -t | json or text |
query
osmo workflow query <workflow_id> [--verbose] [--format-type json|text]Returns detailed status, task states, Grafana URL, K8s dashboard link.
logs
osmo workflow logs <workflow_id> [--task <name>] [--retry-id <n>] [--error] [-n <lines>]Streams logs. Use --task to filter to a specific task. -n limits to last N lines.
events
osmo workflow events <workflow_id> [--task <name>] [--retry-id <n>]Streams Kubernetes events. Useful for debugging PENDING workflows.
cancel
osmo workflow cancel <id> [<id2>...] [--message "reason"] [--force]exec
osmo workflow exec <workflow_id> <task_name> [--entry /bin/bash] [--keep-alive]
osmo workflow exec <workflow_id> --group <group> [--entry <cmd>]Opens interactive shell into a running task or group.
spec
osmo workflow spec <workflow_id> [--template]--template returns the original YAML with template variables unexpanded.
port-forward
osmo workflow port-forward <workflow_id> <task> --port <local>:<remote> [--host localhost] [--udp]rsync
osmo workflow rsync upload <workflow_id> [<task>] <local_path>:<remote_path> [--daemon]
osmo workflow rsync download <workflow_id> [<task>] <remote_path>:<local_path>
osmo workflow rsync status <workflow_id> [<task>]
osmo workflow rsync stop <workflow_id> [<task>]tag
osmo workflow tag --workflow <id> --add <tag>
osmo workflow tag --workflow <id> --remove <tag>
osmo workflow tag # list all admin tagsosmo pool
osmo pool list [--pool <name>...] [--mode used|free] [--format-type json|text]Shows GPU quota and capacity per pool. --mode free shows available instead of used.
osmo resource
osmo resource list [--pool <name>...] [--platform <name>...] [--all] [--mode used|free]
osmo resource info <node_name> [--pool <name> --platform <name>]info requires both --pool and --platform together, or neither.
osmo dataset
Core operations
osmo dataset upload <bucket/name:tag> <local_path> [--desc "..."] [--metadata m.yaml] [--labels l.yaml]
osmo dataset download <name:tag> <local_path> [--regex '.*\.png$'] [--resume]
osmo dataset delete <name:tag> [--force]
osmo dataset info <name> [--all] [--count N] [--order asc|desc]
osmo dataset list [--bucket <b>] [--name <n>] [--count N]
osmo dataset inspect <name:tag> [--format-type text|tree|json] [--regex ...] [--count N]Advanced
osmo dataset update <name:tag> --add <local_path>:<remote_path> [--remove 'regex']
osmo dataset collect <collection_name> <ds1> <ds2:tag> ...
osmo dataset rename <old> <new> [--force]
osmo dataset query <query.yaml> [--bucket <b>]
osmo dataset migrate <name:tag> --target-bucket <b>
osmo dataset tag <name:tag> --set <new_tag> # or --delete
osmo dataset label <name:tag> --set key=val # or --delete key
osmo dataset metadata <name:tag> --set key=val
osmo dataset checksum <local_path> # local MD5 aggregate
osmo dataset check <name> [--access-type ...]Dataset names follow the pattern [bucket/]name[:tag].
osmo data (Direct Storage)
osmo data upload <remote_uri> <local_path> [<more_paths>...] [--regex '...'] [-p N] [-T N]
osmo data download <remote_uri> <local_path> [--regex '...'] [--resume] [-p N] [-T N]
osmo data list <remote_uri> [--prefix <p>] [--recursive] [--regex '...'] [--no-pager]
osmo data delete <remote_uri> [--regex '...']
osmo data check <remote_uri> [--access-type <type>] [--config-file <path>]-p sets parallel processes, -T sets threads per process. Uses the multi-cloud
storage SDK (S3, Azure, GCS, Swift, TOS).
For workflow outputs, use osmo data, including local MicroK8s MinIO. Do not
read /var/snap/microk8s/common/default-storage/minio-operator-data*/ directly:
MinIO chunks objects and encrypts them at rest, so the files are not usable
outside MinIO. Use --no-pager whenever running non-interactively.
# Discover workflow folders
osmo data list --no-pager s3://osmo-workflows
# Inspect one workflow or output prefix
osmo data list --no-pager s3://osmo-workflows/<workflow_id>/
osmo data list --no-pager --recursive s3://osmo-workflows/<workflow_id>/
# Download so the files are available locally
osmo data download s3://osmo-workflows/<workflow_id>/ /tmp/<workflow_id>-dataLocal MinIO client access is also valid when it goes through MinIO. The local
mc client may already have an osmo alias configured:
mc ls osmo/osmo-workflows/
mc ls --recursive osmo/osmo-workflows/<workflow_id>/
mc cp --recursive osmo/osmo-workflows/<workflow_id>/ /tmp/<workflow_id>-data/Do not confuse this with direct disk access under /var/snap/...; that path is
chunked/encrypted and not usable as workflow output files.
osmo app
osmo app create <name> --description "desc" [--file <yaml>] # editor if no --file
osmo app update <name[:version]> [--file <yaml>]
osmo app info <name[:version]> [--count N] [--order asc|desc]
osmo app show <name[:version]>
osmo app spec <name[:version]>
osmo app list [--name <n>] [--user <u>] [--all-users] [--count N]
osmo app delete <name[:version]> [--all] [--force]
osmo app rename <old> <new> [--force]
osmo app submit <name[:version]> --pool <pool> [--set key=val] [--dry-run] [--priority ...]app submit delegates to workflow submission with app context.
osmo profile
osmo profile list # show profile, pools, settings
osmo profile set pool <pool_name> # set default pool
osmo profile set bucket <bucket_name> # set default bucket
osmo profile set notifications <bool> # toggle notificationsosmo user (Admin)
osmo user list [--format-type json|text]
osmo user create <username>
osmo user update <username> [role flags]
osmo user delete <username>
osmo user get <username>osmo token (Personal Access Tokens)
osmo token set <name> [--expires-at <date>] [--description "..."] [--roles ...]
osmo token delete <name>
osmo token list
osmo token roles <name>Admin variants accept --user <username> to manage other users' tokens.
osmo credential
osmo credential set <name> --type REGISTRY|DATA|GENERIC --payload key=value [k2=v2 ...]
osmo credential list
osmo credential delete <name>The CLI takes --payload key=value pairs, NOT --server/--username/--password
flags. Those flags do not exist; using them will fail. See the full narrative
(NGC key resolution, workflow wiring, common gotchas) in SKILL.md →
Set Up an Image Registry Credential.
--type REGISTRY
Used for pulling private container images in OSMO workflows. Keys:
| Key | Meaning |
|---|---|
registry | Registry hostname (e.g. nvcr.io), no scheme, no path |
username | Registry username (for NGC: literal $oauthtoken) |
auth | Raw auth token (for NGC: the raw NGC API key). NOT the base64 user:pass string Docker writes to ~/.docker/config.json |
# nvcr.io (NGC) — most common case
osmo credential set nvcr --type REGISTRY \
--payload registry=nvcr.io username='$oauthtoken' auth="$NGC_API_KEY"Reference it from a workflow task via the task-level credentials: map
(key = this credential name). OSMO auto-wires a REGISTRY credential referenced
this way as an imagePullSecret on the task pod. Example:
tasks:
- name: train
image: nvcr.io/nvidia/pytorch:24.01-py3
credentials:
nvcr: # same name used with `osmo credential set`
NGC_CLI_API_KEY: auth # optional: also expose `auth` as an env varName must match exactly — nvcr_io ≠ nvcr-io. Mismatches show up as
ImagePullBackOff on the task pod. See references/workflow-spec.md for the
full credentials: schema.
--type GENERIC
Arbitrary key-value secrets surfaced into tasks (env vars or credential mount). Commonly used for HuggingFace tokens, API keys, etc. Keys are free-form — the task spec decides how to consume them.
osmo credential set hf-token --type GENERIC --payload token=hf_YOUR_TOKEN
osmo credential set ngc-api-key --type GENERIC --payload key="$NGC_API_KEY"--type DATA
Object-storage credentials for osmo dataset / osmo data and workflow
inputs: / outputs: that point at S3/GCS/Azure/Swift/TOS URIs. Also updates
the local config.yaml so the storage SDK picks it up for subsequent
client-side calls.
osmo credential set my-s3 --type DATA --payload \
access_key=AKIA... secret_key=... endpoint=https://s3.amazonaws.com region=us-east-1Exact keys vary by provider; see osmo credential set --help and
references/workflow-spec.md for the provider-specific fields. For non-AWS S3
(MinIO, TOS, etc.), both region and override_url/endpoint are typically
required.
osmo bucket
osmo bucket listosmo task
osmo task list [--workflow <id>] [--status ...] [--pool ...] [--count N]Configs (Admin)
In 6.3 ConfigMap mode (services.configFile.enabled: true), all configs live in the osmo-service-configs ConfigMap. The osmo config CLI subcommands no-op or 409 here.
# Read
kubectl get cm osmo-service-configs -n osmo-minimal -o yaml
# Apply a change (scripted, idempotent)
kubectl patch cm osmo-service-configs -n osmo-minimal --type=merge -p ...The osmo-service container watches /etc/osmo/configs/config.yaml via inotify and reloads on change.
osmo standalone (Local Docker Execution)
osmo standalone run -f <workflow.yaml> [flags]| Flag | Description |
|---|---|
-f / --file | Workflow YAML (required) |
--work-dir | Working directory for intermediate data |
--keep | Preserve containers after completion |
--docker | Docker binary path (default: docker) |
--resume | Resume from last successful step |
--from-step <task> | Resume from a specific task |
--credential NAME=PATH | Mount credential directory (repeatable) |
--set key=value | Template variable overrides |
--set-string key=value | String-only overrides |
--shm-size <size> | Shared memory size (e.g. 16g) |
Runs tasks serially via docker run. Does NOT support {{host:taskname}} —
use docker-compose for inter-task networking.
osmo docker-compose (Local Parallel Execution)
osmo docker-compose run -f <workflow.yaml> [flags]Same flags as standalone except --compose-cmd replaces --docker:
| Flag | Description |
|---|---|
--compose-cmd | Compose command (default: docker compose) |
Supports {{host:taskname}} via shared Docker networks. Tasks in the same
group run in parallel. Groups execute in topological order (serial between groups,
parallel within).
Global Flags
All commands support:
| Flag | Description |
|---|---|
--log-level | Logging level (default: INFO) |
Most list/query commands support:
| Flag | Description |
|---|---|
--format-type / -t | json or text (default: text) |
Use --format-type json for machine-readable output (recommended for scripting
and when parsing output programmatically).
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an API key, Needs Node.js
The author says: Requires the selected component prerequisites, usually kubectl plus either MicroK8s or Azure CLI/Terraform, and OSMO or inference credentials for the chosen target.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
- Instructions that use the network
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Physical AI
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/physical-ai-infrastructure-setup-and-resilient-scaling · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- openai/plugins
AutoGineer is not affiliated with or endorsed by NVIDIA Physical AI or NVIDIA.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download physical-ai-infrastructure-setup-and-resilient-scaling.zip