Tao Finetune Video Clip

eval.config

{
  "evals": [
    {
      "id": "iv2clip-full-pipeline-local-docker",
      "prompt": "Run the full InternVideo2-CLIP (**video_clip**) pipeline (train → evaluate → inference) on a mini vadr1_chunks dataset, using **local Docker** on the host machine for GPU execution.\n\n## Plugin Installation\n\nThis eval depends on the `tao-skills` plugin (bundled in the `tao-skill-bank` repo). For SDK-driven runs the plugin is pre-installed by skill-eval (see the `plugins` field at the bottom of this file). For interactive TUI runs install it manually:\n\n```bash\nPLUGIN_ROOT=\"${CI_PROJECT_DIR:-$HOME/tao-skill-bank}\"\nrm -rf ~/.claude/plugins/cache/tao-skill-bank/tao-skills\n```\n\n```\n/plugin marketplace add $PLUGIN_ROOT\n/plugin install tao-skills@tao-skill-bank\n/plugin marketplace update tao-skill-bank\n```\n\n## Skill under test\n\n`tao-finetune-video-clip` (InternVideo2-CLIP L14 / TAO `video_clip` task).\n\n## Execution platform\n\n**Local Docker** — host's Docker daemon, NVIDIA runtime.\n\nThe TAO SDK is optional for this platform; plain `docker run` is sufficient.\n\n## Data — pull from the NVIDIA internal S3 (pdx.s8k.io) using s5cmd\n\nThe AWS profile `seanlin` and endpoint `https://pdx.s8k.io` are pre-configured from the env file (see skill-eval's entrypoint, which writes ~/.aws/credentials and ~/.aws/config from S3_* env vars). Use `s5cmd` to mirror the smoke pack to a local staging directory before any docker run.\n\nStaging layout:\n\n```\n$WORKSPACE_DIR/iv2clip/           # WORKSPACE_DIR is bind-mounted at the same absolute path on host + container,\n                                  # so nested `docker run -v $WORKSPACE_DIR/...` resolves correctly through the host daemon\n├── data/\n│   ├── train.json                # video_path entries point at /data/videos/<name>.mp4\n│   ├── val.json\n│   ├── prompts.txt\n│   └── videos/                   # 32 clips (24 .mp4 + 8 .avi)\n├── model/\n│   ├── mobileclip_blt.pt\n│   └── hf/                       # offline InternVideo2_distillation_models snapshot\n├── specs/                        # generated from skill templates + smoke overrides\n└── results/\n```\n\n**Run all `s5cmd` calls in the foreground with `timeout` + `--retry-count`** so a stalled connection can't keep the eval session alive indefinitely. If a download fails, stop and report — do not retry in a background task.\n\nCommands:\n\n```bash\nmkdir -p $WORKSPACE_DIR/iv2clip\ncd $WORKSPACE_DIR/iv2clip\ntimeout 900 s5cmd --retry-count 3 --profile seanlin --endpoint-url https://pdx.s8k.io cp 's3://computex/skill-eval-ci/iv2clip/*' ./\nfind $WORKSPACE_DIR/iv2clip -maxdepth 3\n```\n\nThe pull is complete only when **all** of these exist. A partial or empty pull is a FAIL — do not proceed to the docker runs:\n\n```bash\ncd $WORKSPACE_DIR/iv2clip\nfor f in data/train.json data/val.json data/prompts.txt model/mobileclip_blt.pt; do\n  [ -s \"$f\" ] || echo \"MISSING $f\"\ndone\n[ -d model/hf ] || echo \"MISSING model/hf (offline InternVideo2 snapshot)\"\necho \"clip count: $(ls data/videos/ 2>/dev/null | wc -l)  (expected 32: 24 .mp4 + 8 .avi)\"\n```\n\nResolve these paths after download:\n\n- `DATA_DIR` — `$WORKSPACE_DIR/iv2clip/data`\n- `MODEL_DIR` — `$WORKSPACE_DIR/iv2clip/model`\n- `RESULTS_DIR` — `$WORKSPACE_DIR/iv2clip/results`\n- `SPECS_DIR` — `$WORKSPACE_DIR/iv2clip/specs`\n\nIf `s5cmd cp` fails (credentials / network / empty prefix), **stop and report** — do not synthesize stub data.\n\n## Container image\n\nUse the pinned image from `references/skill_info.yaml`:\n\n```bash\nVIDEO_CLIP_IMAGE=\"<container_image from references/skill_info.yaml>\"\n```\n\nAuthenticate with `NGC_KEY` if the image is not cached locally:\n\n```bash\necho \"$NGC_KEY\" | docker login nvcr.io -u '$oauthtoken' --password-stdin\n```\n\nPreflight — run all three, and record each exit code in `{artifacts_dir}/preflight.json`:\n\n```bash\ndocker run --rm \"$VIDEO_CLIP_IMAGE\" video_clip --help\ndocker run --rm \"$VIDEO_CLIP_IMAGE\" \\\n  python -c \"import nvidia_tao_pytorch.multimodal.video_clip.model.adapters.internvideo2clip\"\ndocker run --rm \"$VIDEO_CLIP_IMAGE\" \\\n  python -c \"import av; print(av.__version__)\"\n```\n\nThe second command is a **data-free model import smoke check** and the third verifies the TAO 7.2 PyAV decode dependency. `video_clip --help` exits 0 even on an image whose `video_clip` package is incomplete (e.g. missing `model.backbones`), so only the imports tell you whether the actions can run at all. If either import fails, report the image as broken and stop — do not install decord or build a derived image. The failure is in the container, not in the skill or the specs, and is independent of whether the S3 data arrived.\n\n## Spec generation\n\nCopy the skill's `references/spec_template_*.yaml` into `$SPECS_DIR` and apply smoke overrides:\n\n- `results_dir: /results`\n- `model.text_encoder: /model/mobileclip_blt.pt`\n- `model.vision_encoder: /model/hf/stage1/L14/L14_dist_1B_stage2/pytorch_model.bin`\n- `model.clip_head: /model/hf/clip/L14/pytorch_model.bin`\n- `dataset.train.video_text.metadata: /data/train.json`\n- `dataset.val.video_text.metadata: /data/val.json`\n- `train.num_epochs: 2`, `train.num_gpus: 1`, `train.gpu_ids: [0]`\n- `dataset.train.batch_size: 2`, `dataset.val.batch_size: 2`\n- `dataset.metrics.mode: classification`\n- `inference.query.text_file: /data/prompts.txt` (inference spec)\n\nThe `model.vision_encoder` / `model.clip_head` overrides are required, not optional. The smoke pack stages `model/hf/` as a plain repo-style snapshot, not a HuggingFace hub cache, so `HUGGINGFACE_HUB_CACHE=/model/hf` alone does NOT resolve it. Leaving them at the template default of `null` sends the resolver into `hf_hub_download`, which fails under `HF_HUB_OFFLINE=1`. Naming the files directly takes the local-path branch instead.\n\n## Docker mount pattern\n\n```bash\nRUN_ROOT=\"$WORKSPACE_DIR/iv2clip\"\nDOCKER_COMMON=(\n  --rm --gpus all --shm-size=8g --network=host\n  --shm-size=64g\n  --ulimit memlock=-1\n  --ulimit stack=67108864\n  -e WANDB_DISABLED=true\n  -e WANDB_MODE=disabled\n  -e HF_HUB_OFFLINE=1\n  -e HUGGINGFACE_HUB_CACHE=/model/hf\n  -e TRANSFORMERS_OFFLINE=1\n  -v \"$RUN_ROOT/data:/data:ro\"\n  -v \"$RUN_ROOT/model:/model:ro\"\n  -v \"$RUN_ROOT/specs:/specs:ro\"\n  -v \"$RUN_ROOT/results:/results\"\n)\n```\n\n## Pipeline (single run, three actions)\n\n1. **Train** — `docker run ... video_clip train -e /specs/train.yaml results_dir=/results`; verify `Execution status: PASS` in logs.\n2. **Evaluate** — checkpoint at `/results/train/video_clip_latest.pth`.\n3. **Inference** — same checkpoint; verify embeddings under `/results/inference/`.\n\nRun docker commands in the **foreground** with a generous timeout (~1 hour). After each docker run, retrieve logs via `docker logs <name>` before removing the container.\n\n## Artifacts to upload (save under {artifacts_dir})\n\n- `{artifacts_dir}/preflight.json` — `video_clip --help` and import-smoke exit codes\n- `{artifacts_dir}/train_log.txt`\n- `{artifacts_dir}/evaluate_log.txt`\n- `{artifacts_dir}/inference_log.txt`\n- `{artifacts_dir}/docker_runs.json` — per-action: image used, container name, exit code, start/end timestamps\n- `{artifacts_dir}/metrics.json` — parsed classification metrics from evaluate logs\n- `{artifacts_dir}/s5cmd_download.log` — stdout/stderr of the s5cmd cp\n\n## Cleanup (run at the very end, before reporting success)\n\nAfter all three actions complete AND artifacts are saved, free the disk:\n\n```bash\nrm -rf $WORKSPACE_DIR/iv2clip\ndocker container prune -f >/dev/null 2>&1 || true\n```\n\nLog the cleanup result (success / failure) to `{artifacts_dir}/cleanup.log` but do NOT fail the eval if cleanup fails.\n\n## Expected outcome\n\nAll three actions (train, evaluate, inference) complete with `Execution status: PASS`. Logs + parsed metrics saved under `{artifacts_dir}`. `s5cmd_download.log` shows a successful pull from S3. Local staging dir removed.",
      "expected_outcome": "Preflight: `video_clip --help`, the data-free model import smoke (`python -c \"import nvidia_tao_pytorch.multimodal.video_clip.model.adapters.internvideo2clip\"`), and the PyAV import smoke (`python -c \"import av; print(av.__version__)\"`) all exit 0; all three exit codes are recorded in `preflight.json`. A non-zero import smoke means the pinned container is broken and must be reported rather than worked around; do not install decord or build a derived image. `s5cmd_download.log` shows a successful pull from `s3://computex/skill-eval-ci/iv2clip/` that produced every required object: `data/train.json`, `data/val.json`, `data/prompts.txt`, 32 clips under `data/videos/` (24 `.mp4` + 8 `.avi`), `model/mobileclip_blt.pt`, and the `model/hf/` snapshot directory — a partial pull is a failure. `video_clip --help` advertises train/evaluate/inference/export/default_specs (action names valid), and, separately, all three local-docker actions (train, evaluate, inference) reach `Execution status: PASS`. `metrics.json` contains numeric classification metrics from the evaluate step and `docker_runs.json` records per-action container metadata. Local staging dir cleaned up at the end. No Lepton-side activity; pipeline is host-local. Distinguish container/image defects from skill defects when grading: missing PyAV and a PyTorch `weights_only` checkpoint-load error are properties of the pinned image. The supported TAO 7.2 decoder contract uses PyAV with FFmpeg CLI and OpenCV fallbacks."
    }
  ],
  "credentials": [
    "ACCESS_KEY",
    "SECRET_KEY",
    "NGC_KEY"
  ],
  "plugins": {
    "claude": [
      {
        "marketplace": "${CI_PROJECT_DIR}",
        "plugin": "tao-skills@tao-skill-bank"
      }
    ],
    "codex": [
      {
        "marketplace": "${CI_PROJECT_DIR}",
        "plugin": "tao-skill-bank@tao-skill-bank"
      }
    ]
  }
}