Cdf Auto Labeling

evals/evals.json

[
  {
    "id": "1",
    "question": "I just cloned the repo and have my own warehouse-safety videos. I'm new to this - how do I get started producing auto-labels for my domain?",
    "expected_skill": "cdf-auto-labeling",
    "ground_truth": "Response uses the auto-labeling skill to give a guided end-to-end path: first confirm the critical inputs with the user (input data path, output path, VLM/LLM endpoints and model names, model cache, GPU ids) and ask for anything missing; verify the environment; run a shipped example cookbook with --container-dry-run to prove the stack before customizing; then plan the scenario (scenario-planning), adapt the closest shipped cookbook (cookbook-authoring), author prompts/question banks (prompt-authoring), configure stages starting with detection-and-tracking, and dry-run then execute and validate (workflow-runner-debugging). It routes to the matching references instead of duplicating their detail and does not claim it executed anything.",
    "expected_behavior": [
      "Response triggers the auto-labeling skill for a new user/new domain onboarding request.",
      "Response confirms critical inputs and asks the user for any that are missing or ambiguous before running.",
      "Response tells the user to run a shipped example first (dry-run) before customizing.",
      "Response sequences plan -> adapt cookbook -> author prompts -> configure stages -> dry-run/execute/validate and routes to the specialized skills.",
      "Response provides a container dry-run command and does not claim execution."
    ]
  },
  {
    "id": "2",
    "question": "Kick off the auto-labeling pipeline for my new use case and just run it.",
    "expected_skill": "cdf-auto-labeling",
    "ground_truth": "Response uses the auto-labeling skill and refuses to guess. Because the input data path, output path, and VLM/LLM endpoints/model names are not provided, it asks the user to supply and confirm those critical inputs (plus model cache, GPU ids, and reasoning max_tokens if a reasoning model is used) before executing. It restates the confirmed values, requires a dry-run before any real run, and does not invent paths, endpoints, or credentials.",
    "expected_behavior": [
      "Response triggers the auto-labeling skill.",
      "Response asks the user for the missing critical inputs (data path, output path, endpoints/model names) rather than guessing or inventing defaults.",
      "Response states it will restate/confirm inputs and dry-run before any real run.",
      "Response does not fabricate paths, endpoint URLs, model IDs, or credentials."
    ]
  },
  {
    "id": "3",
    "question": "What's the fastest way to confirm the stack works end to end before I customize anything for my own domain?",
    "expected_skill": "cdf-auto-labeling",
    "ground_truth": "Response uses the auto-labeling skill to recommend running a shipped example pipeline (the closest operator cookbook - video data augmentation, event-and-person-attribute-search, or event-verification-reasoning) via workflow-runner with --container-dry-run first, then a real run, before adapting anything for the custom domain. It confirms environment prerequisites (model cache, reachable endpoints, GPU) and notes that customizing before a clean example run makes domain debugging ambiguous.",
    "expected_behavior": [
      "Response triggers the auto-labeling skill.",
      "Response recommends running a shipped example cookbook first (dry-run then real run) to validate the stack.",
      "Response confirms environment prerequisites before running.",
      "Response advises against customizing a cookbook before the shipped example runs clean."
    ]
  },
  {
    "id": "4",
    "question": "I don't want to build all the stage containers myself. Where do I get the Auto-Labeling images, and do I need to log in to nvcr.io?",
    "expected_skill": "cdf-auto-labeling",
    "ground_truth": "Response uses the auto-labeling skill and routes to the ngc-containers reference. It explains that the stage images are public in the NGC Catalog under nvcr.io/nvidia/cdf-* with the 1.1.0 release tag, that anonymous pulls work so docker login nvcr.io is optional, and that an authenticated login uses the literal username $oauthtoken with an NGC personal API key passed through --password-stdin from an environment variable. It recommends dry-running the cookbook to find the images the run needs, then running make pull with NGC_IMAGES limited to those images so each image is tagged with the local cdf-* name the workflow runner expects, and verifying with docker image ls and another dry run.",
    "expected_behavior": [
      "Response triggers the auto-labeling skill and uses the ngc-containers reference.",
      "Response points to nvcr.io/nvidia/cdf-* images and the NGC Catalog pages.",
      "Response states that login is optional because the images allow anonymous pulls.",
      "Response shows docker login nvcr.io with username $oauthtoken and --password-stdin, reading the key from an environment variable.",
      "Response recommends make pull (optionally with NGC_IMAGES) so images get the local names the workflow runner uses.",
      "Response never asks the user to paste an API key into chat or commit it to a file."
    ]
  }
]