Generated banner for Tao Generate Video Reasoning Annotations
Generated logo for Tao Generate Video Reasoning Annotations

Tao Generate Video Reasoning Annotations

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Data and AI
  • Apache-2.0
  • Medium risk
  • Has scripts

From the author

Multi-step video annotation pipeline that turns raw videos into Chain-of-Thought training data — multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning traces, via VLM/LLM distillation. Use when the user wants to "create video training data", "generate video QA datasets", "build CoT reasoning traces from videos", "auto-label videos", or run the video_reasoning_annotation pipeline. Triggers include "video annotation", "video CoT", "video QA", "chain-of-thought", "video captioning pipeline", "video distillation".
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-generate-video-reasoning-annotations skill provides a structured pipeline to transform raw video files into annotated datasets suitable for training video understanding models. It utilizes Vision-Language Models (VLMs) for captioning and Large Language Models (LLMs) for synthesizing descriptions and generating various QA pairs, including MCQ, binary, and open-ended questions with reasoning traces. The pipeline is designed to be domain-agnostic, supporting custom prompt modules for specialized use cases like traffic or warehouse monitoring, and includes features for anomaly detection and resume-capable processing.

What it does

  • Filters and classifies videos as normal or anomalous.
  • Generates global, dense, and chunked video captions using VLM distillation.
  • Synthesizes captions into structured narratives using LLMs.
  • Produces diverse QA pairs with step-by-step reasoning traces.
  • Parses outputs into standardized JSON files following the tao-vl-reason-v1.0 format.
  • Supports resume-on-failure to skip already-processed videos.

When to use it

  • When you need to create training data for video understanding models.
  • When you want to generate Chain-of-Thought reasoning traces from video content.
  • When you need to auto-label large collections of video files.
  • When you are working with specific domains like traffic or industrial warehouse monitoring.

Advantages

  • Modular pipeline allows for selecting specific processing steps.
  • Built-in support for domain adaptation through custom prompt modules.
  • Resume capability ensures efficiency during long-running annotation tasks.
  • Supports both Gemini and OpenAI-compatible VLM/LLM backends.
  • Provides structured output compatible with downstream training workflows.

Usage examples

“Run the video annotation pipeline on my traffic footage.”
The agent will guide you through selecting the traffic prompt module, configuring your VLM/LLM endpoints, and executing the pipeline on your video directory.
“How do I create a custom dataset for my warehouse security videos?”
The agent will initiate the domain adaptation workshop to help you define your annotation goals and customize the prompt templates for warehouse-specific events.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/tao-generate-video-reasoning-annotations/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-generate-video-reasoning-annotations.zip -d .agents/skills

Documentation by NVIDIA Corporation

Video Reasoning Annotation Pipeline

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Generate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain.

Purpose

Transform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as "teacher" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls).

Pipeline architecture

Step 0:  [Optional] Filter & classify videos  → Keep domain-relevant, classify anomaly vs normal
Step 1a: Global + dense captions               → VLM: narrative summary + timestamped events
Step 1b: Chunk captions                         → VLM: fixed-duration segment micro-captions
Step 1c: [Optional, anomaly only] Highlight     → LLM extracts anomaly timestamp, VLM captions clip
Step 2:  Description synthesis                  → LLM: synthesize captions into structured narrative
Step 3:  QA generation                          → LLM: MCQ, binary, open-ended with reasoning
Step 4:  Parse outputs                          → Per-task `tao-vl-reason-v1.0` JSON files

Steps are individually selectable via workflow.steps. The pipeline has built-in resume — each step skips already-processed videos, so re-running after a prompt tweak is safe.

Initial consultation

When the user invokes this skill, walk through these questions in order. Don't skip — getting domain and VLM access right up front prevents wasted runs.

1. Videos
  • Path to the video directory and/or a JSONL with {"video_path": "..."} per line.

  • Confirm format (.mp4 preferred; .avi, .mov, .mkv also walked).

2. Domain — drives prompt selection

Ask the user: "What domain are these videos from?" Choose one of the following branches:

DomainWhat to do
generalUse the default prompts. Set prompts_module: "" (or omit). The built-in nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts covers domain-agnostic content.
traffic (CCTV intersections, highways; dashcam excluded)Use the reference module. Set prompts_module: "nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_traffic", or copy references/prompts_traffic.py into the user's project and tune for their specific camera angles, then point prompts_module at the copy.
warehouse (industrial site CCTV — safety, operations, security)Same pattern. Set prompts_module: "nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_warehouse", or copy references/prompts_warehouse.py and tune.
custom (any other domain)Run the workshop in references/domain_adaptation.md. It walks through: Phase 1 — question types the user wants the model to answer; Phase 2 — caption-requirements checklist; Phase 3 — fill the [PLACEHOLDER] markers in nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template. The two reference modules above are working examples to model after. Do this before any pipeline runs.
3. Anomaly / normal / mixed
  • Mixed dataset → workflow.mode: "auto" (Step 0 classifies each video).

  • Pre-split anomaly only → workflow.mode: "anomaly", drop Step 0.

  • Pre-split normal only → workflow.mode: "normal", drop Steps 0 and 1c.

4. VLM / LLM endpoint — confirm access before running
  • Gemini (default for both vlm.backend and llm.backend): user needs GOOGLE_API_KEY set, or to put the key in the YAML.

  • OpenAI-compatible (Qwen via vLLM, NIM endpoint, etc.): user provides base_url, model_name, and api_key.

  • Steps 2–3 are text-only — a smaller/cheaper LLM is fine for llm.backend even when vlm.backend is a frontier video model.

If the user has no endpoint at all and wants to self-host, point them at the skills/applications/tao-run-inference-service skill — a workflow that stands up a network-specific TAO inference microservice locally and exposes an OpenAI-compatible endpoint. Should support Cosmos, Qwen, and Gemma. Check skills/applications/tao-run-inference-service/references/service.yaml for the current valid_network_arch_config_basenames list before relying on a specific model.

If the user doesn't have endpoint access ready and isn't ready to set one up, stop here and help them figure it out first.

5. Pilot vs full run
  • Recommend a 5–10 video pilot when domain is custom, when any prompt was edited, or when this is the user's first run.

  • Full-run is fine for general / traffic / warehouse once the user has previously verified output quality on the same data type.

  • The pipeline has built-in resume, so a pilot followed by a full run does not re-process the pilot videos.

Quick start

The pipeline runs inside the TAO Toolkit container via the auto_label CLI:

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    video_reasoning_annotation.data.video_root=/videos \
    video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY \
    video_reasoning_annotation.workflow.mode=auto

Generate a default spec to start from:

auto_label default_specs results_dir=/results module_name=auto_label
# then set:  autolabel_type: "video_reasoning_annotation"

All fields support Hydra dot-notation overrides on the command line. For the full YAML reference (every field, model/endpoint setup, error patterns), see references/configuration.md.

Pilot workflow

Use this when running a 5–10 video pilot:

  1. Run the pipeline on the pilot subset with the chosen prompts_module and workflow.mode.

  2. Inspect results_dir/step_1a_caption/captions.jsonl — captions accurate, capturing the right level of detail?

  3. Inspect results_dir/step_3_qa/qa_output.jsonl — questions meaningful, answers correct, reasoning logical?

  4. If quality is insufficient: adjust the prompts (in prompts_module if domain-customized, or fall back to general if a domain module is over-tuned), and re-run. The pipeline auto-skips already-processed videos.

  5. Once satisfied, scale to the full dataset by pointing data.video_root (or data.input_jsonl_files) at the full set and re-running with the same results_dir (resume) or a fresh one (full re-run).

Quality compounds downstream — bad captions produce bad descriptions which produce bad QA. Focus iteration on Step 1a/1b output first; descriptions and QA usually improve once captions are right.

Configuration summary

Key fields (full reference in references/configuration.md):

FieldDefaultDescription
workflow.steps["0","1a","1b","1c","2","3","4"]Which pipeline steps to execute
workflow.mode"auto""auto", "anomaly", or "normal"
vlm.backend"gemini""gemini" or "openai" (OpenAI-compatible)
llm.backend"gemini"Same options; text-only, cheaper model works
workflow.max_workers4Parallel threads per step (watch API rate limits)
license""Optional: written to metadata.license in step 4 outputs (e.g. "CC-BY-4.0")
description_extra""Optional: extra text appended to per-task descriptions in step 4 metadata
prompts_module""Dotted import path to custom prompts module

Prompts

  • Built-in (general): nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts — domain-agnostic, used by default.

  • Template: nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template — same 26 keys with [PLACEHOLDER] markers for domain customization.

  • Reference modules (working examples for the consultation's traffic / warehouse branches): references/prompts_traffic.py, references/prompts_warehouse.py.

  • Custom domains: see references/domain_adaptation.md for the full workshop and placeholder reference.

Inputs

  • video_root: Directory of videos (walked recursively for .mp4, .avi, .mov, .mkv).

  • input_jsonl_files: List of JSONL files with {"video_path": "..."} per line. The video key is also accepted; extra fields are allowed.

  • filter_field: Optional boolean field to filter JSONL entries.

Provide video_root, input_jsonl_files, or both (lists merge).

Outputs

All outputs go to results_dir/ with per-step subdirectories (step_0_filter/, step_1a_caption/, …, step_4_output/):

  • Steps 0–3: JSONL — one JSON object per video per line.

  • Step 4: One <task>.json per non-empty task type, in the tao-vl-reason-v1.0 envelope. Up to 10 files: mcq.json, mcq_openended.json, bcq.json, bcq_openended.json, open_qa.json, causal_linkage.json, temporal_localization.json, temporal_description.json, scene_description.json, video_summarization.json.

Each step 4 file looks like:

{
  "format": "tao-vl-reason-v1.0",
  "metadata": {"type": "annotation", "task": "<task>", "date": "YYYY-MM-DD",
               "description": "<per-task + description_extra>", "license": "<from config>"},
  "media_root": "<data.video_root>" | null,
  "items": [{"video_id": "...", "question": "...", "answer": "...", "reasoning": "..."}, ...]
}

media_root mirrors data.video_root (or null when unset); each item's video_id is the entry's video path with the video_root prefix stripped. Set license and description_extra in the spec to populate the metadata.

Prerequisites

  • Container: nvcr.io/nvidia/tao/tao-toolkit:7.3.0-pyt. <!-- versions-key: images.tao_toolkit.pyt -->

  • ffmpeg / ffprobe: required for chunk captioning (Step 1b) and highlight extraction (Step 1c).

  • VLM endpoint: at least one — Gemini API key or OpenAI-compatible endpoint.

Files

references/prompts_warehouse.py

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Warehouse CCTV domain prompts for the video CoT annotation pipeline.

Reference prompts for warehouse, distribution center, and industrial site
surveillance footage. Covers anomaly subcategories:
  - Safety-Liability: forklift strikes, falls, unsecured loads, spills
  - Operational Oversight: unattended equipment, blocked exits, PPE violations
  - Criminal-Suspicious: unauthorized access, theft, tampering
  - Security Incidents: perimeter breach, tailgating, after-hours activity

USAGE:
    Set ``prompts_module`` in your video_reasoning_annotation YAML config to the dotted import
    path of this file after copying it into your project:
        video_reasoning_annotation:
          prompts_module: "my_package.prompts_warehouse"

    Or copy and modify — this file is a starting point, not a final product.
    Tune the prompts based on your specific facility layout, camera positions,
    and annotation goals.
"""

PROMPT_TEMPLATES = {

    # =========================================================================
    # VIDEO FILTERING
    # =========================================================================
    "video_filtering": (
        "Determine if this video is warehouse or industrial site CCTV "
        "footage.\n\n"
        "A warehouse/industrial CCTV video should have:\n"
        "- A fixed overhead or elevated camera angle viewing an indoor or "
        "semi-outdoor industrial space\n"
        "- Visible industrial infrastructure (racking, shelving, conveyors, "
        "loading docks, pallets, crates)\n"
        "- Workers and/or industrial equipment (forklifts, pallet jacks, "
        "hand trucks, conveyor belts)\n\n"
        "Negative examples (NOT warehouse/industrial CCTV):\n"
        "- Retail store surveillance (shopping aisles, checkout counters)\n"
        "- Office or residential indoor footage\n"
        "- Outdoor traffic or street surveillance\n"
        "- Construction site footage without warehouse infrastructure\n\n"
        "Answer with ONLY \"Yes\" or \"No\"."
    ),

    # =========================================================================
    # ANOMALY CLASSIFICATION
    # =========================================================================
    "video_anomaly_classification": (
        "You are an expert warehouse safety analyst. Watch this video and "
        "classify whether it contains an anomalous event or only normal "
        "warehouse activity.\n\n"
        "An anomaly in warehouse/industrial CCTV is any event involving a "
        "safety incident, operational oversight, equipment malfunction, or "
        "suspicious/criminal activity.\n\n"
        "Examples of anomalies:\n"
        "- Forklift strikes a person, rack, or other equipment\n"
        "- Worker falls from height or trips over an obstruction\n"
        "- Unsecured load falls from a forklift, rack, or conveyor\n"
        "- Worker operates equipment without required PPE (hard hat, vest, "
        "gloves)\n"
        "- Unauthorized person enters a restricted area\n"
        "- Equipment left running unattended in an aisle\n\n"
        "Examples of normal activity:\n"
        "- Workers operating forklifts along designated aisles, moving "
        "pallets between racks\n"
        "- Loading dock operations with trucks being loaded/unloaded "
        "following standard procedure\n"
        "- Workers wearing proper PPE conducting routine inventory checks\n\n"
        "Near-misses count as anomalies even without contact or injury. "
        "PPE violations count even if no incident results. If ambiguous, "
        "default to Normal.\n\n"
        "First, briefly describe what you observe in the video (2-3 "
        "sentences). Then explain your reasoning for the classification "
        "(2-3 sentences). Finally, on the LAST line, write your "
        "classification as exactly one word: \"Anomaly\" or \"Normal\"."
    ),

    # =========================================================================
    # ANOMALY CAPTIONING
    # =========================================================================
    "anomaly_global_caption": (
        "You are an expert warehouse safety analyst. Watch the video "
        "carefully.\n\n"
        "Describe this video in detail, addressing the following points in "
        "a cohesive narrative:\n"
        "- **Facility Layout and Zone**: Describe the area \u2014 aisle "
        "identifiers if visible, rack positions, dock numbers, zone "
        "markings, restricted area signage, floor markings (pedestrian "
        "walkways, forklift lanes, hazard zones).\n"
        "- **Personnel and Equipment**: Count of visible workers, their "
        "PPE status (hard hat, high-visibility vest, gloves, safety "
        "shoes), equipment in use (forklift, pallet jack, conveyor, hand "
        "truck), equipment state (moving/idle/loaded/unloaded).\n"
        "- **The Safety Incident**: Describe the anomalous event in "
        "detail. What exactly happened? Who was involved? What equipment "
        "or materials were involved? What was the sequence of actions "
        "leading up to, during, and immediately after the incident?\n"
        "- **Contributing Factors and Consequences**: What safety protocol "
        "was violated or what went wrong? What was the immediate outcome "
        "(injury, damage, near-miss, spill)? What secondary hazards "
        "were created?\n\n"
        "Provide a holistic summary that captures the complete story and "
        "context. Pay close attention to temporal and spatial details. "
        "Do not hallucinate or make up information."
    ),

    "anomaly_dense_caption": (
        "You are an expert warehouse safety analyst. Provide a dense, "
        "event-level video caption with precise timestamps for a video "
        "containing a warehouse safety incident.\n\n"
        "Structure your response as a list of events. For each event, "
        "provide the start and end timestamp in the format "
        "<HH:MM:SS><HH:MM:SS> followed by a detailed description.\n\n"
        "Focus particularly on:\n"
        "1. **Precise Timing**: Accurately capture the start and end of "
        "each distinct action or state change.\n"
        "2. **Incident Actions**: Describe in detail the actions of "
        "personnel and equipment involved in the incident \u2014 equipment "
        "movements, worker positions, load handling, the moment of "
        "failure or contact.\n"
        "3. **Personnel Identification**: For every key person, note "
        "their role if distinguishable (forklift operator, floor worker, "
        "supervisor), PPE status, position in the facility, and what "
        "they are carrying or operating.\n"
        "4. **Facility Context**: Note aisle dimensions, rack heights, "
        "clearance zones, floor conditions (wet/dry/obstructed), and "
        "signage.\n"
        "5. **Environmental Factors**: Mention lighting level, floor "
        "condition, alarm indicators, and any obstructions to visibility "
        "or movement.\n\n"
        "Format each line as:\n"
        "<Start_Timestamp><End_Timestamp> Description of the event.\n\n"
        "Example:\n"
        "<00:00:00><00:00:03> A wide warehouse aisle between tall pallet "
        "racks; a forklift loaded with a large pallet is traveling toward "
        "the camera from the far end.\n"
        "<00:00:03><00:00:06> A floor worker in a yellow vest and hard "
        "hat enters the aisle from a cross-aisle on the right, walking "
        "toward the approaching forklift.\n"
        "<00:00:06><00:00:08> The forklift's load is raised approximately "
        "2 meters, blocking the operator's forward view; the operator "
        "does not slow down or sound the horn.\n"
        "<00:00:08><00:00:12> The forklift clips the floor worker's leg "
        "as it passes. The worker stumbles and falls to the ground. The "
        "forklift stops a few meters later."
    ),

    "anomaly_chunk_caption": (
        "You are an expert warehouse safety analyst. You are viewing a "
        "strict {chunk_duration}-second window of a video that contains a "
        "safety incident somewhere in the full video.\n"
        "Your job is NOT to tell the whole story, but to capture the "
        "**micro-dynamics** and **state changes** within this specific "
        "timeframe.\n\n"
        "Note: This specific chunk may or may not contain the incident \u2014 "
        "it covers only a fixed time window. Describe only what you observe "
        "in this clip; do not assume or fabricate incident content if it "
        "is not visible.\n\n"
        "Please describe the events in this {chunk_duration}-second window, "
        "focusing on:\n\n"
        "1. **Operational Dynamics**:\n"
        "   - Equipment movement paths, speeds, and load status\n"
        "   - Worker movement patterns, tasks being performed\n"
        "   - Load handling actions (lifting, placing, stacking, "
        "transporting)\n\n"
        "2. **Facility and Safety Context**:\n"
        "   - Floor condition (clear, obstructed, wet)\n"
        "   - PPE compliance for visible workers\n"
        "   - Signage, zone markings, and lighting conditions\n\n"
        "3. **Incident or Interaction Details**:\n"
        "   - If a safety incident occurs in this chunk (collision, fall, "
        "dropped load, equipment failure, PPE violation), describe the "
        "sequence precisely.\n"
        "   - If the aftermath is visible (injured worker, scattered "
        "materials, stopped equipment), describe the post-event state.\n"
        "   - If no incident is visible, describe the normal operations.\n\n"
        "Do not speculate on what happened before or after this clip. "
        "Report only what is visually confirmed in these "
        "{chunk_duration} seconds. Be precise and objective."
    ),

    # =========================================================================
    # NORMAL CAPTIONING
    # =========================================================================
    "normal_global_caption": (
        "You are an expert warehouse operations analyst. Watch the video "
        "carefully.\n\n"
        "Describe this video in detail, addressing the following points in "
        "a cohesive narrative:\n"
        "- **Facility Layout and Zone**: Describe the area \u2014 aisle "
        "identifiers, rack positions, dock numbers, zone markings, floor "
        "markings.\n"
        "- **Personnel and Equipment**: Count of visible workers, PPE "
        "status, equipment in use, equipment state.\n"
        "- **Activities and Operations**: What work is being performed? "
        "Describe the main activities, task sequences, material handling "
        "flows, and interactions between personnel and equipment.\n"
        "- **Overall Context**: Summarize the operational situation \u2014 "
        "what area of the facility, what shift or activity phase this "
        "appears to be, general pace and efficiency.\n\n"
        "Provide a holistic summary that captures the complete picture. "
        "Pay close attention to temporal and spatial details. "
        "Do not hallucinate or make up information."
    ),

    "normal_dense_caption": (
        "You are an expert warehouse operations analyst. Provide a dense, "
        "event-level video caption with precise timestamps for this video "
        "showing normal warehouse activity.\n\n"
        "Structure your response as a list of events. For each event, "
        "provide the start and end timestamp in the format "
        "<HH:MM:SS><HH:MM:SS> followed by a detailed description.\n\n"
        "Focus particularly on:\n"
        "1. **Precise Timing**: Accurately capture the start and end of "
        "each distinct action, movement, or state change.\n"
        "2. **Operational Activities**: Describe tasks being performed \u2014 "
        "picking, packing, loading, stacking, transporting, sorting.\n"
        "3. **Personnel Identification**: For every key worker, note "
        "their role, PPE status, position, and what they are handling.\n"
        "4. **Facility Context**: Note spatial relationships, equipment "
        "positions, and material flow directions.\n\n"
        "Format each line as:\n"
        "<Start_Timestamp><End_Timestamp> Description of the event.\n\n"
        "Example:\n"
        "<00:00:00><00:00:04> A loading dock area with two open bay "
        "doors; a worker in a yellow vest is scanning boxes on a pallet "
        "near bay 3.\n"
        "<00:00:04><00:00:08> A forklift approaches from the warehouse "
        "interior carrying a loaded pallet; the operator sounds the horn "
        "before entering the dock area.\n"
        "<00:00:08><00:00:12> The forklift places the pallet next to the "
        "truck at bay 2; a second worker begins transferring boxes from "
        "the pallet into the truck.\n"
        "<00:00:12><00:00:16> The first worker finishes scanning and "
        "affixes a shipping label to the pallet; the forklift reverses "
        "back toward the warehouse."
    ),

    "normal_chunk_caption": (
        "You are an expert warehouse operations analyst. You are viewing a "
        "strict {chunk_duration}-second window of a video showing normal "
        "warehouse activity with no safety incidents.\n"
        "Your job is NOT to tell the whole story, but to capture the "
        "**micro-dynamics** and **state changes** within this specific "
        "timeframe.\n\n"
        "Please describe the events in this {chunk_duration}-second window, "
        "focusing on:\n\n"
        "1. **Operational Dynamics**:\n"
        "   - Equipment movement paths and load status\n"
        "   - Worker task steps (picking, placing, scanning, stacking)\n"
        "   - Material flow direction and handling\n\n"
        "2. **Facility and Safety Context**:\n"
        "   - Floor condition, PPE compliance, zone markings\n"
        "   - Lighting and visibility conditions\n\n"
        "3. **Coordination and Interactions**:\n"
        "   - How workers coordinate with each other and with equipment\n"
        "   - Task handoffs, communication signals, spatial awareness\n\n"
        "Do not speculate on what happened before or after this clip. "
        "Report only what is visually confirmed in these "
        "{chunk_duration} seconds. Be precise and objective."
    ),

    # =========================================================================
    # HIGHLIGHT CHUNK (ANOMALY ONLY)
    # =========================================================================
    "highlight_timestamp_extraction": (
        "You are an expert warehouse safety analyst. Given the captions "
        "below for a video containing a safety incident, identify the "
        "EXACT timestamp (in seconds) when the primary incident occurs "
        "\u2014 the critical moment of contact, failure, or violation itself, "
        "not the lead-up behavior before it and not the aftermath.\n\n"
        "Use the chunk captions to cross-reference and narrow down the "
        "precise moment, since they provide detailed observations for "
        "specific time windows.\n\n"
        "Respond with ONLY a single number representing the timestamp in "
        "seconds (e.g., \"20\" or \"12.5\"). Nothing else.\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}"
    ),

    "highlight_chunk_caption": (
        "You are an expert warehouse safety analyst. You are viewing a "
        "critical {duration}-second highlight clip extracted from a longer "
        "video, centered on the moment a safety incident occurs.\n\n"
        "This clip spans from {start_time}s to {end_time}s of the "
        "original video. The incident is estimated to occur at "
        "approximately {anomaly_time}s.\n\n"
        "Provide an extremely detailed description of what happens in "
        "this clip, with special attention to:\n\n"
        "1. **Pre-Event State** (first 1\u20133 seconds):\n"
        "   - Exact positions of workers and equipment\n"
        "   - Equipment speed and load status (loaded/unloaded, load "
        "height)\n"
        "   - PPE status of all visible workers\n"
        "   - Floor condition and visibility in the area\n\n"
        "2. **The Critical Moment** (the incident itself):\n"
        "   - Exactly what happens \u2014 the collision, fall, dropped load, "
        "equipment failure, or safety violation\n"
        "   - Which workers and equipment are involved\n"
        "   - What safety protocol was violated or what went wrong\n\n"
        "3. **Immediate Aftermath** (last 1\u20133 seconds):\n"
        "   - Where workers and equipment end up\n"
        "   - Whether anyone is injured or materials are scattered\n"
        "   - Other workers' reactions (stopping, rushing to help, "
        "alerting others)\n\n"
        "Be extremely precise about identifying workers (PPE, role, "
        "position) and who does what. Do NOT hallucinate or assume "
        "actions you cannot confirm from the video."
    ),

    # =========================================================================
    # DESCRIPTION GENERATION (TEXT-TO-TEXT)
    # =========================================================================
    "anomaly_description": (
        "You are an expert warehouse safety analyst. You are provided with "
        "captions for a video containing a safety incident:\n"
        "1. Global Caption: General description of the facility area and "
        "events.\n"
        "2. Event-level Dense Caption: Detailed event sequence with "
        "timestamps.\n"
        "3. Chunk Captions: Detailed captions for small time windows.\n\n"
        "Task: Generate a detailed, organized, and cohesive description "
        "of the video with exactly these 3 parts:\n"
        "1. Holistic Scene Description \u2014 Describe the warehouse area: "
        "aisle layout, rack configuration, equipment present, lighting, "
        "floor condition, and general operational context.\n"
        "2. Temporal and Spatial Localization of Incident Events \u2014 List "
        "each key event in chronological order using this exact format "
        "(one event per line, timestamp in MM:SS):\n"
        "<start timestamp MM:SS><end timestamp MM:SS> [where in the "
        "facility] what happened\n"
        "3. Description of the Incident \u2014 Covering:\n"
        "   - Category of incident (equipment collision, fall, dropped "
        "load, PPE violation, unauthorized access, equipment failure)\n"
        "   - Detailed description of the event\n"
        "   - Time and duration\n"
        "   - Spatial location within the facility\n"
        "   - Safety protocol analysis: what rule or procedure was "
        "violated, what was the root cause (operator error, equipment "
        "malfunction, environmental hazard, inadequate training)\n"
        "   - Consequences (injury, damage, near-miss, operational "
        "disruption, secondary hazards)\n\n"
        "Instructions for using input data:\n"
        "- Use the [Global Caption] for general scene/event descriptions.\n"
        "- Use the [Dense Caption] to identify timestamps and temporal "
        "sequence of events.\n"
        "- Use [Chunk Captions] for detailed micro-level observations "
        "prior to, during, and after the incident.\n"
        "- If a [Highlight Chunk Caption] is provided, treat it as the "
        "most reliable source for the incident moment. Prioritize it "
        "over chunk captions for Parts 2 and 3. If no highlight is "
        "provided, rely on the other captions.\n"
        "- Resolve conflicts by using information most captions agree on.\n\n"
        "Output formatting rules (strictly enforced):\n"
        "- NEVER reference the source of information. Do not write "
        "phrases like \"the global caption states\", \"captions indicate\", "
        "or any similar attribution. Write as if you are directly "
        "observing the video.\n"
        "- Refer to workers by their visual attributes (PPE, clothing, "
        "role, position), not by ID numbers or labels.\n\n"
        "Input Captions:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "{dense_section}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "{highlight_section}"
    ),

    "normal_description": (
        "You are an expert warehouse operations analyst. You are provided "
        "with captions for a video showing normal warehouse activity.\n\n"
        "Task: Generate a detailed, organized, and cohesive description "
        "of the video with exactly these 3 parts:\n"
        "1. Holistic Scene Description \u2014 Describe the facility area: "
        "layout, equipment, lighting, floor condition, and operational "
        "context.\n"
        "2. Temporal and Spatial Localization of Key Events \u2014 List each "
        "notable event in chronological order using this exact format "
        "(one event per line, timestamp in MM:SS):\n"
        "<start timestamp MM:SS><end timestamp MM:SS> [where in the "
        "facility] what happened\n"
        "3. Event Description \u2014 An activity summary. Structure this "
        "section as bullet points, each in \"Key: value\" format. Cover "
        "the operational activities, worker coordination, material "
        "handling patterns, equipment usage, and safety compliance.\n\n"
        "Instructions:\n"
        "- Use the [Global Caption] for general scene and activity "
        "descriptions.\n"
        "- Use the [Dense Caption] for the timestamped sequence of events.\n"
        "- Use [Chunk Captions] for detailed micro-level observations.\n"
        "- Resolve conflicts by using what most sources agree on.\n"
        "- Do not fabricate events not present in the captions.\n"
        "- NEVER reference the source of information. Write as if you "
        "are directly observing the video.\n\n"
        "Input Captions:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "{dense_section}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}"
    ),

    # =========================================================================
    # QA GENERATION — MCQ
    # =========================================================================
    "anomaly_mcq": (
        "You are an expert warehouse safety analyst. You are provided with "
        "captions and a description for a video containing a safety "
        "incident.\n\n"
        "Task:\n"
        "1. Design a multiple-choice question about the safety incident "
        "that requires perception and reasoning to answer.\n"
        "2. The question should focus on one of: the root cause of the "
        "incident, the safety protocol violated, the direct consequence, "
        "the sequence of events, who or what was involved, or the "
        "category of incident.\n"
        "3. The question must be unambiguous: phrased so there is exactly "
        "one correct interpretation and one correct answer based on the "
        "video.\n"
        "4. Each wrong option must be clearly incorrect based on the "
        "video, not merely less likely. All options should be specific "
        "and parallel in structure.\n"
        "5. Provide step-by-step reasoning to derive the answer. The "
        "reasoning should proceed from observations (worker positions, "
        "equipment state, facility conditions) through causal inference "
        "to the conclusion.\n"
        "6. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\", \"Dense Caption\", or "
        "\"Chunk Captions\" \u2014 only mention the video.\n"
        "7. Ground your Question, Answer, and Reasoning on the provided "
        "data. Don't hallucinate or make up information.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Multiple-Choice Question: What safety violation led to the "
        "incident in the warehouse aisle?\n"
        "A. The forklift operator drove with an elevated load obscuring "
        "forward visibility\n"
        "B. A floor worker entered the aisle without checking for active "
        "forklift traffic\n"
        "C. The rack was overloaded beyond its rated capacity, causing "
        "structural failure\n"
        "D. The emergency stop system on the forklift was disabled\n"
        "======\n"
        "Answer: A\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the forklift traveling down the main aisle with "
        "a large pallet raised to approximately 2 meters, fully blocking "
        "the operator's forward line of sight. A floor worker is walking "
        "in the same aisle ahead of the forklift. The operator does not "
        "slow down or sound the horn because the load obstructs the view. "
        "The forklift clips the worker's leg near the end of the aisle. "
        "The rack structure shows no signs of overloading or failure, and "
        "the floor worker did check for traffic before entering \u2014 the "
        "issue is that the operator was driving with the load elevated "
        "rather than lowered to floor level as required by standard "
        "operating procedure.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Multiple-Choice Question: [Question]\n"
        "A. [Option A]\n"
        "B. [Option B]\n"
        "C. [Option C]\n"
        "D. [Option D]\n"
        "======\n"
        "Answer: [Answer] (should be a single letter choice)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_mcq": (
        "You are an expert warehouse operations analyst. You are provided "
        "with captions for a video with normal warehouse activity.\n\n"
        "Task:\n"
        "1. Design a multiple-choice question that requires perception "
        "and preferably reasoning to answer.\n"
        "2. The question should focus on: identifying what operation is "
        "being performed, who is involved, the sequence of task steps, "
        "material handling procedures, equipment usage, or safety "
        "compliance.\n"
        "3. The question must be unambiguous with exactly one correct "
        "answer. Each wrong option must be clearly incorrect, not "
        "merely less likely. All options should be specific and parallel.\n"
        "4. Provide step-by-step reasoning to derive the answer.\n"
        "5. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\" or \"Dense Caption\" \u2014 only "
        "mention the video.\n"
        "6. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Multiple-Choice Question: What task does the worker in the "
        "yellow vest perform after the forklift places the pallet at "
        "bay 2?\n"
        "A. Begins transferring boxes from the pallet into the truck\n"
        "B. Signals the forklift operator to bring another pallet\n"
        "C. Scans the pallet barcode and affixes a shipping label\n"
        "D. Moves the pallet to a staging area using a hand truck\n"
        "======\n"
        "Answer: A\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the forklift placing a loaded pallet at bay 2 "
        "at approximately 00:00:09. The worker in the yellow vest, who "
        "had been standing near the truck, immediately begins picking up "
        "boxes from the pallet and placing them into the truck bed. No "
        "scanner is visible and no barcode scanning occurs. The worker "
        "does not signal the forklift operator, who reverses independently. "
        "No hand truck is used \u2014 the worker carries boxes manually.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Multiple-Choice Question: [Question]\n"
        "A. [Option A]\n"
        "B. [Option B]\n"
        "C. [Option C]\n"
        "D. [Option D]\n"
        "======\n"
        "Answer: [Answer] (should be a single letter choice)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    # =========================================================================
    # QA GENERATION — BINARY
    # =========================================================================
    "anomaly_bcq": (
        "You are an expert warehouse safety analyst. You are provided with "
        "captions and a description for a video containing a safety "
        "incident.\n\n"
        "Task:\n"
        "1. Design two binary questions, one with answer \"Yes.\" and "
        "one with answer \"No.\". Don't make the questions too "
        "complicated. They should be straightforward but still require "
        "observing and reasoning about the video carefully.\n"
        "2. For each question, provide a step-by-step reasoning of how "
        "to derive the answer based on the video. Start from observation "
        "of the warehouse scene and the safety incident, then "
        "reason/analyze the question.\n"
        "3. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Don't mention the description "
        "in the Question, Answer, or Reasoning. Only mention the video.\n"
        "4. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Question: Was the forklift operator's load at a safe transport "
        "height when the incident occurred?\n"
        "======\n"
        "Answer: No. The forklift's load was elevated to approximately "
        "2 meters, well above the recommended transport height of "
        "15-20 cm from the floor.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the forklift traveling down the aisle with a "
        "large pallet raised high on the forks. The bottom of the pallet "
        "is roughly level with the top of the first rack tier, which is "
        "approximately 2 meters from the floor. Standard forklift "
        "operating procedure requires loads to be carried at the lowest "
        "practical height during transport. The elevated load blocks the "
        "operator's forward visibility and raises the vehicle's center "
        "of gravity, both of which contributed to the incident.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "1. Question: [Question]\n"
        "======\n"
        "Answer: Yes. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n"
        "=====\n"
        "2. Question: [Question]\n"
        "======\n"
        "Answer: No. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_bcq": (
        "You are an expert warehouse operations analyst. You are provided "
        "with captions for a video with normal warehouse activity.\n\n"
        "Task:\n"
        "1. Design two binary questions, one with answer \"Yes.\" and "
        "one with answer \"No.\". Don't make the questions too "
        "complicated. They should be straightforward but still require "
        "observing and reasoning about the video carefully. Focus on "
        "operational procedures, equipment usage, worker behavior, "
        "PPE compliance, or material handling.\n"
        "2. For each question, provide a step-by-step reasoning of "
        "how to derive the answer based on the video. Start from "
        "observation and description, then reason/analyze.\n"
        "3. Treat the description as a representation of the video "
        "itself. Don't mention the description. Only mention the video.\n"
        "4. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Question: Are all workers in the video wearing the required "
        "PPE for the warehouse floor?\n"
        "======\n"
        "Answer: Yes. Both workers visible in the video are wearing "
        "high-visibility vests and hard hats throughout.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows two workers in the loading dock area. The first "
        "worker, who is scanning boxes, wears a yellow high-visibility "
        "vest and a white hard hat throughout the clip. The second "
        "worker, who transfers boxes to the truck, wears an orange "
        "high-visibility vest and a white hard hat. Neither worker "
        "removes their PPE at any point during the video. Both meet "
        "the standard warehouse floor PPE requirement of vest and "
        "hard hat.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "1. Question: [Question]\n"
        "======\n"
        "Answer: Yes. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n"
        "=====\n"
        "2. Question: [Question]\n"
        "======\n"
        "Answer: No. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    # =========================================================================
    # QA GENERATION — OPEN-ENDED
    # =========================================================================
    "anomaly_open_qa": (
        "You are an expert warehouse safety analyst. You are provided with "
        "captions and a description for a video containing a safety "
        "incident.\n\n"
        "Task:\n"
        "1. Design an open-ended question about the safety incident that "
        "requires perception and reasoning to answer.\n"
        "2. The question should ask about the root cause, the safety "
        "protocol violated, the consequence, the sequence of events, or "
        "what should have been done differently to prevent the incident.\n"
        "3. Provide step-by-step reasoning to derive the answer. The "
        "reasoning should proceed from observations (worker positions, "
        "equipment state, facility conditions) through causal inference "
        "to the conclusion.\n"
        "4. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\", \"Dense Caption\", or "
        "\"Chunk Captions\" \u2014 only mention the video.\n"
        "5. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Open-ended Question: What safety procedures should the forklift "
        "operator have followed to prevent this incident?\n"
        "======\n"
        "Answer: The forklift operator should have lowered the load to "
        "the standard transport height of 15-20 cm from the floor before "
        "driving through the aisle. Transporting the load at "
        "approximately 2 meters blocked forward visibility entirely, "
        "preventing the operator from seeing the floor worker ahead. "
        "Additionally, the operator should have sounded the horn before "
        "entering the aisle and at the cross-aisle intersections, as "
        "required by standard operating procedure. With the load "
        "lowered and the horn sounded, the operator would have had a "
        "clear line of sight and the floor worker would have been "
        "alerted to the approaching forklift.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the forklift operator traveling through a "
        "warehouse aisle with the load elevated to approximately 2 "
        "meters \u2014 roughly the height of the first rack tier. This is "
        "far above the recommended transport height. The elevated load "
        "completely blocks the operator's view forward along the aisle.\n\n"
        "A floor worker enters the aisle from a cross-aisle and walks "
        "toward the approaching forklift. The worker checked for traffic "
        "before entering, but the forklift was not visible due to the "
        "racking. The operator, unable to see past the elevated load, "
        "does not slow down, brake, or sound the horn.\n\n"
        "The incident was preventable through two standard procedures: "
        "(1) lowering the load to transport height, which would have "
        "restored forward visibility, and (2) sounding the horn at "
        "cross-aisle intersections, which would have alerted the floor "
        "worker. Both are standard forklift operating requirements "
        "designed precisely for this scenario.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Open-ended Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_open_qa": (
        "You are an expert warehouse operations analyst. You are provided "
        "with captions for a video with normal warehouse activity.\n\n"
        "Task:\n"
        "1. Design an open-ended question that requires perception and "
        "preferably reasoning to answer.\n"
        "2. The question should ask about the operational workflow, how "
        "workers coordinate, what tasks are being performed, equipment "
        "usage patterns, or what the observable activity reveals about "
        "the facility's operations.\n"
        "3. Provide step-by-step reasoning to derive the answer.\n"
        "4. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\" or \"Dense Caption\" \u2014 only "
        "mention the video.\n"
        "5. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Open-ended Question: How do the two workers coordinate the "
        "loading operation in the video?\n"
        "======\n"
        "Answer: The two workers follow a clear division of labor in "
        "the loading operation. The first worker (yellow vest) handles "
        "inventory verification \u2014 scanning each box's barcode and "
        "affixing shipping labels to the pallets. The second worker "
        "(orange vest) handles the physical loading, transferring boxes "
        "from pallets into the truck. They work in parallel without "
        "blocking each other: the first worker processes pallets at "
        "bay 3 while the second loads at bay 2. The forklift operator "
        "delivers new pallets to the staging area between the bays, "
        "sounding the horn as required before entering the dock area. "
        "This spatial separation and role-based division allows "
        "continuous throughput with minimal idle time.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows a loading dock with two active bays. At bay 3, "
        "the worker in the yellow vest has a handheld scanner and is "
        "processing boxes on a pallet \u2014 scanning and labeling. At bay 2, "
        "the worker in the orange vest picks up boxes from a pallet and "
        "places them into the truck bed. The forklift arrives at 00:00:04 "
        "with a new pallet and deposits it between the bays. Neither "
        "worker stops their task during the forklift delivery. The "
        "spatial layout \u2014 verification at one bay, loading at another, "
        "forklift delivering to the staging area \u2014 creates a pipeline "
        "where all three roles operate concurrently.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Open-ended Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    "scene_description": (
        "You are an expert facility operations analyst. You are provided "
        "with a holistic and detailed description for a "
        "warehouse/facility surveillance video. Your task is to produce a "
        "**scene description caption** that captures the static, spatial "
        "layout of the scene — not the events that unfold over time.\n\n"
        "Task:\n"
        "1. Write a **question** — a natural-language question whose "
        "answer is the scene description caption. The question should ask "
        "about the physical setting, facility layout, or environmental "
        "context of the video (e.g., \"What does the warehouse interior "
        "look like in this video?\" or \"Describe the layout and "
        "infrastructure visible in the surveillance video.\").\n"
        "2. Write an **answer** — a concise yet thorough scene "
        "description covering:\n"
        "   - The camera perspective (overhead, ceiling-mounted CCTV, "
        "dock-mounted, perimeter-mounted, etc.).\n"
        "   - Lighting (overhead fluorescent / LED, natural light through "
        "bay doors, low-light/after-hours), and approximate time of day "
        "if discernible.\n"
        "   - Facility layout: aisle layout, racking/shelving rows and "
        "direction, dock bays, conveyor lines, staging areas, doorways, "
        "and floor markings (use frame-relative terms such as \"top to "
        "bottom\", \"left to right\").\n"
        "   - Infrastructure: rack tiers, dock doors and bay numbers, "
        "signage and floor labels, fire-safety equipment, security "
        "barriers, charging stations, and mounted surveillance.\n"
        "   - Stationary objects and environment: parked forklifts/pallet "
        "jacks, staged pallets, crates, signage text overlays, and any "
        "visible inventory.\n"
        "   - People/equipment present and their attributes (worker PPE "
        "color and type, equipment make/identifier when discernible).\n"
        "   Do NOT describe events, worker movements, equipment "
        "operation, or incidents — those belong in the event summary.\n"
        "3. Write a **reasoning** — a coherent paragraph explaining how "
        "observations from the video lead to each element of the answer. "
        "Proceed from what is directly visible to the inferences drawn "
        "(e.g., \"The double-deep racking and floor-marked aisles indicate "
        "a high-density storage layout …\"). Do not use bullet points or "
        "numbered lists.\n"
        "4. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Do not mention the "
        "description in the output.\n"
        "5. Ground your output on the video description. Do not "
        "hallucinate or fabricate details.\n\n"
        "Here are examples of the desired answer style and depth. (Note: "
        "these are for illustration only — your output must be based on "
        "the actual content of the video provided, not these examples.)\n"
        "<example>\n"
        "Answer: This is a fixed overhead surveillance recording from a "
        "ceiling-mounted CCTV camera inside a distribution warehouse "
        "during operating hours, lit by overhead LED fixtures. Two "
        "parallel aisles run from the bottom to the top of the frame, "
        "separated by double-deep selective racking populated with brown "
        "corrugated cartons and shrink-wrapped pallets. A cross-aisle "
        "bisects the frame horizontally near the bottom, with yellow "
        "floor tape demarcating pedestrian and equipment paths. A "
        "pallet-jack charging station is mounted against the right wall, "
        "and a fire extinguisher with red mounting bracket is visible at "
        "the cross-aisle pillar. A timestamp overlay reading \"2026-04-22 "
        "14:32\" appears in the top-left corner.\n"
        "</example>\n"
        "<example>\n"
        "Answer: The video captures a wide-angle view of a loading dock "
        "from a camera mounted above bay 4, looking down toward the dock "
        "floor. Three roll-up dock doors are visible (bays 2, 3, and 4), "
        "with bay 3's door open onto a backed-in delivery truck and bays "
        "2 and 4 closed. The dock floor extends from the foreground to "
        "the staging area in the background, where stacks of palletized "
        "cartons rest against the back wall. Yellow safety bollards line "
        "the dock-edge transition, and a hi-vis floor strip marks the "
        "operator-only zone. A digital sign on the back wall reads \"BAY 3 "
        "— INBOUND 14:00\", and overhead lighting comes from suspended LED "
        "panels. Two parked pallet jacks rest in the right corner near a "
        "charging post.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question about the scene layout]\n"
        "=====\n"
        "Answer: [Scene description caption]\n"
        "=====\n"
        "Reasoning: [Coherent paragraph deriving each answer element from "
        "video observations]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "event_summary": (
        "You are an expert facility operations analyst. You are provided "
        "with a holistic and detailed description for a "
        "warehouse/facility surveillance video. Your task is to produce "
        "an **event summary caption** that describes the key events and "
        "actions in the video — not the static scene layout.\n\n"
        "Task:\n"
        "1. Write a **question** — a natural-language question whose "
        "answer is the event summary caption. The question should ask "
        "about what happens in the video (e.g., \"What are the key events "
        "in this warehouse surveillance video?\" or \"Summarize the "
        "activity and any incidents observed in the video.\").\n"
        "2. Write an **answer** — a concise yet thorough event summary "
        "covering:\n"
        "   - A brief, high-level summary of all key events in the video.\n"
        "   - For normal operations: describe the activity flow, "
        "role-based handoffs, equipment usage, and any notable "
        "interactions between workers and equipment.\n"
        "   - For anomalous activity: briefly describe normal operations, "
        "then describe the incident in detail, including the actors "
        "involved (PPE color, role), equipment used, the sequence of "
        "events leading to the incident, and the physical aftermath "
        "(injury, dropped/damaged load, blocked egress, halted "
        "operations, etc.).\n"
        "   - **Root cause**: For any anomalous video, explicitly state "
        "the root cause of the incident (e.g., load lifted above "
        "transport height, missing PPE, ignored cross-aisle horn, "
        "unauthorized access, blocked dock-edge guard).\n"
        "   - Use frame-relative directions (top, bottom, left, right) "
        "and timestamps (MM:SS) where possible.\n"
        "3. Write a **reasoning** — a coherent paragraph explaining the "
        "chain of observations and inferences that support the answer. "
        "Start from what is visible, trace the sequence of events, and "
        "connect them to the stated root cause and outcome. Do not use "
        "bullet points or numbered lists.\n"
        "4. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Do not mention the "
        "description in the output.\n"
        "5. Ground your output on the video description. Do not "
        "hallucinate or fabricate details.\n\n"
        "Here are examples of the desired answer style and depth. (Note: "
        "these are for illustration only — your output must be based on "
        "the actual content of the video provided, not these examples.)\n"
        "<example>\n"
        "Answer: A counterbalance forklift carrying a pallet of "
        "shrink-wrapped cartons enters the aisle from the bottom of the "
        "frame at 00:00:03 with the load elevated to roughly 2 meters — "
        "well above standard transport height. A floor worker in an "
        "orange hi-vis vest enters the aisle from the right at 00:00:06 "
        "and begins walking up the aisle. The forklift operator, with "
        "forward visibility blocked by the elevated load, does not slow "
        "down or sound the horn at the cross-aisle, and the forklift's "
        "leading-edge fork strikes the worker's leg at 00:00:09, causing "
        "the worker to fall to the floor. The forklift halts immediately "
        "and a second worker rushes in from the top of the frame to "
        "assist. The root cause is the operator transporting a load above "
        "safe travel height (load lifted ~2 m vs. ~15-20 cm standard), "
        "which fully obscured forward visibility and prevented the "
        "operator from seeing the worker entering the aisle.\n"
        "</example>\n"
        "<example>\n"
        "Answer: Two dock workers complete a routine outbound load at bay "
        "3. The yellow-vested worker scans cartons with a handheld "
        "barcode scanner and applies shipping labels at the pallet "
        "staging area in the bottom of the frame between 00:00:01 and "
        "00:00:08. In parallel, the orange-vested worker transfers "
        "cartons from the labeled pallet into the trailer at bay 3 from "
        "00:00:04 onward. A counterbalance forklift enters at 00:00:10 "
        "from the right, sounds its horn at the cross-aisle, and deposits "
        "a fresh pallet in the staging area before reversing out of frame "
        "by 00:00:14. Both workers continue their tasks without "
        "interruption, exemplifying a normal parallel-pipeline loading "
        "operation.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question about the events in the video]\n"
        "=====\n"
        "Answer: [Event summary caption]\n"
        "=====\n"
        "Reasoning: [Coherent paragraph tracing the sequence of events "
        "and root cause from video observations]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_temporal_event_desc": (
        "You are an expert facility security and safety analyst and video "
        "annotator. You are provided with a holistic and detailed "
        "description for a warehouse/facility surveillance video "
        "containing an anomaly.\n\n"
        "Task:\n"
        "1. Identify a single anomalous event (e.g., a forklift strike, a "
        "fall, an unsecured-load drop, a PPE violation, an unauthorized "
        "access, a tampering event) and select [t1, t2] as the tight "
        "temporal window covering just that one event.\n\n"
        "2. Generate a question asking what happened in the video between "
        "[t1] and [t2]. The question must contain ONLY the timestamps and "
        "a neutral phrasing — no workers, equipment, actions, events, or "
        "outcomes.\n\n"
        "3. Provide an Answer as a single coherent paragraph describing "
        "what occurred in that window, including the inferred cause of "
        "the incident woven naturally into the description.\n\n"
        "4. Provide a Reasoning trace in 2-3 coherent paragraphs "
        "explaining your observations in addition to what changed and why "
        "during the window itself. Where applicable, include pre-event "
        "context before t1 (the normal operational baseline that makes "
        "the anomaly identifiable) and post-event context after t2 (the "
        "resulting state, response, or consequence).\n\n"
        "Constraints:\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use only timestamps supported "
        "by the provided captions. Do not hallucinate times.\n"
        "- The Question must contain ONLY the timestamps and a neutral "
        "phrasing — no workers, equipment, events, outcomes, or causes.\n"
        "- The Answer must be a single coherent paragraph. Do not use "
        "bullet points or numbered lists.\n"
        "- The Reasoning must be coherent paragraphs. Do not use bullet "
        "points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Do not mention the description in "
        "the Question, Answer, or Reasoning.\n\n"
        "Here is an example of the desired output:\n"
        "<example>\n"
        "Question: What happened in the video between 00:00:06 and "
        "00:00:09?\n"
        "======\n"
        "Answer: A counterbalance forklift carrying a pallet at well "
        "above standard transport height advances through the aisle and "
        "strikes a floor worker entering from the cross-aisle, causing "
        "the worker to fall to the floor. The operator's forward view was "
        "obstructed by the elevated load and no horn was sounded at the "
        "cross-aisle, so the worker — who had no auditory or visual "
        "warning of the forklift's approach — was hit at the moment they "
        "stepped into the lane. The root cause is travelling with the "
        "load lifted to roughly first-tier height instead of the standard "
        "15-20 cm, which fully obscured forward visibility.\n"
        "=====\n"
        "Reasoning:\n"
        "Before 00:00:06, the warehouse aisle is operating normally. The "
        "forklift is moving up the aisle from the bottom of the frame "
        "with a pallet of cartons elevated approximately 2 m, and the "
        "floor worker in an orange hi-vis vest is approaching the "
        "cross-aisle from the right but has not yet entered the lane. "
        "This baseline of an obstructed-vision operator and a worker "
        "about to enter an unannounced lane is what makes the impending "
        "impact identifiable as an anomaly the moment those paths "
        "converge.\n\n"
        "At 00:00:06, the worker steps into the aisle directly in front "
        "of the forklift. The operator, unable to see past the elevated "
        "load and not sounding the horn at the cross-aisle, does not slow "
        "or brake. The leading edge of the lower fork makes contact with "
        "the worker's leg and the worker is knocked to the floor.\n\n"
        "After 00:00:09, the forklift comes to an immediate stop, the "
        "operator dismounts, and a second worker enters from the top of "
        "the frame to assist. The aisle, which was an active travel lane "
        "moments earlier, is now an incident scene. The root cause of the "
        "strike is the operator's elevated-load travel — a procedural "
        "failure that made the impact effectively unavoidable once the "
        "worker's path crossed the forklift's path.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Neutral question with only timestamps]\n"
        "======\n"
        "Answer: [Single paragraph]\n"
        "=====\n"
        "Reasoning: [2-3 coherent paragraphs]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_temporal_event_desc": (
        "You are an expert video annotator and warehouse operations "
        "analyst. You are provided with a holistic and detailed "
        "description for a normal warehouse/facility surveillance video.\n\n"
        "Task:\n"
        "1. Identify a single, self-contained operational event (e.g., a "
        "forklift completing a pallet pick-up or drop-off, a worker "
        "scanning and labeling a pallet, a dock worker loading a carton, "
        "a pallet jack handoff, a dock door cycling open or closed) and "
        "select [t1, t2] as the tight temporal window covering just that "
        "one event.\n\n"
        "2. Generate a question asking what happened in the video between "
        "[t1] and [t2]. The question must contain ONLY the timestamps and "
        "a neutral phrasing — no workers, equipment, actions, events, or "
        "outcomes.\n\n"
        "3. Provide an Answer as a single coherent paragraph describing "
        "what occurred in that window and why the events unfolded as they "
        "did (standard operating procedure, role-based responsibilities, "
        "dock-flow rules, equipment-handling norms — woven naturally).\n\n"
        "4. Provide a Reasoning trace in 2-3 coherent paragraphs "
        "explaining your observations in addition to what changed and why "
        "during the window itself. Where applicable, include pre-event "
        "context before t1 (what led to the event or set the conditions "
        "for it) and post-event context after t2 (the resulting state "
        "once the event completed).\n\n"
        "Constraints:\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use only timestamps supported "
        "by the provided captions. Do not hallucinate times.\n"
        "- The Question must contain ONLY the timestamps and a neutral "
        "phrasing — no workers, equipment, events, outcomes, or causes.\n"
        "- The Answer must be a single coherent paragraph. Do not use "
        "bullet points or numbered lists.\n"
        "- The Reasoning must be coherent paragraphs. Do not use bullet "
        "points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Do not mention the description in "
        "the Question, Answer, or Reasoning.\n\n"
        "Here is an example of the desired output:\n"
        "<example>\n"
        "Question: What happened in the video between 00:00:04 and "
        "00:00:10?\n"
        "======\n"
        "Answer: A counterbalance forklift approaches the staging area "
        "between dock bays 2 and 3, sounds its horn at the cross-aisle, "
        "lowers a pallet of shrink-wrapped cartons onto the floor at the "
        "marked staging position, and reverses out of the frame. The "
        "operator follows standard dock-floor procedure throughout — horn "
        "before entering the bay area, load lowered to deposit height "
        "before approach, and a clean reverse exit without crossing the "
        "active loading lane.\n"
        "=====\n"
        "Reasoning:\n"
        "Before 00:00:04, the staging area between bays 2 and 3 is empty "
        "and the bay 3 worker (orange vest) is loading the trailer at the "
        "dock-edge while the bay 2 worker (yellow vest) is scanning "
        "cartons at a separate pallet. The forklift, carrying a fresh "
        "pallet, is rolling toward the staging area from the right side "
        "of the frame. This baseline of two active loading roles with a "
        "clear staging gap is the precondition for the upcoming pallet "
        "drop.\n\n"
        "At 00:00:04, the forklift sounds its horn at the cross-aisle and "
        "continues into the staging area. As it approaches the deposit "
        "position, the operator lowers the forks until the pallet rests "
        "on the marked floor zone, then begins reversing. The horn signal "
        "alerts both bay workers without interrupting their tasks, and "
        "the pallet is deposited cleanly without contacting either worker "
        "or the dock-edge guard.\n\n"
        "After 00:00:10, the forklift has fully exited the frame to the "
        "right, the new pallet is staged for the bay-3 worker to begin "
        "loading next, and both bay workers continue uninterrupted. The "
        "event is a textbook example of horn-first cross-aisle entry and "
        "lowered-load deposit, and the staging area is now repopulated "
        "for the next loading cycle.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Neutral question with only timestamps]\n"
        "======\n"
        "Answer: [Single paragraph]\n"
        "=====\n"
        "Reasoning: [2-3 coherent paragraphs]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_causal_linkage": (
        "You are an expert facility security and safety analyst. You are "
        "provided with a holistic and detailed description for a "
        "warehouse/facility surveillance video containing an anomaly.\n\n"
        "Task:\n"
        "1. Identify two distinct moments:\n"
        "   - [Timestamp A] (Reference Point): The triggering action, "
        "violation, or initial environmental condition.\n"
        "   - [Timestamp B] (Focus Point): The subsequent outcome, final "
        "rest position, or resulting consequence.\n"
        "2. Generate a \"Blind\" Open-ended Question about the warehouse "
        "incident or any significant event in the video: Ask about the "
        "relationship between the two moments using ONLY their timestamps.\n"
        "   - FORMAT example: \"Explain the relationship between the event "
        "at [Timestamp A] and the situation at [Timestamp B].\"\n"
        "   - CONSTRAINT: Do not mention any specific workers, equipment, "
        "actions, or outcomes in the question.\n"
        "   - Please design questions that require perception and "
        "cognition (reasoning) to answer.\n"
        "   - Note: The relationship between the two moments can be "
        "causal (cause/effect), procedural (rule/violation), or "
        "environmental (condition/reaction).\n"
        "3. Provide a Detailed Answer in a single paragraph:\n"
        "   - Begin by explicitly identifying the specific event "
        "occurring at [Timestamp A] and the specific situation at "
        "[Timestamp B].\n"
        "   - Follow with a narrative explanation of the logical, "
        "physical, or procedural connection.\n"
        "4. Provide a Reasoning Trace in 2–3 coherent paragraphs:\n"
        "   - Start from observation and description of the video, "
        "identify the events at A and B (what happens at [Timestamp A], "
        "what happens at [Timestamp B]). Describe the 'Normal' baseline "
        "state of the video before [Timestamp A] and identify "
        "environmental clues (floor markings, signage, PPE, charging "
        "stations, dock-edge guards) that define the rules of the scene.\n"
        "   - Trace the \"Logical Bridge\" and relevant events between A "
        "and B. Explain the mechanics (e.g., load momentum, obscured "
        "sightlines, blocked egress, missed audible alert) that connect "
        "the two moments. Highlight the key events between A and B that "
        "are needed to answer the question.\n"
        "   - Reason through the causal chain, and state the conclusion "
        "that answers the question.\n\n"
        "Constraints:\n"
        "- The question MUST be \"blind\" (refer only to timestamps).\n"
        "- The Answer must be a single, comprehensive narrative paragraph.\n"
        "- The Reasoning must be coherent paragraphs (no bullet points or "
        "lists).\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use the exact timestamps from "
        "the provided captions.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Don't mention the description in "
        "the Question, Answer, or Reasoning. Only mention the video "
        "itself.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: Explain the relationship between the event at 00:00:03 "
        "and the situation at 00:00:09.\n"
        "======\n"
        "Answer: At 00:00:03, a counterbalance forklift enters the aisle "
        "from the bottom of the frame transporting a palletized load "
        "lifted to approximately 2 m — well above the standard transport "
        "height — while at 00:00:09, a floor worker in an orange hi-vis "
        "vest is on the floor of the aisle in front of the now-stopped "
        "forklift. The relationship is a direct causal chain: the "
        "elevated load at 00:00:03 fully obscured the operator's forward "
        "view, so when the worker entered the cross-aisle and stepped "
        "into the forklift's path, the operator had no visual or auditory "
        "cue (no horn was sounded at the cross-aisle), and the leading "
        "fork made contact with the worker, producing the "
        "down-on-the-floor situation observed at 00:00:09.\n"
        "=====\n"
        "Reasoning:\n"
        "The video at 00:00:03 shows a forklift moving up a warehouse "
        "aisle with a pallet held at first-tier height, and the situation "
        "at 00:00:09 shows a worker fallen on the aisle floor with the "
        "forklift halted immediately behind. Looking at the wider "
        "context, the aisle is marked with yellow "
        "pedestrian-and-equipment lines and a \"horn before crossing\" "
        "decal at the cross-aisle pillar, and the cross-aisle is "
        "otherwise quiet. This baseline establishes the operator's "
        "elevated-load entry as a clear procedural deviation in an "
        "environment that explicitly warns against unannounced "
        "cross-aisle approaches.\n\n"
        "The logical bridge between these two moments is the convergence "
        "of an obstructed-vision operator and an entering pedestrian. The "
        "pallet at ~2 m fully blocks the operator's forward view; the "
        "operator does not lower the load, slow down, or sound the horn. "
        "Around 00:00:06, the worker enters the aisle from the right "
        "cross-aisle. The forklift, still travelling at steady speed, "
        "contacts the worker's leg with its leading fork by 00:00:08, "
        "halting only after impact.\n\n"
        "We can conclude that the situation at 00:00:09 is the terminal "
        "physical resolution of the procedural failure initiated at "
        "00:00:03. Tracing the chain — elevated-load entry, no horn, no "
        "slowdown, no avoidance — it is clear that the worker's fall is a "
        "direct, foreseeable consequence of an unannounced approach with "
        "obstructed forward vision. The relationship is a complete causal "
        "sequence in which the decision to travel with a high load made "
        "the impact effectively inevitable once any pedestrian crossed "
        "the lane.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. Do not "
        "use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_causal_linkage": (
        "You are an expert warehouse operations analyst. You are provided "
        "with a holistic and detailed description for a normal "
        "warehouse/facility surveillance video.\n\n"
        "Task:\n"
        "1. Identify two distinct moments in the video:\n"
        "   - [Timestamp A] (Reference Point): An initial action or "
        "condition (e.g., a worker beginning a scan-and-label step, a "
        "forklift sounding its horn at a cross-aisle, a dock door cycling "
        "open, the start of a pallet pick-up or drop-off).\n"
        "   - [Timestamp B] (Focus Point): The subsequent outcome (e.g., "
        "a labeled pallet ready for loading, the forklift completing its "
        "deposit, a trailer being loaded, a worker resuming throughput).\n"
        "2. Generate a \"Blind\" Open-ended Question about the relationship "
        "between the two moments using ONLY their timestamps:\n"
        "   - FORMAT example: \"Explain the relationship between the event "
        "at [Timestamp A] and the situation at [Timestamp B].\"\n"
        "   - CONSTRAINT: Do not mention any specific workers, equipment, "
        "actions, or outcomes in the question.\n"
        "   - Design questions that require perception and cognition "
        "(reasoning) to answer.\n"
        "   - The relationship can be causal (cause/effect), sequential "
        "(action/completion), or procedural (signal/response).\n"
        "3. Provide a Detailed Answer in a single paragraph:\n"
        "   - Begin by explicitly identifying the specific event at "
        "[Timestamp A] and the specific situation at [Timestamp B].\n"
        "   - Follow with a narrative explanation of the logical, "
        "physical, or procedural connection.\n"
        "4. Provide a Reasoning Trace in 2–3 coherent paragraphs:\n"
        "   - Start from observation and description of the video; "
        "identify the events at A and B (what happens at [Timestamp A], "
        "what happens at [Timestamp B]). Describe the baseline state "
        "before [Timestamp A] and any relevant environmental clues (floor "
        "markings, bay signage, PPE, charging stations).\n"
        "   - Trace the \"Logical Bridge\" and relevant events between A "
        "and B (e.g., scan-then-label sequence, horn-then-cross, "
        "pick-then-deposit). Highlight the key events between A and B "
        "needed to answer the question.\n"
        "   - Reason through the chain and state the conclusion that "
        "answers the question.\n\n"
        "Constraints:\n"
        "- The question MUST be \"blind\" (refer only to timestamps).\n"
        "- The Answer must be a single, comprehensive narrative paragraph.\n"
        "- The Reasoning must be coherent paragraphs (no bullet points or "
        "lists).\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use the exact timestamps from "
        "the provided captions.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Don't mention the description in "
        "the Question, Answer, or Reasoning. Only mention the video "
        "itself.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: Explain the relationship between the event at 00:00:04 "
        "and the situation at 00:00:10.\n"
        "======\n"
        "Answer: At 00:00:04, a counterbalance forklift sounds its horn "
        "at the cross-aisle as it begins approaching the dock staging "
        "area between bays 2 and 3, and at 00:00:10 a fresh pallet of "
        "shrink-wrapped cartons is resting on the marked staging zone "
        "with the forklift reversing out of frame. The relationship is "
        "sequential and procedural: the event at 00:00:04 is the operator "
        "following the horn-before-cross procedure that signals an "
        "inbound pallet to the bay workers, and the situation at 00:00:10 "
        "is the natural completion of that delivery — the pallet has been "
        "deposited at the marked staging position and the staging area is "
        "restocked for the bay-3 worker's next loading cycle without "
        "interrupting either worker.\n"
        "=====\n"
        "Reasoning:\n"
        "First, from observation of the video: the dock has two active "
        "loading bays. Before 00:00:04, the bay 3 worker (orange vest) is "
        "loading the trailer at the dock-edge and the bay 2 worker "
        "(yellow vest) is scanning and labeling cartons at a separate "
        "pallet. The forklift is rolling toward the staging area between "
        "the bays from the right side of the frame. The two timestamps to "
        "focus on are 00:00:04 and 00:00:10. At 00:00:04, the key event "
        "is the forklift's horn at the cross-aisle as the staging "
        "approach begins. At 00:00:10, the key situation is the freshly "
        "deposited pallet in the marked staging zone with the forklift "
        "reversing out of view.\n\n"
        "The logical bridge between these two moments is a controlled "
        "approach-deposit-reverse sequence. After the horn signal, the "
        "forklift continues into the staging area at the standard "
        "dock-floor approach speed and lowers the forks until the pallet "
        "rests on the marked floor zone. The operator then reverses "
        "cleanly without crossing the active bay-3 loading lane. "
        "Throughout, neither bay worker stops their task — the horn "
        "alerted them without requiring them to clear the area.\n\n"
        "We can conclude that the situation at 00:00:10 is the direct "
        "procedural result of the action at 00:00:04. The relationship is "
        "one of signal-and-completion: a properly announced staging-area "
        "entry leads to a clean pallet deposit and a restocked staging "
        "zone, ready for the next loading cycle. There is no anomaly; the "
        "sequence follows standard horn-before-cross and lowered-load "
        "deposit procedure.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. Do not "
        "use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_temporal_localization": (
        "You are an expert video annotator and logical analyst "
        "specializing in temporal localization. You are provided with a "
        "holistic and detailed description for a warehouse/facility "
        "surveillance video containing an anomaly.\n\n"
        "Task:\n"
        "1. Identify a significant event or anomaly that requires logical "
        "reasoning to localize.\n"
        "2. Generate a Temporal Localization Question in the format: "
        "\"When does [Event X] occur in the video?\"\n"
        "3. Provide the Answer with exact start and end timestamps.\n"
        "4. Provide a detailed Reasoning paragraph following a "
        "three-phase logical trace:\n"
        "   - Describe the state of the video immediately before the "
        "start timestamp. Identify the specific procedures, PPE rules, or "
        "operational norms that were being followed to establish why the "
        "incident had NOT yet begun.\n"
        "   - Pinpoint the exact visual 'state-change' at the start "
        "timestamp. Explain the root cause (e.g., elevated-load travel, "
        "missed horn, blocked egress, ignored PPE) that initiated the "
        "incident.\n"
        "   - Justify the end timestamp by describing the transition into "
        "a new, stable, or static state (e.g., forklift coming to a halt, "
        "worker on the floor receiving aid, dropped load fully settled, "
        "dock-edge guard re-engaged, intruder leaving the frame).\n\n"
        "Constraints:\n"
        "- The reasoning must follow the 'Sandwich Structure': Baseline "
        "-> Trigger -> Resolution.\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Do not hallucinate times. "
        "Cross-reference all provided captions.\n"
        "- The Reasoning section must be a coherent, dense paragraph. Do "
        "not use bullet points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Do not mention the description; describe the video directly.\n\n"
        "Here is an example of the desired content style and depth. You "
        "don't need to follow this example exactly, but you should follow "
        "the same format and the style:\n"
        "<example>\n"
        "Question: When does the forklift strike against the floor worker "
        "occur?\n"
        "======\n"
        "Answer: Start_Time: 00:00:06, End_Time: 00:00:09\n"
        "=====\n"
        "Reasoning:\n"
        "The video begins with a stable baseline of normal aisle "
        "activity. Between 00:00:00 and 00:00:06, a counterbalance "
        "forklift travels up the aisle from the bottom of the frame "
        "carrying a pallet at approximately 2 m height — already a "
        "procedural deviation from the 15-20 cm transport standard, but "
        "with no other actors in the lane. The cross-aisle on the right "
        "is clear, and the floor markings and \"horn before crossing\" "
        "decal at the cross-aisle pillar establish the expected "
        "operational rules. The event is triggered at 00:00:06 when a "
        "floor worker in an orange hi-vis vest enters the aisle from the "
        "right cross-aisle and steps into the forklift's path. The "
        "root-cause failure is the elevated load fully blocking the "
        "operator's forward view combined with no horn at the "
        "cross-aisle, which left both the operator and the worker without "
        "any cue to the impending convergence. The leading edge of the "
        "lower fork strikes the worker's leg at 00:00:08, and the worker "
        "is knocked to the floor. The event reaches a resolution at "
        "00:00:09, by which time the forklift has come to a complete halt "
        "with the worker on the aisle floor in front of it and a second "
        "worker beginning to enter the frame from the top to assist, "
        "transitioning the scene from an active operation to a stationary "
        "incident.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: Start_Time: [MM:SS], End_Time: [MM:SS]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as a coherent paragraph. "
        "Describe the 'before-state' that led to the start timestamp, the "
        "visual triggers during the event, and the 'after-state' or "
        "resolution that confirms the event ended at the specific "
        "timestamp.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_temporal_localization": (
        "You are an expert video annotator and logical analyst "
        "specializing in temporal localization. You are provided with a "
        "holistic and detailed description for a normal "
        "warehouse/facility surveillance video.\n\n"
        "Task:\n"
        "1. Identify a significant operational event that requires "
        "logical reasoning to localize. Focus on routine warehouse events "
        "such as:\n"
        "   - A forklift completing a pallet pick-up or drop-off\n"
        "   - A worker scanning and labeling a carton or pallet\n"
        "   - A dock worker completing a loading or unloading cycle for a "
        "bay\n"
        "   - A pallet jack or hand truck handoff between workers\n"
        "   - A dock door cycling open or closed\n"
        "   - Activity flow pattern changes (e.g., shift handoff, queue "
        "forming or clearing at a bay)\n\n"
        "2. Generate a Temporal Localization Question in the format: "
        "\"When does [Event X] occur in the video?\"\n\n"
        "3. Provide the Answer with exact start and end timestamps.\n\n"
        "4. Provide a detailed Reasoning paragraph following a "
        "three-phase logical trace:\n"
        "   - Describe the state of the video immediately before the "
        "start timestamp. What was the operational state, worker "
        "position, or equipment configuration that preceded this event?\n"
        "   - Pinpoint the exact visual 'state-change' at the start "
        "timestamp. What action or movement initiated the event?\n"
        "   - Justify the end timestamp by describing the transition into "
        "a new, stable state (e.g., pallet at rest in the staging zone, "
        "scanner returning to its dock, dock door fully open, worker "
        "resuming the next cycle).\n\n"
        "Constraints:\n"
        "- The reasoning must follow the 'Sandwich Structure': "
        "Before-State -> Action/Transition -> After-State.\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Do not hallucinate times. "
        "Cross-reference all provided captions.\n"
        "- The Reasoning section must be a coherent, dense paragraph. Do "
        "not use bullet points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Do not mention the description; describe the video directly.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: When does the forklift complete its pallet drop-off at "
        "the dock staging area?\n"
        "======\n"
        "Answer: Start_Time: 00:00:04, End_Time: 00:00:10\n"
        "=====\n"
        "Reasoning:\n"
        "Prior to 00:00:04, the counterbalance forklift is rolling toward "
        "the dock staging area between bays 2 and 3 from the right of the "
        "frame, carrying a pallet of shrink-wrapped cartons at standard "
        "transport height. The bay 2 worker (yellow vest) is scanning "
        "cartons at a separate pallet, the bay 3 worker (orange vest) is "
        "loading the trailer at the dock-edge, and the staging zone is "
        "empty — the precondition for an incoming deposit. At 00:00:04, "
        "the forklift sounds its horn at the cross-aisle, signalling the "
        "dock workers and initiating the staged-approach phase of the "
        "deposit. As the forklift continues into the staging area, the "
        "operator gradually lowers the forks; by 00:00:08, the pallet has "
        "touched down on the marked staging zone, and the operator begins "
        "reversing cleanly without crossing the active bay-3 loading "
        "lane. By 00:00:10, the forklift has fully exited the frame to "
        "the right and the pallet is at rest in the staging zone, "
        "restocked and ready for the bay-3 worker's next loading cycle, "
        "marking the conclusion of the drop-off maneuver.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: Start_Time: [MM:SS], End_Time: [MM:SS]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as a coherent paragraph. "
        "Describe the 'before-state' that led to the start timestamp, the "
        "action/transition during the event, and the 'after-state' that "
        "confirms the event ended at the specific timestamp.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),
}


def get_prompt(key, **kwargs):
    """Retrieve and optionally format a prompt template by key."""
    template = PROMPT_TEMPLATES.get(key)
    if template is None:
        raise ValueError(f"No prompt template found for key: {key}")
    if kwargs:
        return template.format(**kwargs)
    return template

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs an API key, Needs Python

The author says: Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-generate-video-reasoning-annotations · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-generate-video-reasoning-annotations.zip