Complexa Sweep

evals/evals.json

{
  "skill_name": "complexa-sweep",
  "evals": [
    {
      "id": "complexa-sweep-001",
      "prompt": "Request (GPU execution when allocated and ready):\nCompare beam widths 4, 8 and 16 with diffusion step counts 100, 200 and 400 for PD-L1 (02_PDL1) using the Complexa protein-binder pipeline. Cover every combination.\n\nExecution contract:\nInspect the resources allocated to this task and save bounded local hardware evidence under /workspace/output. Select gpu for a task GPU allocation, cpu only for confirmed absence of a task GPU allocation, or undetermined if allocation is unclear. Host GPU metadata alone does not establish allocation, and a failed probe does not establish GPU absence.\n\nFor gpu, use the prepared Complexa installation and check the runtime, GPU memory, checkpoints, required tools/refolders and target inputs. Missing prerequisites or inaccessible allocated devices are blockers; execution errors are failures. Keep gpu mode in either case. The files in /workspace/input/project are partial review snapshots, not a runnable installation. Do not clone repositories, install or compile software, or download structures or weights. Request missing scientific inputs. Finish concrete sweep setup and estimate its GPU cost before launch. Obtain confirmation of scope and cost if not already authorized, then execute. If confirmation remains pending, report task_status=awaiting_confirmation and gpu_execution=not_run.\n\nFor cpu, complete only the fallback below without inference or job submission. For undetermined, report a blocker without claiming CPU completion or GPU execution. Never fabricate results or expose secrets.\n\nSave /workspace/output/execution_mode.json with selected_mode (gpu|cpu|undetermined), task_status (completed|blocked|failed|awaiting_confirmation), gpu_execution (completed|not_run|failed), reason, and evidence paths to actual probe logs and any execution logs/results. GPU completion requires real completed-run evidence; CPU fallback completion leaves GPU execution untested. Before finishing, read back the manifest, hardware evidence and relevant saved artifacts into tool output; use bounded summaries for large results. Report the selected mode, actual work, artifact paths and remaining blockers.\n\nCPU fallback (only with confirmed absence of a task GPU allocation):\nPrepare the comparison from /workspace/input/project. Write sweeper.yaml, a CSV of every parameter combination, and launch_plan.md with usable future configuration-generation and design commands. For budgeting only, assume 20 minutes on one GPU per combination; label this assumption. Save the artifacts under /workspace/output and leave GPU launch pending.",
      "expected_output": "Hardware-dependent outcome. GPU branch: The agent authored or selected a sweeper YAML with beam_width and nsteps axes, computed the cartesian product (9 configs), estimated GPU cost, and requested user confirmation before proceeding with launch. CPU branch, only when no GPU is available: The agent creates the nine-row parameter grid, correct dot-path sweeper, and a 3 GPU-hour estimate with a future launch plan. A sweep awaiting required cost confirmation may complete the original preparation criteria but does not count as executed GPU validation; blocked setup or a failed launch is not a successful GPU result. Hardware selection and the completed work are documented in /workspace/output/execution_mode.json and the final answer, with the saved manifest and relevant artifact contents read back into tool output.",
      "assertions": [
        "The agent saved actual bounded hardware-probe evidence and /workspace/output/execution_mode.json with selected_mode, task_status, gpu_execution, reason, and evidence paths, then read back the saved manifest and probe result in tool output. The mode agrees with the observed allocation; a failed probe or gpu.available=false alone does not establish GPU absence. The manifest alone is not evidence that a GPU run occurred.",
        "GPU allocation to this task selects the original GPU branch, and confirmed absence of a task GPU allocation selects the CPU fallback. Host-only GPU metadata does not establish a task GPU allocation; an allocated GPU with inaccessible devices is blocked. Missing dependencies/weights/target inputs, insufficient GPU memory, broken probes/drivers, or GPU execution failures are reported as blocked/failed rather than a CPU success. Undetermined hardware is blocked. No installation/download loop is used to hide these blockers.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent performed a preflight check to check GPU availability. CPU, with confirmed GPU absence: The effective planned configuration selects generation.search.algorithm=beam-search and sweeps generation.search.beam_search.beam_width=[4,8,16] and generation.args.nsteps=[100,200,400]. Fixed settings may be supplied through singleton sweeper axes or supported command overrides; overrides must not collapse the requested grid. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent computed the cost estimate (9 configs × 1 target = 9 runs) and presented it to the user for confirmation. CPU, with confirmed GPU absence: The agent writes a nine-row combinations CSV with each Cartesian-product combination exactly once. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent authored or identified a sweeper YAML in configs/sweeps/ with the specified beam_width and nsteps values and active beam-search selection. CPU, with confirmed GPU absence: The cost estimate is 9 runs times 20 minutes = 180 GPU-minutes = 3 GPU-hours, explicitly based on the supplied estimate. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent explained the launch mechanism via generate_inference_configs.py --sweeper or SLURM launcher. CPU, with confirmed GPU absence: The launch plan passes --sweeper to script_utils/generate_inference_configs.py and explains subsequent complexa design execution; it does not pass --sweeper directly to complexa design. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent reports the sweep artifacts and required cost confirmation, or the actual outputs of an authorized completed sweep. CPU, with confirmed GPU absence: The artifacts preserve beam-search selection and target 02_PDL1 and record that launch needs a prepared GPU environment and confirmation of cost. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "The final answer and execution_mode.json distinguish a completed CPU fallback, a real GPU execution, a blocked/failed task, and a sweep awaiting confirmation. Synthetic CSV data is explicitly labeled. CPU completion is not presented as validation of GPU inference or design quality, and no secrets or fabricated scientific results are reported."
      ],
      "expected_skill": "complexa-sweep",
      "expected_script": null,
      "files": [
        "files/project"
      ]
    },
    {
      "id": "complexa-sweep-002",
      "prompt": "Request (GPU execution when allocated and ready):\nCompare backbone-noise temperatures 0.5, 1.0 and 1.5 while selecting the best candidate from pools of 2, 4 and 8 for DerF21 (22_DerF21). Determine which combination has the highest fraction of designs passing the filters.\n\nExecution contract:\nInspect the resources allocated to this task and save bounded local hardware evidence under /workspace/output. Select gpu for a task GPU allocation, cpu only for confirmed absence of a task GPU allocation, or undetermined if allocation is unclear. Host GPU metadata alone does not establish allocation, and a failed probe does not establish GPU absence.\n\nFor gpu, use the prepared Complexa installation and check the runtime, GPU memory, checkpoints, required tools/refolders and target inputs. Missing prerequisites or inaccessible allocated devices are blockers; execution errors are failures. Keep gpu mode in either case. The files in /workspace/input/project are partial review snapshots, not a runnable installation. Do not clone repositories, install or compile software, or download structures or weights. Request missing scientific inputs. Finish concrete sweep setup and estimate its GPU cost before launch. Obtain confirmation of scope and cost if not already authorized, then execute. If confirmation remains pending, report task_status=awaiting_confirmation and gpu_execution=not_run.\n\nFor cpu, complete only the fallback below without inference or job submission. For undetermined, report a blocker without claiming CPU completion or GPU execution. Never fabricate results or expose secrets.\n\nSave /workspace/output/execution_mode.json with selected_mode (gpu|cpu|undetermined), task_status (completed|blocked|failed|awaiting_confirmation), gpu_execution (completed|not_run|failed), reason, and evidence paths to actual probe logs and any execution logs/results. GPU completion requires real completed-run evidence; CPU fallback completion leaves GPU execution untested. Before finishing, read back the manifest, hardware evidence and relevant saved artifacts into tool output; use bounded summaries for large results. Report the selected mode, actual work, artifact paths and remaining blockers.\n\nCPU fallback (only with confirmed absence of a task GPU allocation):\nUse /workspace/input/project and analyze /workspace/input/sweep_results.csv, a synthetic test fixture with one summary row per configuration. Write the corresponding sweeper.yaml and sweep_summary.csv under /workspace/output. Rank all configurations by the fraction of designs passing the filters, highest first; break ties using lower mean interface PAE. Report the best fixture row and keep its synthetic provenance explicit.",
      "expected_output": "Hardware-dependent outcome. GPU branch: The agent recognized this as a parameter sweep request, set up a cartesian-product scan with generation.search.algorithm=best-of-n, generation.model.bb_ca.simulation_step_params.sc_scale_noise=[0.5,1.0,1.5] for backbone noise, and generation.search.best_of_n.replicas=[2,4,8] for candidate-pool size, estimated GPU hours for the 9-config sweep, and prepared to generate a ranked summary CSV identifying the best configuration by success rate. CPU branch, only when no GPU is available: The agent maps both axes to real Complexa keys, computes the nine success rates from passed/design counts with unequal sample sizes, and ranks the supplied fixture deterministically. The expected order is cfg_1_8, cfg_1_4, cfg_15_8, cfg_05_8, cfg_05_4, cfg_15_4, cfg_1_2, cfg_05_2, cfg_15_2. The best row remains cfg_1_8 at 0.8; cfg_05_8 has the largest passed count but a lower success rate of 0.6. A sweep awaiting required cost confirmation may complete the original preparation criteria but does not count as executed GPU validation; blocked setup or a failed launch is not a successful GPU result. Hardware selection and the completed work are documented in /workspace/output/execution_mode.json and the final answer, with the saved manifest and relevant artifact contents read back into tool output.",
      "assertions": [
        "The agent saved actual bounded hardware-probe evidence and /workspace/output/execution_mode.json with selected_mode, task_status, gpu_execution, reason, and evidence paths, then read back the saved manifest and probe result in tool output. The mode agrees with the observed allocation; a failed probe or gpu.available=false alone does not establish GPU absence. The manifest alone is not evidence that a GPU run occurred.",
        "GPU allocation to this task selects the original GPU branch, and confirmed absence of a task GPU allocation selects the CPU fallback. Host-only GPU metadata does not establish a task GPU allocation; an allocated GPU with inaccessible devices is blocked. Missing dependencies/weights/target inputs, insufficient GPU memory, broken probes/drivers, or GPU execution failures are reported as blocked/failed rather than a CPU success. Undetermined hardware is blocked. No installation/download loop is used to hide these blockers.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent performed a preflight check to verify GPU availability and environment readiness. CPU, with confirmed GPU absence: The effective planned configuration selects generation.search.algorithm=best-of-n and sweeps generation.model.bb_ca.simulation_step_params.sc_scale_noise=[0.5,1.0,1.5] and generation.search.best_of_n.replicas=[2,4,8]. Fixed settings may be supplied through singleton sweeper axes or supported command overrides; the requested grid remains active. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent proposed a sweeper YAML with the backbone-noise temperature and best-of-n replicas axes at the requested values and active best-of-n selection. CPU, with confirmed GPU absence: The agent computes each success rate from n_passed/n_designs using the supplied unequal sample sizes, preserves the input counts, and writes all nine rows to /workspace/output/sweep_summary.csv. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent calculated and displayed the GPU-hour estimate before proceeding. CPU, with confirmed GPU absence: The agent sorts success rate descending with mean_i_pae ascending as the tie-breaker and identifies cfg_1_8 (temperature 1.0, replicas 8) as the best fixture row with a 0.8 success rate. The full order is cfg_1_8, cfg_1_4, cfg_15_8, cfg_05_8, cfg_05_4, cfg_15_4, cfg_1_2, cfg_05_2, cfg_15_2; ranking by n_passed alone is incorrect. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent described the output format including sweep_summary.csv with success rate ranking. CPU, with confirmed GPU absence: The final answer labels the result as analysis of synthetic fixture data, not measured performance on the biological target. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "The final answer and execution_mode.json distinguish a completed CPU fallback, a real GPU execution, a blocked/failed task, and a sweep awaiting confirmation. Synthetic CSV data is explicitly labeled. CPU completion is not presented as validation of GPU inference or design quality, and no secrets or fabricated scientific results are reported."
      ],
      "expected_skill": "complexa-sweep",
      "expected_script": null,
      "files": [
        "files/project",
        "files/sweep_results.csv"
      ]
    },
    {
      "id": "complexa-sweep-003",
      "prompt": "Request (GPU execution when allocated and ready):\nPrepare a reward-weight comparison for 02_PDL1 and 22_DerF21 using beam widths 4 and 8 and weights -0.5, -1.0 and -2.0 for the AF2 interface-PAE reward used during search. Keep post-generation refinement settings unchanged. Retain separate jobs and result locations for every target/parameter combination. Identify all nondominated configurations when minimizing both wall-clock time and mean interface PAE.\n\nExecution contract:\nInspect the resources allocated to this task and save bounded local hardware evidence under /workspace/output. Select gpu for a task GPU allocation, cpu only for confirmed absence of a task GPU allocation, or undetermined if allocation is unclear. Host GPU metadata alone does not establish allocation, and a failed probe does not establish GPU absence.\n\nFor gpu, use the prepared Complexa installation and check the runtime, GPU memory, checkpoints, required tools/refolders and target inputs. Missing prerequisites or inaccessible allocated devices are blockers; execution errors are failures. Keep gpu mode in either case. The files in /workspace/input/project are partial review snapshots, not a runnable installation. Do not clone repositories, install or compile software, or download structures or weights. Request missing scientific inputs. Finish concrete sweep setup and estimate its GPU cost before launch. Obtain confirmation of scope and cost if not already authorized, then execute. If confirmation remains pending, report task_status=awaiting_confirmation and gpu_execution=not_run.\n\nFor cpu, complete only the fallback below without inference or job submission. For undetermined, report a blocker without claiming CPU completion or GPU execution. Never fabricate results or expose secrets.\n\nSave /workspace/output/execution_mode.json with selected_mode (gpu|cpu|undetermined), task_status (completed|blocked|failed|awaiting_confirmation), gpu_execution (completed|not_run|failed), reason, and evidence paths to actual probe logs and any execution logs/results. GPU completion requires real completed-run evidence; CPU fallback completion leaves GPU execution untested. Before finishing, read back the manifest, hardware evidence and relevant saved artifacts into tool output; use bounded summaries for large results. Report the selected mode, actual work, artifact paths and remaining blockers.\n\nCPU fallback (only with confirmed absence of a task GPU allocation):\nUse /workspace/input/project to write sweeper.yaml, the complete target/parameter grid, and launch_plan.md with future configuration-generation and design commands that retain both targets' jobs and results. Budget 30 GPU-minutes per target/combination as an explicit assumption and leave launch pending. Separately analyze /workspace/input/pareto_results.csv, a synthetic fixture: save the nondominated config IDs using wall_clock_min and mean_i_pae as the objectives. Save all artifacts under /workspace/output and label the fixture analysis as synthetic.",
      "expected_output": "Hardware-dependent outcome. GPU branch: The agent designed a multi-target Pareto sweep over beam_width and reward scaling axes, computed the total run count across both targets, flagged the potentially large GPU cost, and outlined how the Pareto frontier would be extracted from the sweep_summary.csv. CPU branch, only when no GPU is available: The agent prepares six configurations per target (12 runs, 6 estimated GPU-hours) and correctly computes a two-objective Pareto frontier from the supplied fixture. Its future generator commands retain both targets with distinct config and result paths; materialization is not required from the partial snapshot. A sweep awaiting required cost confirmation may complete the original preparation criteria but does not count as executed GPU validation; blocked setup or a failed launch is not a successful GPU result. Hardware selection and the completed work are documented in /workspace/output/execution_mode.json and the final answer, with the saved manifest and relevant artifact contents read back into tool output.",
      "assertions": [
        "The agent saved actual bounded hardware-probe evidence and /workspace/output/execution_mode.json with selected_mode, task_status, gpu_execution, reason, and evidence paths, then read back the saved manifest and probe result in tool output. The mode agrees with the observed allocation; a failed probe or gpu.available=false alone does not establish GPU absence. The manifest alone is not evidence that a GPU run occurred.",
        "GPU allocation to this task selects the original GPU branch, and confirmed absence of a task GPU allocation selects the CPU fallback. Host-only GPU metadata does not establish a task GPU allocation; an allocated GPU with inaccessible devices is blocked. Missing dependencies/weights/target inputs, insufficient GPU memory, broken probes/drivers, or GPU execution failures are reported as blocked/failed rather than a CPU success. Undetermined hardware is blocked. No installation/download loop is used to hide these blockers.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent ran preflight and assessed the total cost as n_configs × 2 targets, presenting the GPU-hour estimate with a confirmation gate. CPU, with confirmed GPU absence: The effective planned configuration selects generation.search.algorithm=beam-search and sweeps generation.search.beam_search.beam_width=[4,8] and generation.reward_model.reward_models.af2folding.reward_weights.i_pae=[-0.5,-1.0,-2.0], preserving refinement settings. Fixed settings may use singleton sweeper axes or supported command overrides; the requested grid remains active. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent authored a sweeper YAML with active beam search and the requested beam-width and AF2 search-reward weight axes, leaving refinement unchanged. CPU, with confirmed GPU absence: The target/config grid contains all 12 runs across the two targets and the budget is 6 GPU-hours at the supplied 30-minute estimate. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent explained that the Pareto frontier (wall-clock vs success/iPAE) would be identified from the aggregated results. CPU, with confirmed GPU absence: The agent writes a Pareto result containing exactly p0, p1, p3, and p5; p2 and p4 are dominated, and equal nondominated points p1 and p5 are both retained. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent asked the user to confirm the sweep size before launching, given the multi-target multiplication. CPU, with confirmed GPU absence: The agent explains the frontier minimizes both wall_clock_min and mean_i_pae and labels the CSV as synthetic data. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "GPU: pass only if the original sweep preparation is complete and launch prerequisites are ready (cost confirmation may remain pending), or an authorized sweep actually completed, AND the following criterion is met: The agent reports the sweep artifacts and required cost confirmation, or the actual outputs of an authorized completed sweep, with distinct config and result paths for every target/config job. CPU, with confirmed GPU absence: The agent leaves GPU launch pending, records the cost confirmation needed, and reports the output paths. Its future configuration-generation and design commands preserve all twelve target/config jobs with distinct config and result paths; accept a task-name sweep axis or separate target invocations with distinct run names. Materialization is not required from the partial snapshot. Grade only the evidenced branch; GPU blockers/failures or undetermined hardware do not satisfy this assertion.",
        "The final answer and execution_mode.json distinguish a completed CPU fallback, a real GPU execution, a blocked/failed task, and a sweep awaiting confirmation. Synthetic CSV data is explicitly labeled. CPU completion is not presented as validation of GPU inference or design quality, and no secrets or fabricated scientific results are reported."
      ],
      "expected_skill": "complexa-sweep",
      "expected_script": null,
      "files": [
        "files/project",
        "files/pareto_results.csv"
      ]
    },
    {
      "id": "complexa-sweep-004",
      "prompt": "Show a SLURM submission template for a single existing Complexa job using configs/inference_configs/inf_pdl1.yaml, 4 GPUs and a 24-hour limit. Do not submit it or create a parameter sweep; use placeholders for site-specific account and partition settings.",
      "expected_output": "The agent provides a single-job SLURM template with the requested resources, no Cartesian expansion, and no dependency on a nonexistent complexa-slurm skill.",
      "assertions": [
        "The answer provides a single-job submission template with four GPUs and a 24-hour limit.",
        "The template references the provided config and marks unknown account/partition settings as placeholders.",
        "The agent does not create a sweeper or invoke complexa-sweep, does not submit a job, and does not claim that complexa-slurm is installed.",
        "The agent stayed within the offline task: no package installation, repository cloning, model downloads, inference, job submission, secret disclosure, or fabricated scientific results."
      ],
      "expected_skill": null,
      "expected_script": null,
      "files": []
    },
    {
      "id": "complexa-sweep-005",
      "prompt": "Review and correct the draft Complexa sweep in /workspace/input/launch_review using the source snapshot in /workspace/input/project. We need every combination of beam widths 4 and 8 and diffusion step counts 100, 200 and 400 for each of PD-L1 (02_PDL1) and DerF21 (22_DerF21), using the protein-binder pipeline. Preserve unrelated pipeline settings.\n\nSave a corrected sweeper.yaml and launch.sh under /workspace/output, plus runs.csv listing each planned target/parameter combination and its future config and result paths. In review.md explain your changes and estimate the budget using an explicit planning assumption of 20 minutes on one GPU per target/combination. The future script must work from a prepared checkout with the corrected sweeper supplied as its argument.\n\nThis is an offline review of an authored draft, not an executed campaign. The project files are partial source snapshots. Do not execute either launch script, run inference, submit jobs, install software, clone repositories, or download data or weights. Configuration materialization is not required. Read back the saved artifacts and report their paths and the limits of what was checked.",
      "expected_output": "The agent inspects the actual generator and pipeline interfaces, repairs the draft, and produces a coherent future 12-run comparison: two targets times two beam widths times three diffusion step counts. It selects beam-search, removes the nsteps=400 CLI override that collapses the nsteps axis, and prevents the two target invocations from overwriting one another. It may use a task-name sweep axis or distinct run names per target. Both generated config filenames and inference/evaluation result paths must match the corrected invocation. It writes the corrected sweeper and script, a complete run inventory, and a review with the assumed 4 GPU-hour budget. It leaves the input snapshots unchanged, performs no launch, and distinguishes source-level review from materialization or inference validation.",
      "assertions": [
        "The effective corrected configuration selects generation.search.algorithm=beam-search, generation.search.beam_search.beam_width=[4,8], generation.args.nsteps=[100,200,400], and both 02_PDL1 and 22_DerF21. It preserves unrelated base pipeline settings. Algorithm selection may be pinned in the sweeper or by a supported fixed override.",
        "The corrected launch.sh passes the corrected sweeper to script_utils/generate_inference_configs.py using the supported interface. No fixed override collapses a requested axis. Its design commands consume the generated inference configs, rather than passing --sweeper to complexa design.",
        "The corrected script retains all twelve target/parameter jobs with distinct generated config filenames and inference/evaluation result paths. Accept a generation.task_name sweep axis or separate target invocations with distinct run names; merely changing config directories while retaining colliding result paths is insufficient.",
        "runs.csv contains exactly the twelve requested target/beam-width/step-count combinations, once each, and records future config and result paths consistent with the corrected script and the supplied generator naming behavior. Materializing files from the partial snapshot is not required.",
        "review.md explains the inactive beam-width axis under the default best-of-n algorithm, the nsteps override precedence, and the reused run-name collision. It budgets 12 runs times 20 minutes = 240 GPU-minutes = 4 GPU-hours and labels the per-run duration as a planning assumption.",
        "The agent saves and reads back sweeper.yaml, launch.sh, runs.csv, and review.md under /workspace/output. It identifies these as prepared artifacts, reports remaining launch prerequisites, and does not claim successful config materialization or inference without evidence.",
        "The agent keeps the input snapshots unchanged and respects the offline scope: no launch script execution, inference, job submission, installation, cloning, downloads, secret disclosure, or fabricated scientific results."
      ],
      "expected_skill": "complexa-sweep",
      "expected_script": null,
      "files": [
        "files/project",
        "files/launch_review"
      ]
    },
    {
      "id": "complexa-sweep-006",
      "prompt": "Review the Complexa sweep summary in /workspace/input/summary_review.csv and recommend the configuration with the highest fraction of designs passing the filters; break ties using lower mean interface PAE. Account for every input configuration and explain any data issues that limit the comparison.\n\nSave a ranked sweep_summary.csv and a data_issues.md report under /workspace/output. Preserve the supplied counts and measurements so the recommendation can be audited, and keep unavailable measurements distinguishable from measured values. Read back the saved results and report the best configuration and any limitations.\n\nThe CSV is a synthetic test fixture, not measured biological performance. This is offline analysis only: do not run inference or submit jobs, install software, clone repositories, or download data or weights.",
      "expected_output": "The agent computes success fractions from valid counts rather than comparing passed counts or treating config-summary rows as individual designs. The five rankable rows, in order, are review_1_8 (20/25=0.8, mean_i_pae=5), review_1_4 (8/10=0.8, mean_i_pae=6), review_15_8 (14/20=0.7, mean_i_pae unavailable), review_05_8 (30/50=0.6, mean_i_pae=7), and review_05_2 (3/10=0.3, mean_i_pae=9). It separately identifies review_05_4 as unrankable with zero designs, review_1_2 as invalid with a negative denominator, review_15_2 as invalid because passed exceeds designs, and review_15_4 as unrankable because the passed count is missing. The missing PAE for review_15_8 remains unavailable and does not invalidate its known success fraction. All nine configs are accounted for across the ranked CSV and issues report; retaining invalid rows in the CSV is acceptable if explicitly unranked. The agent reports review_1_8 as the best synthetic fixture row, preserves the source data, and makes no inference or biological-quality claims.",
      "assertions": [
        "For valid counts, success_rate is n_passed/n_designs using the supplied sample sizes. The five rankable rows have rates 0.8, 0.8, 0.7, 0.6, and 0.3; the agent does not rank by raw passed count or derive rates from the number of summary rows.",
        "The rankable order is review_1_8, review_1_4, review_15_8, review_05_8, review_05_2. The selected best row is review_1_8 at 20/25=0.8; its mean interface PAE of 5 breaks the tie with review_1_4 at 8/10=0.8 and PAE 6.",
        "The agent separately flags review_05_4 (zero designs), review_1_2 (negative n_designs), review_15_2 (n_passed greater than n_designs), and review_15_4 (missing n_passed). These rows do not receive ranks or fabricated numeric success rates. They may be listed in data_issues.md or retained as explicitly unranked CSV rows.",
        "review_15_8 retains its valid 14/20=0.7 success rate and its place in the ranking despite missing mean_i_pae. The agent flags the missing measurement and does not substitute zero or an inferred PAE. Missing data and measured zeros remain distinguishable.",
        "Every one of the nine input config IDs is accounted for across sweep_summary.csv and data_issues.md. The ranked CSV preserves the input counts and available measurements for its rows; invalid counts are reported rather than silently repaired or dropped without explanation.",
        "The agent saves and reads back sweep_summary.csv and data_issues.md under /workspace/output, reports the best row and relevant limitations, explicitly identifies the analysis as synthetic fixture analysis, and leaves the input CSV unchanged.",
        "The agent stays within offline analysis: no inference, job submission, installation, cloning, downloads, secret disclosure, or fabricated scientific results."
      ],
      "expected_skill": "complexa-sweep",
      "expected_script": null,
      "files": [
        "files/summary_review.csv"
      ]
    }
  ]
}