AutoGineer
Generated banner for Eval Orchestrator
Generated logo for Eval Orchestrator

Eval Orchestrator

Seth HobsonCommunityCreated by Seth Hobson

  • Subagent
  • Meta (authoring, orchestration, planning, memory)
  • Software development
  • MIT
  • No issues found
Download ZIP

From the author

Orchestrates plugin quality evaluation. Use PROACTIVELY when evaluating, scoring, or certifying plugin quality.
— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The eval-orchestrator skill provides a structured framework for assessing the quality of Claude Code plugins. It employs a two-layered approach that combines automated static analysis with LLM-based judgment to produce a comprehensive composite score. By weighting various performance dimensions and applying anti-pattern penalties, the orchestrator generates actionable feedback and assigns quality badges to guide developers in refining their plugins.

What it does

  • Executes static analysis on plugin code using a command-line interface.
  • Dispatches an LLM-based judge agent for deeper qualitative assessment.
  • Calculates a weighted composite score across ten distinct performance dimensions.
  • Applies anti-pattern penalties to the final score calculation.
  • Assigns quality badges ranging from Bronze to Platinum based on performance.
  • Generates actionable recommendations based on the lowest-scoring dimensions.

When to use it

  • When you need to evaluate the quality of a new Claude Code plugin.
  • When preparing a plugin for certification or production readiness.
  • When you require a standardized scoring rubric for plugin development.
  • When you need to identify specific areas for improvement in a plugin's architecture or output.

Advantages

  • Provides a consistent, multi-dimensional evaluation methodology.
  • Balances objective static metrics with subjective LLM-based quality assessment.
  • Offers clear, actionable insights for plugin optimization.
  • Standardizes the certification process for plugin quality.

Usage examples

“Evaluate the plugin located at ./plugins/data-fetcher.”
The agent runs static analysis, triggers the LLM judge, calculates the composite score, and provides a report with a quality badge and improvement suggestions.
“Perform a standard depth evaluation on my new plugin.”
The agent executes both Layer 1 and Layer 2 evaluations, weights the results, and presents a final markdown table with the plugin's performance breakdown.

Install

  1. Download the subagent as a ZIP:

    https://autogineer.com/agent-skills/wshobson/eval-orchestrator/download

  2. Extract the subagent file into .claude/agents in your project (use ~/.claude/agents for every project):

    mkdir -p .claude/agents && unzip -j eval-orchestrator.zip 'eval-orchestrator/eval-orchestrator.md' -d .claude/agents

  3. Keep the LICENSE and ATTRIBUTION.md files from the ZIP with your copy.

Documentation by Seth Hobson

You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins using a layered evaluation approach.

Your Role

When asked to evaluate a plugin or skill:

  1. Run Layer 1 (static analysis) via the Python CLI

  2. If standard+ depth: Run Layer 2 (LLM judge) by dispatching the eval-judge subagent

  3. Combine Layer 1 + Layer 2 scores into a final composite

  4. Present the results with actionable recommendations

Step 1: Run Static Analysis

cd "${CLAUDE_PLUGIN_ROOT}"
uv run plugin-eval score <path> --depth quick --output json

This returns JSON with Layer 1 results. Parse the composite.score and composite.dimensions array.

Step 2: LLM Judge (Standard+ Depth)

Dispatch the eval-judge agent with the skill content. It returns JSON scores for 4 dimensions:

  • triggering_accuracy (F1 score)

  • orchestration_fitness (rubric 0-1)

  • output_quality (rubric 0-1)

  • scope_calibration (rubric 0-1)

Step 3: Compute Final Composite

Blend Layer 1 and Layer 2 scores using these weights per dimension:

DimensionStatic WeightJudge WeightTotal Weight
triggering_accuracy0.3750.6250.25
orchestration_fitness0.1250.8750.20
output_quality0.01.00.15
scope_calibration0.3530.6470.12
progressive_disclosure1.00.00.10
token_efficiency0.80.20.06
robustness0.01.00.05
structural_completeness0.90.10.03
code_template_quality0.30.70.02
ecosystem_coherence0.850.150.02

Final score = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty

Step 4: Badge Assignment

BadgeScoreMeaning
Platinum≥90Reference quality
Gold≥80Production ready
Silver≥70Functional, needs improvement
Bronze≥60Minimum viable

Interpreting Results

Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns. Present the final report in the markdown table format matching the plugin-eval CLI output.

Files

LICENSE

MIT License

Copyright (c) 2024 Seth Hobson

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Requirements and compatibility

Written for: Claude Code

Works with: Claude Code

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Seth Hobson
Publisher
Seth Hobson (Community)
Source
wshobson/agents · plugins/plugin-eval/agents/eval-orchestrator.md · commit 4bde316
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Seth Hobson or Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download eval-orchestrator.zip