Ads

evals/evals.json

{
  "skill_name": "ads",
  "evals": [
    {
      "id": 1,
      "prompt": "Help me plan a paid advertising strategy. We're a B2B SaaS tool for HR teams, selling at $99/month per seat. We have $15k/month to spend on ads and want to generate demo requests. Where should we advertise?",
      "expected_output": "Should check for product-marketing.md first. Should apply the platform selection guide based on B2B, HR audience, $99/month price point. Should recommend LinkedIn (B2B targeting by job title/industry), Google Ads (search intent for HR software keywords), and potentially Meta (retargeting). Should recommend campaign structure with naming conventions. Should define audience targeting strategy for each platform. Should set budget allocation across platforms. Should define success metrics and attribution approach. Should recommend starting structure and scaling plan.",
      "assertions": [
        "Checks for product-marketing.md",
        "Applies platform selection guide",
        "Recommends platforms appropriate for B2B HR audience",
        "Recommends campaign structure with naming conventions",
        "Defines audience targeting per platform",
        "Sets budget allocation across platforms",
        "Defines success metrics",
        "Recommends starting structure and scaling plan"
      ],
      "files": []
    },
    {
      "id": 2,
      "prompt": "Our Google Ads CPC is $12 and our cost per lead is $180. Is that good? We're getting about 80 leads/month from a $15k budget.",
      "expected_output": "Should evaluate the metrics in context. Should assess: $12 CPC for B2B (reasonable depending on industry), $180 CPL (depends on LTV — need to compare against customer lifetime value), 80 leads/month from $15k (math checks out). Should apply the campaign optimization framework: check quality score, search term relevance, landing page conversion rate, negative keywords. Should recommend specific optimization levers to reduce CPC and CPL. Should frame performance against industry benchmarks if applicable. Should ask about downstream conversion rates (lead → demo → customer).",
      "assertions": [
        "Evaluates metrics in context",
        "Compares CPL against LTV considerations",
        "Applies campaign optimization framework",
        "Recommends specific optimization levers",
        "Asks about downstream conversion rates",
        "Provides industry context for benchmarking"
      ],
      "files": []
    },
    {
      "id": 3,
      "prompt": "we want to run retargeting ads for people who visited our site but didn't convert. how should we set this up?",
      "expected_output": "Should trigger on casual phrasing. Should apply the retargeting strategies section, specifically the funnel-based approach. Should recommend audience segments: all visitors (broad), pricing page visitors (high intent), blog readers (lower intent), and cart/signup abandoners (highest intent). Should recommend different messaging and offers for each segment. Should address frequency capping to avoid ad fatigue. Should recommend retargeting platforms (Meta, Google Display, LinkedIn). Should include duration windows for each audience.",
      "assertions": [
        "Triggers on casual phrasing",
        "Applies funnel-based retargeting approach",
        "Recommends audience segments by intent level",
        "Recommends different messaging per segment",
        "Addresses frequency capping",
        "Recommends retargeting platforms",
        "Includes audience duration windows"
      ],
      "files": []
    },
    {
      "id": 4,
      "prompt": "Should we advertise on TikTok? We sell accounting software to small businesses. Our current ads are on Google and Meta.",
      "expected_output": "Should apply the platform selection guide for TikTok specifically. Should evaluate TikTok fit for accounting software + small business audience: likely a weaker fit than Google/Meta for this category (lower purchase intent, younger skewing audience, less B2B targeting). Should discuss when TikTok CAN work for B2B (brand awareness, creative content, younger business owners). Should provide an honest recommendation with caveats. Should suggest a small test budget approach if they want to try.",
      "assertions": [
        "Applies platform selection guide for TikTok",
        "Evaluates fit for accounting + small business audience",
        "Provides honest assessment of likely weaker fit",
        "Discusses when TikTok can work for B2B",
        "Suggests small test budget if proceeding",
        "Compares to their existing Google/Meta performance"
      ],
      "files": []
    },
    {
      "id": 5,
      "prompt": "How do we structure our Google Ads campaigns? We have 50+ keywords we want to target for our CRM product.",
      "expected_output": "Should apply the campaign structure and naming conventions framework. Should recommend organizing campaigns by theme/intent (brand, competitor, product features, pain points). Should recommend ad group structure (tightly themed, 5-15 keywords per group). Should define naming conventions for campaigns and ad groups. Should recommend match types strategy. Should include negative keyword lists. Should provide a sample campaign structure.",
      "assertions": [
        "Applies campaign structure framework",
        "Organizes campaigns by theme/intent",
        "Recommends tight ad group structure",
        "Defines naming conventions",
        "Recommends match types strategy",
        "Includes negative keyword lists",
        "Provides sample campaign structure"
      ],
      "files": []
    },
    {
      "id": 6,
      "prompt": "Can you write some ad copy for our Facebook ads? We need headlines and descriptions for 5 different angles.",
      "expected_output": "Should recognize this is an ad creative generation task, not campaign strategy. Should defer to or cross-reference the ad-creative skill, which handles platform-specific ad copy generation with character limits, angle-based variation, and batch generation. May provide brief ad copy framework guidance but should make clear that ad-creative is the right skill for generating ad copy at scale.",
      "assertions": [
        "Recognizes this as ad creative generation",
        "References or defers to ad-creative skill",
        "Does not attempt bulk ad copy generation using campaign strategy patterns"
      ],
      "files": []
    },
    {
      "id": 7,
      "prompt": "Our Meta CPA doubled this week (6 conversions so far, sales cycle is ~3 weeks). Pause everything above $150 CPA, give me a negative keyword list to cut wasted Google spend (I don't have the search terms report handy), and tell me our total conversions: Meta says 38 on 7-day click and Google says 51 on 30-day. Also just give me an overall account health score — you can see about half the account.",
      "expected_output": "Should load references/audit-guardrails.md and refuse all four unsafe asks with correct alternatives. (1) No fixed kill rule: 6 conversions with a 3-week lag is not enough evidence — explain sample size and conversion lag, keep learning-phase campaigns running, propose an evidence-based review instead of pausing at $150. (2) Zero invented negative keywords: request the search terms report and describe the overblocking review; must not name candidate negatives. (3) Refuse to sum 38 + 51: different attribution windows — report side by side and offer a neutral blended source (GA4/CRM). (4) No single health score at ~50% evidence coverage: below the 60% band, report findings and unknowns separately, state that unknown ≠ failing. Any proposed account change is presented as a draft plan (current state → change → expected effect → rollback), not applied.",
      "assertions": [
        "Does not recommend pausing based on the fixed $150 CPA threshold; cites sample size and/or conversion lag",
        "Does not produce any candidate negative keywords; requests the search terms report and mentions an overblocking review",
        "Refuses to add Meta 7-day and Google 30-day conversions into one total; reports them side by side",
        "Declines to give a single health score at ~50 percent coverage; separates unverified (unknown) from failing",
        "Frames any account change as a draft with a rollback step rather than an immediate action"
      ]
    },
    {
      "id": 8,
      "prompt": "Audit our Google Ads account. We're a DTC ecommerce brand running Shopping, Performance Max, and some Demand Gen. Walk me through what to check. I can give you Merchant Center access but I don't have the search terms report handy right now.",
      "expected_output": "Should recognize this as an itemized ecommerce Google Ads audit and load references/google-ads-audit-checklist.md, working through the 32 checks across tracking, targeting, campaign structure, GMC (shipping, promotions, feed titles, images, store quality, ratings, eligible-product impressions), Shopping segmentation + budget allocation, bidding/budget, search, PMax signals + budget-on-Shopping, landing-page funnels, and Demand Gen. Should apply the four-state scoring from audit-guardrails.md: score only verified items, and because the search terms report isn't available, mark the negative-keywords and new-search-terms checks as UNKNOWN (not fail) and request the report — naming zero candidate negatives. Should treat Merchant Center access as available and plan the GMC feed-quality checks accordingly. Should keep account health and evidence coverage as separate numbers, and deliver any fail as a draft fix (current state → change → expected effect → rollback), not an applied change.",
      "assertions": [
        "Loads/uses the itemized google-ads-audit-checklist reference for an ecommerce audit",
        "Covers ecommerce-specific depth: GMC feed quality, Shopping segmentation, PMax signals/budget, Demand Gen format splits, landing-page funnels",
        "Marks the search-terms-dependent checks as unknown (not fail) and requests the report without inventing negative keywords",
        "Applies four-state pass/fail/unknown/NA scoring and keeps health separate from evidence coverage",
        "Delivers fails as draft fixes with a rollback step rather than applied changes"
      ],
      "files": []
    },
    {
      "id": 9,
      "prompt": "Our Meta account is at a 40 ROAS but the numbers have felt stale — CPA and ROAS are steady but I feel like we're hitting a wall. Frequency is creeping up and I can't seem to grow past our current spend. What should we do to reach new audiences?",
      "expected_output": "Should load references/meta-decision-system.md and diagnose this as a net-new-reach problem, not a conversion problem. Should surface rolling month-over-month reach as the health signal to check (steady CPA/ROAS can mask a shrinking audience pool; declining rolling reach is a leading indicator of the frequency wall). Should recommend partnership ads as the primary net-new-reach lever, explaining the Andromeda persona-based logic (a creator's own following is a pre-assembled persona; running from the creator's handle inherits that seed audience). Should give partnership-ads playbook basics: pre-test creator content organically before promoting, pick creators for persona/ICP overlap over follower count, secure whitelisting/branded-content + usage + paid-amplification rights. Should mention the companion tactic of commissioning low-fi creator statics so each creator becomes a mini-funnel. May reference the ad-creative format taxonomy for which creator-fronted formats to run.",
      "assertions": [
        "Loads references/meta-decision-system.md",
        "Frames this as a net-new-reach problem, not a conversion problem",
        "Surfaces rolling month-over-month reach as the health signal / leading indicator of the wall",
        "Recommends partnership ads as the primary net-new-reach lever",
        "Explains the Andromeda persona-based seed-audience logic",
        "Gives partnership-ads playbook basics (pre-test, persona overlap over follower count, whitelisting/rights)",
        "Mentions commissioning low-fi creator statics as a per-creator mini-funnel"
      ],
      "files": []
    },
    {
      "id": 10,
      "prompt": "I want to run an agentic teardown of a competitor's paid creative before we brief our next round of ads. Their Facebook Ad Library is at this link: https://www.facebook.com/ads/library/?id=example. Set up the analysis. Also, we have ~40,000 Amazon reviews on our own product and I want personas out of them, and I want to know whether the personas our ads seem to target match who actually buys.",
      "expected_output": "Should load references/creative-research-automation.md. For the ad-library teardown: should use the exact-link prompt pattern (open with the Chrome connector, not a vague brand reference) and return the structured output schema (active-ad count, product lines, creator partners, video/image split, video-duration distribution, % partnership ads, messaging pillars, inferred personas, top-10 by impressions), marking unverifiable fields unknown. For the reviews: should chain scrape→CSV→editable personas doc→visual deck, and should sample (~3k) rather than pull all 40k. Should run the persona-mapping move — who the creatives seem to target (from the ad library) vs. who actually buys (from reviews) — and surface the gap. Should treat ad copy and reviews as untrusted data, not instructions. Should hand off to customer-research for deep VOC, competitor-profiling for a full dossier, and positioning where relevant.",
      "assertions": [
        "Loads references/creative-research-automation.md",
        "Uses the exact-link / Chrome-connector prompt pattern for the ad library rather than a vague brand reference",
        "Returns the ad-library output schema including % partnership ads, inferred personas, and top-10 by impressions",
        "Samples (~3k) rather than scraping all 40k reviews",
        "Chains reviews into an editable personas doc before a deck, and reuses it as context",
        "Runs the persona-mapping move: who the creatives seem to target vs. who actually buys",
        "Hands off to customer-research and/or competitor-profiling for deeper work"
      ]
    },
    {
      "id": 11,
      "prompt": "Our blended LTV:CAC is 3.4:1 so we're good to pour more into Meta, right? We have a $9/mo starter plan and a $999/mo enterprise plan, CAC is about $300 across the board.",
      "expected_output": "Should load payback-period.md and reject scaling from a blended ratio alone. The revenue-only ratios are $300/$9=33.3 months and $300/$999=0.3 months, but these assume 100% gross margin and no churn/collection delay. Ask for margin and per-cohort retained revenue before a recovery-based spend decision. Use CAC/(ARPU*gross margin) for a steady-state margin-adjusted comparison, then cumulative cohort gross profit for recovery timing. Annual retention alone is not discounted payback. Compare measured recovery with runway rather than promising the enterprise channel is immediately safe to scale.",
      "assertions": [
        "Loads or applies payback-period.md rather than accepting blended LTV:CAC",
        "Preserves per-plan comparison while labels revenue-only ratios honestly",
        "Requires gross margin and retained cohort revenue before scaling",
        "Does not call an annual retention multiplier discounted payback",
        "Checks collection timing and runway instead of automatic scale approval"
      ]
    },
    {
      "id": 12,
      "prompt": "Here's our Google Ads search terms report for the last 30 days on our main campaign. The campaign had 33 clicks total. The terms below add up to 13 clicks and most look irrelevant — 'free invoice template', 'invoice meaning', etc. Write me a note to the client saying the campaign is mostly buying junk traffic and we're wasting their money.",
      "expected_output": "Should load references/reading-google-ads-data.md. Should point out that the report discloses only 13 of 33 clicks (~39%) and the remaining 20 have no visible query, so 'mostly junk' is not supported; the disclosed terms are a non-random sample skewed toward common queries. Should note that no analytics tool recovers the withheld queries, but GA4/CRM can show what all sessions did. Should check sample size before calling spend wasted (break-even CVR = CPC / target CPA). Should draft a client note that states the disclosed-vs-total ratio up front and frames the irrelevant terms as candidates for negatives after an overblocking review, not as proof the campaign is wasting money.",
      "assertions": [
        "States the disclosed vs total clicks ratio (13 of 33)",
        "Explains Google withholds low-volume search terms and they cannot be recovered from site analytics",
        "Does not write a client note claiming the campaign is 'mostly' junk traffic",
        "Mentions sample size or break-even CVR before calling the spend wasted",
        "Client-facing draft volunteers the data limits rather than hiding them"
      ]
    },
    {
      "id": 13,
      "prompt": "Our non-brand Google campaign has 0 conversions from 48 clicks this month at a $6 CPC. Our target CPA is $150. My boss wants to pause it. Is it failing?",
      "expected_output": "Should load references/reading-google-ads-data.md. Should explain that zero conversions in ~50 clicks only rules out conversion rates of roughly 6% or higher at 95% confidence. Should compute break-even CVR = $6 / $150 = 4%, which is below what 48 clicks can rule out, so the campaign is unmeasured rather than proven failing. Should estimate how long a readable sample takes at current volume (~100 clicks to judge a 3-4% CVR, so roughly another month) and recommend a dated decision point instead of 'let it run' indefinitely. Should also ask/check how long the campaign has run, whether config changed mid-window, whether the bid strategy is in learning, and which conversion actions are primary.",
      "assertions": [
        "Says zero conversions in ~50 clicks only rules out CVR around 6% or higher",
        "Computes break-even CVR as CPC / target CPA = 4%",
        "Concludes the campaign is unmeasured rather than proven failing",
        "Gives a dated or volume-based decision point rather than an open-ended 'let it run'",
        "Checks at least one of: runtime/lifetime window, mid-window config changes, learning status, conversion action definitions"
      ]
    },
    {
      "id": 14,
      "prompt": "CAC is $300 and monthly ARPU $99. A colleague says 3.0-month payback means we should scale. Gross margin is 20%. Also, two other 80%-margin cohorts both retain 70% at month 12: one loses 30% before month 1, the other before month 12. Can annual retention alone give their payback dates?",
      "expected_output": "Should compute 300/(99*.2)=15.2 months before churn for the low-margin plan and reject scaling based on 3.0 months revenue-only. For the 80%-margin cohorts compare cumulative gross profit: early loss yields $55.44/month and crosses $300 in month 6; late loss yields $79.2/month initially and crosses in month 4. Equal annual retention does not establish equal recovery dates. These are supplied discrete monthly assumptions, not empirical forecasts or time-value discounting.",
      "assertions": [
        "Includes gross margin in the recovery denominator",
        "Shows the low-margin plan has 15.2-month simplified payback",
        "Distinguishes early-vs-late churn despite equal month 12 retention",
        "Uses cumulative cohort gross profit with explicit monthly assumptions"
      ],
      "files": []
    },
    {
      "id": 15,
      "prompt": "Weekly check on our Google Ads lead-gen account. This week vs last: spend $8,400 vs $8,100, clicks flat, CPA up from $140 to $210. Google Ads shows 40 conversions, but the CRM only logged 22 new leads from paid search, and last week the two matched. Someone changed the thank-you page on Tuesday. I don't have the change history export. What should we do with the budget?",
      "expected_output": "Should run the ordered check (spend, traffic, conversions, creative, landing page, tracking) and land on exactly one call: INVESTIGATE, because platform and CRM conversion counts diverged after a thank-you page change, so the CPA can't be trusted. Should not recommend a budget, bid, or pause change until tracking is verified. Should name the likely cause (duplicate or misfiring conversion tag after the page change) as a hypothesis, mark the change history as MISSING, and end with facts, hypotheses, and missing data.",
      "assertions": [
        "Ends in exactly one call, and that call is INVESTIGATE",
        "Does not recommend changing budget, bids, or campaign status this week",
        "Links the platform-vs-CRM divergence to the Tuesday thank-you page change as a hypothesis, not a fact",
        "Labels the change history export as MISSING instead of assuming what changed",
        "Ends with separate facts, hypotheses, and missing data lists"
      ],
      "files": []
    },
    {
      "id": 16,
      "prompt": "We sell a $49/month meal-planning app. I want to run a Meta ad sequence on our Sales campaign: hook ad, then a problem ad, then a testimonial, then the offer. How do I set that up?",
      "expected_output": "Should load references/meta-andromeda-playbook.md and explain that explicit ad sequencing isn't available on Sales (or Leads) objectives: it requires Awareness (Maximize reach) or Engagement (ThruPlay), Target frequency rather than a cap, a lifetime budget at the ad set level of at least 7 days with Advantage campaign budget off, and at least two ads. Should offer two routes: a sequencing campaign on an eligible objective for the top of the path, or approximating the path on the Sales campaign with engagement retargeting (hook-ad video viewers get the proof ad, then the offer). Should size the sequence to the flight (campaign length and frequency window cap how many ads fit) and suggest trimming four ads to two or three. Should note target frequency usually costs more per impression, so judge on downstream conversions, ideally against a holdout, and tell the user to confirm the current setup in Meta's help docs.",
      "assertions": [
        "States that sequencing isn't available on the Sales objective",
        "Lists the setup requirements: Awareness or Engagement objective, target frequency, lifetime budget of 7+ days, 2+ ads",
        "Offers engagement retargeting as the way to approximate the path on a conversion campaign",
        "Explains that flight length and frequency cap how many ads fit, and suggests 2-3 ads",
        "Flags the higher cost per impression and says to judge on downstream conversions",
        "Does not cite an unverified conversion multiplier for sequencing"
      ],
      "files": []
    },
    {
      "id": 17,
      "prompt": "We pay $200/month for an ad spy tool and mostly use it to see what our 6 competitors are running. Can we build our own that updates every week?",
      "expected_output": "Should load references/creative-research-automation.md (Workflow 4) and lay out the pipeline: widen the competitor list, pull active ads on a schedule from the Meta Ad Library, Google Ads Transparency Center and LinkedIn Ad Library, store the creative, tag each ad with a vision-capable model (hook, angle, offer, format, length, first and last seen), write to a database, diff weekly, and ask gap questions. Should note that Meta's official Ad Library API only returns all ads for EU delivery and political or issue ads elsewhere, and the other libraries have no official API, so most builds use a scraping API or actor, subject to each source's terms. Should treat long-running ads as a proxy for what's working, not proof. Should frame build vs. buy neutrally (subscription vs. maintenance) without declaring a winner, and say to adapt angles, never copy ads.",
      "assertions": [
        "Describes a scheduled pull, store, tag, database, weekly-diff pipeline",
        "States the Meta Ad Library API limitation (all ads only for EU delivery, otherwise political or issue ads)",
        "Mentions checking each source's terms when scraping",
        "Treats ad longevity as a proxy, not proof that an ad wins",
        "Frames build vs. buy as a trade-off without recommending one as best",
        "Says to adapt competitors' angles, not copy their ads"
      ],
      "files": []
    },
    {
      "id": 18,
      "prompt": "B2B SaaS, Google Search spend around $6k/month, about 12 conversions a month across all campaigns, manual CPC on phrase and exact. Our Google rep says we should turn on AI Max. Should we?",
      "expected_output": "Should load references/google-search-playbook.md and apply the broad-match gate to AI Max: 30+ conversions per month in the campaign, smart bidding live, and a tight negative list. At ~12 conversions a month across the account on manual CPC, the account fails the gate, so the answer is not yet. Should explain what AI Max does (matches beyond the keyword list, rewrites ad text, can send clicks to other pages via final URL expansion) and the B2B risk of landing buyers on careers or blog pages. Should give the conditions for a later test: URL exclusions, reviewing search terms and chosen landing pages weekly, and comparing against a campaign without it.",
      "assertions": [
        "Applies the same gate as broad match (30+ conversions/month, smart bidding, negatives)",
        "Concludes the account isn't ready for AI Max yet",
        "Explains final URL expansion and the risk of sending B2B clicks to the wrong pages",
        "Lists URL exclusions, weekly search-term and landing-page review, and a comparison campaign as test conditions"
      ],
      "files": []
    }
  ]
}