Oleksandr Moccogni AI course Join the waitlist

AI Marketing Agents: How to Know They Did Not Miss Something

Published Updated 20 min read
In short

AI marketing agents fail quietly. My design: a script collects everything, the AI reads all of it, and a check that does not depend on the AI proves the result.

AI Marketing Agents: How to Know They Did Not Miss Something

An AI marketing agent rarely fails with an error message. It fails by reporting that everything is fine.

AI marketing agents are programs in which an AI model chooses its own steps and its own tools to reach a goal, such as watching ad accounts or reading comments. That freedom is the selling point, and it is also where mistakes hide.

The question I ask of any agent is how I would notice that it was wrong.

I build these tools for my own work and my consulting work, and the answer I settled on is a design with three moves. A script collects everything, the AI reads all of it, and a check that does not depend on the AI proves the result.

This article covers how agents fail quietly, the three moves, four marketing jobs built on them, and how I test the tests.

Key Takeaways

  • Judge an AI marketing agent by how you notice its mistakes, not by how much it can do.
  • A fixed rule only finds what you predicted. Let a script collect everything and the AI read all of it. Use thresholds to sort, never to decide whether to look.
  • An AI that checks an AI can say “all good” when it is not. Add a proof that does not depend on the AI.
  • Break your own checks on purpose. One check I tested had tests that missed about a quarter of the faults I planted in it.
  • Measure the cost in percent of your plan, not in dollars. An agent reads and writes far more than a chat does.
  • Often you need a workflow, not an agent. Use the simplest thing that works.

What AI Marketing Agents Are, and What Most Vendors Sell Instead

Anthropic, the company behind Claude, draws a line worth borrowing. In its guide Building effective agents it writes: “Workflows are systems where LLMs and tools are orchestrated through predefined code paths.” Then: “Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.”

In plain words, a workflow is a recipe written in advance: if X arrives, do Y. An agent decides the steps itself. A lot of what the market sells as an agent is a recipe, and a recipe is fine when a job must run the same way every time, like my newsletter automation pipeline. By this definition much of what I run is a workflow with one AI reading step in the middle, which is the right choice when a recipe alone cannot see what you did not predict.

The same guide gives advice that is easy to skip: “we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all.” It also warns that “The autonomous nature of agents means higher costs, and the potential for compounding errors.” Compounding is the part that matters here, because a mistake in step two is read as a fact in step three.

The guide dates from December 2024 and covers agents in general, not marketing work, so I take its concepts and none of its tools. The map of which marketing tasks to automate first is in my article on AI marketing automation, so here I stay with the agent itself.

Three Ways an AI Marketing Agent Fails Without Telling You

None of these failures produces an error message. Each one produces a calm, confident report.

A clay caricature in a suit points at the viewer and aims a flashlight into the open door beside two closed doors

The Rule Finds Only What You Predicted

On October 24 I asked for four daily agents, covering Google Ads, conversion rates, news and social posts. The first design the AI drafted put thresholds in the script, and only a flagged signal woke up the AI to read.

My objection was short: a script cannot tell that something is wrong, unless it extracts everything and then lets the AI read it. A threshold finds only what you had already predicted. In the final design, thresholds and comparisons sort the list and trigger immediate alerts, but they are never the condition for the AI to look.

Here is an invented example. Say a rule alerts you when ad spend rises more than 30 percent. A site that stops recording contact requests while spend stays flat triggers nothing, and the report says all is well.

To test the design, I wrote a test for the news agent, which reads security and platform news against a register of the sites and components I look after. The test takes one item off that register, PHP (a programming language many websites run on), and checks that two news items about a PHP vulnerability are still flagged from their content alone.

On a stand-in server, one that only pretends to be the real one, the mechanism holds. With PHP in the register, the two items are marked “register: yes”, meaning they concern something I look after, and two immediate alerts go out. Without it, the items stay in the daily digest, the numbered list of everything the agent reads, marked “register: no”, no immediate alert fires, and the advice comes out labeled “unverified” at low severity. Removing an item from the register changes the labels, never whether the AI reads the news. The run with the real model is the next step.

The Checker Says “All Good”

On October 11 an AI assistant reviewed the first piece of a short film generated by another AI. It called the result a huge leap forward and said the hands looked fine. It had not counted the arms: a third forearm filled the frame for 4.5 seconds. The full story is in my article on AI slop and the human review bottleneck.

The lesson is not that an AI checker is useless. Anthropic’s own advice on testing agents, in its article Demystifying evals for AI agents, is: “We recommend choosing deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously for additional validation.” An AI grader has to be calibrated against people, and it works better when it is allowed to say “Unknown” when it does not have enough information.

The same evals article shows that a check can be the broken part. A model scored 42 percent on a public test until an Anthropic researcher found “rigid grading that penalized ‘96.12’ when expecting ‘96.124991…’”, among other problems. After the grading bugs were fixed and the setup was made less constraining, the same model scored 95 percent.

Where the number comes from: a public capability test for AI models. What it means: much of the “failure” sat in the checks, not in the model. What changes: before you trust a score, read how it was given.

I apply the same reading to AI cyberattack numbers, with six questions in what the tests show about AI cyberattacks.

Anthropic’s multi-agent research write-up adds a marketing-shaped lesson: “Human evaluation catches what automation misses.” Human testers noticed that the early agents “consistently chose SEO-optimized content farms over authoritative but less highly-ranked sources like academic PDFs or personal blogs.” Any marketing agent that researches the web can fail the same way: the answer sounds right and the sources are the wrong ones.

The Reading Is Partial

The third failure is quieter. An agent reads only what a rule flagged, or only the summary a tool produced, and the answer sits in the part nobody looked at.

One client’s ads run only Monday to Friday, so 28 days hold 20 days when they can be live, not 28. In the previous period they were live on 20 of those 20 days, in the latest on 7 of 20, because the ads had been paused for a platform check. A summary that only says traffic fell reads like a collapse. Counting live days from the hourly schedule, not from the calendar, showed where the drop came from and put the right base under the comparison.

On a site that collects contact requests, 242 of 525 sessions, 46 percent, came from countries where data centers sit. Nearly all of those visits had no referring site, viewed one page and stayed 0.4 seconds on average, which is how a program behaves, not a person. The number was in the table the whole time, and no threshold looked for it. How I turned those two cases into a report is in my article on the marketing report.

The Three-Move Design: Script, Reading, Proof

Infographic "Three moves": a script collects, the AI reads all of it, then a proof that does not depend on the AI, with a clay caricature

Move one: a script collects everything. No AI and no cost in tokens, the small pieces of text a model reads and writes. The script downloads everything the agent will look at and removes duplicates.

Then it calculates the comparisons (with yesterday and with the 7-day and 28-day averages), labels every entry, and writes one numbered list. I call it the digest.

Move two: the AI reads the whole digest. On each day with something new, the model reads the entire list. It has no tools, no web access and no connections: it reads and returns structured proposals.

Text from other people, such as feeds and web pages, is treated as untrusted data. Severity, links and codes are computed by code, and the AI picks entries by number, so it cannot invent a link. I explain the risk behind that rule in AI agent security and prompt injection.

Move three: a proof that does not depend on the AI. Anthropic’s guide says “it’s crucial for the agents to gain “ground truth” from the environment at each step (such as tool call results or code execution) to assess its progress.” Anthropic uses the term for results the agent gets from its tools at each step, and adds that an agent can pause for human feedback at checkpoints. I extend the idea to checks outside the loop. In marketing, ground truth is a check that can fail and comes from somewhere else: the ad schedule, a second data source, a test that forces the system to behave as it does in production, a human who approves the spend.

Why this order? The script does the work that needs no intelligence, cheaply and the same way every time. The AI is called once, to read material that is already in order.

The same split works inside Claude: a skill can hold written steps plus small scripts that Claude runs when a task matches. What deserves a skill, and why one with code needs a check first, is in Claude Skills and their risks.

Four Marketing Jobs, Three Moves Each

Here are four jobs built on this design, with the status of each. The client report is running. The Google Ads reading runs inside it. The tracking check is a tested procedure, not yet a daily agent. The comment reader has its scoring tool built, and the daily agent is still a plan.

The script collects, for every campaign, spend, clicks, conversions, cost per contact, impression share (how often the ads showed when they could have), the search terms people typed, the days the ads were live according to the hourly schedule, and the comparison with yesterday and with the 7-day and 28-day averages. The AI reads the whole list and returns advice with the number, the window and the comparison, never actions.

The proof: the script counts live days from the schedule, conversions are checked against a second source (contacts in the site’s analytics, orders in the shop), and nothing is spent or changed without my explicit go-ahead, even a pause. The read-only reading already runs inside my weekly client report. The daily agent is planned in two steps: advice first, then a separate approved action from a closed list, such as adding a negative keyword (a word that stops the ads from showing for a search) or pausing one keyword or ad, and never budgets or bids.

Tracking Checks

The script counts events and distinct people per event, by country and source, plus duration, pages and engagement per session. It also sends a test submission with a test parameter. The AI reads the whole table, country by source by engagement, and the repeated events.

The proof is a browser test in which every call to the site’s server gets a fake “ok” and every call to analytics is recorded, then blocked. That test reproduced a form event arriving twice, 50 milliseconds apart, on a client site. It explained why four “lead” events in the data were three form submissions from two people, all test contacts from the client’s own team.

An earlier version of the test had blocked the calls to the site’s server, so the second firing of the event never showed. I fixed the tag setup on October 11 and rechecked it with the same test. The consent side is in my checklist for tracking consent in Switzerland.

Client Reports

The script reads analytics, search and ads data in read-only mode, plus order totals from the shop with no personal data and a set of site checks, such as server response, certificate, security headers, email authentication, sitemap and speed measured on real visitors. The AI reads the week’s log of what was done for the client and writes the draft of the narrative, which I read before anything goes out.

The proof: numbers and summary sentences come from formulas, so no model writes a figure. A missing value prints “n/a”, never zero, and comparisons use the same days.

A tone gate blocks the draft when real damage has not been told to the client, and only one message goes out per period. The report is running, and I approve each message before it goes to the client. Related reading: marketing operations.

Reading Comments

The script gathers the numbers at 24 hours and at 7 days (impressions, reactions, comments, shares, saves, profile visits) from an export or screenshots, the text of the comments and who wrote them, and a mechanical score of the draft. The AI reads the real comments, groups them into themes and recurring questions, and proposes content ideas, with a summary and a new rule every four posts.

The proof: a score of 12 checks for what code can verify, and my own judgment for the rest, such as one precise idea and the voice. I look at orders of magnitude, not 10 percent differences.

This agent never replies or comments on its own, and the first comment on a post is written by hand. The score and the importer exist, with 44 automated tests, and the daily agent is still a plan.

A clay caricature with open arms behind four instruments: a meter, a radar, a dial and a wave screen, all wired to one open notebook

How I Test the Checks Around an AI Marketing Agent

Green tests say little until you try to make them fail. My method is to invent faults, put them one at a time into a copy of the code, and count how many the tests catch.

I did this three times. On the check that validates the list of client sites and components, I invented 399 faults, of which 397 could be run. The tests caught 284 at once, about 72 percent, and 113 slipped through.

Of those 113, 12 were equivalent, meaning they did not change the behavior, and 101 were real holes that I closed with new checks. The real holes are about 25 percent of the 397.

Where the number comes from: invented faults in a copy of the check. What it means: about one fault in four that I planted in my most delicate check would have gone unnoticed by its own tests. What changes: “green” stops being proof and becomes a question, what did this test try to break?

On the script that sends a file to the server, I tried 84 faults. The tests caught 78 and 6 slipped through, with 2 real holes closed and 4 equivalents documented. After I found a hole in the test itself, I added 4 more faults and the tests caught all of them.

An earlier round of 56 invented faults, against a suite I considered good, ended with new checks and 56 of 56 caught. That story is in the AI slop article.

Poster "I broke my checks on purpose": 397 faults planted, 284 caught (about 72 percent), 113 slipped through, a clay caricature with a hammer

The second tool is a second AI, used as a reviewer that has not heard my conclusions. It gets the code, not my story about it, and it can only read. That is what makes it useful where the checker in the film story was not: it has no conclusion to agree with.

On the news agent, one such reviewer, started fresh with no memory of the work and set to think hard, gave 9 findings. I fixed 6, each with a test that failed first, because a fix without a failing test proves nothing. For the wider habit of verifying AI output, see my article on human in the loop AI verification.

What an AI Marketing Agent Costs: Percent of the Plan, Not Dollars

With a subscription, a cost in dollars is only theoretical. The number that matters is the share of the plan that gets used, in the 5-hour window and in the weekly window that the plan reports.

The case that made this real: on October 17 and 18, three long agent runs in one night, about 15 million tokens in all (11.1 plus 1.0 plus 3.1 million), used up the weekly limit halfway through the work. The agents compete with everything else for the same cap.

Anthropic gives the direction: “agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats.” Those multipliers come from Anthropic’s own research system, not from marketing tasks, so I read them as a warning and not as a measurement.

My own numbers, with what they are:

  • Calibration, an order of magnitude. From my October 18 transcripts, 1 percent of the week is about 0.64 million output tokens, the text the model writes. The 15 million above also counts everything the agents read, so the two figures cannot be divided into each other.
  • Estimates, not measurements. One reading costs about 8,000 to 25,000 tokens in and 1,000 to 2,000 out. The news agent plus the second AI that checks its proposals should use about 0.1 percent of the week at most. A daily agent that reads one site should use about 0.2 to 0.5 percent of the week.

All of it applies to my whole account, not to one run, and the percent of a plan only applies to someone who pays a subscription. What changes is the order of work: estimate first, launch in batches, and let the script do everything that does not need reading.

A clay caricature with a pencil behind his ear leans on a large dial showing a thin green wedge, linked to a notepad with three small bar charts

When You Do Not Need an AI Marketing Agent

  • The recipe is known. If you can write “if X, then Y”, a workflow costs less and fails in predictable ways. My client reports run on fixed formulas for exactly this reason.
  • A wrong step is expensive and hard to see. Anthropic warns of compounding errors. Where a mistake costs money or trust, I keep spend, messages to clients and comments behind a human go-ahead and I lean toward a workflow.
  • You cannot name the proof. If nobody can say which check would show the agent was wrong, I suggest not starting.

If you run a business and someone proposes an agent, I suggest asking one question: which check would tell us it was wrong? A good answer names a second source, a schedule or a test. A weak answer is “the AI reviews itself”. Failure of whole projects is a different topic, covered in why AI marketing implementations fail; this article is about the quieter failure of an agent that keeps running.

A Checklist Before an AI Marketing Agent Runs Alone

I suggest this order:

  1. Name the job in one sentence, and the decision it informs.
  2. Let a script collect everything. Keep thresholds as labels for sorting and alerts, never as the condition to look.
  3. Make the AI read the whole list, and allow it to answer “Unknown”.
  4. Name a proof that does not depend on the AI: a second source, a schedule, or a test that forces production behavior.
  5. Break the checks on purpose before you trust a green result.
  6. Estimate the cost in percent of the plan and launch in batches.
  7. Keep spend, sending and comments behind a human go-ahead.

Checklist "7 steps before you trust an agent" with seven ticked lines, from "Name the job in one sentence" to a human OK, and a clay caricature

When the script, the reading and the proof are all named, an agent can still be wrong, but it cannot hide it. The broader picture of where AI belongs in a marketing plan is in my article on AI marketing strategy and in using AI for marketing.

Frequently Asked Questions

What are AI marketing agents?

Software in which an AI model decides its own steps and tools to reach a marketing goal, in contrast to a workflow that follows a recipe written in advance. In practice, many products sold as agents are workflows.

What is agentic marketing?

Marketing in which part of the work is handed to an agent that chooses its own steps. I use Anthropic’s definition of an agent and do not invent a separate industry one.

What is the difference between an AI agent and marketing automation?

A workflow follows paths set in code; an agent directs its own process. The map of what to automate first is in my article on AI marketing automation.

What are examples of AI marketing agents?

Four jobs from my own work: monitoring Google Ads campaigns, checking tracking, preparing client reports, and reading comments on posts. Each has the same three moves: a script collects, the AI reads, a proof outside the AI checks.

What are the best AI agents for marketing?

The best one for you is the one whose mistakes you can see. Before buying any agent, ask what it collects, what it reads and which independent proof checks it.

How do I know an AI agent did not miss something?

Not with a threshold. Let a script collect everything, let the AI read all of it, then add a proof that does not depend on the AI and a second AI that reviews the code without having heard your conclusions.

Do I need an agent at all?

Often not. If the recipe is known, a workflow is cheaper and fails in predictable ways. Anthropic puts it this way: “This might mean not building agentic systems at all.”

How much of my AI plan does an agent use?

Measure it in percent of the plan, not dollars. Agents use more than a chat, so I estimate first and let the script do everything that needs no reading.

Tags: ai marketing agentsai agents for marketingagentic marketingai marketing agentmarketing automationai verificationai marketing

Keep reading

All writing →
Newsletter

Stay ahead.

Occasional notes on AI marketing, growth, and speaking. No spam.