Oleksandr Moccogni AI course Join the waitlist

AI Cyberattacks: Six Questions Before You Trust a Test

Published 24 min read
In short

How to read AI cyberattack tests: what CAISI, the UK AISI and Anthropic measured on GLM-5.3 or its predecessor, why numbers differ, and six questions to ask.

AI Cyberattacks: Six Questions Before You Trust a Test

Since July, three organizations have published cyber tests of Chinese open-weight AI models, GLM among them, and their numbers do not fit in one table. Headlines about what an AI can do to computer systems rarely say why.

AI cyberattacks are attacks on computer systems in which an AI model does part of the work, such as finding a flaw in software or writing the code that abuses it. Researchers measure this ability with benchmarks, and the results change with who runs the test and how.

I work in marketing and use AI models every working day, mostly local ones trained for my own needs. When a headline says an AI can attack systems, I read the primary reports, because models like the ones I use are named in them. This article reads them for a reader who needs to judge a claim without becoming a specialist.

The case is GLM-5.3, released by Z.ai on August 14, 2026. Three sources tested it or its predecessor: CAISI in the United States, the AI Security Institute in the United Kingdom and the company Anthropic.

CAISI and Anthropic both find that GLM-5.3 is a clear step beyond earlier open-weight models. They frame its distance from the closed frontier differently and measure different things, and the reasons teach more than the numbers do.

Key Takeaways

  • Three sources measure different things, so their numbers do not line up. CAISI and Anthropic tested GLM-5.3 and AISI its predecessor; they used different tasks, counted success differently and allowed different attempts, so the percentages cannot be set side by side.
  • CAISI, the US government tester, calls GLM-5.3 “the most cyber-capable open-weight model released to date”. It also finds its capabilities “significantly lower” than the US frontier, about four months behind on an aggregate measure.
  • The UK AI Security Institute puts the open-weight gap at four to seven months, down from six to ten in 2025. It tested GLM-5.2 and DeepSeek V4-Pro, not GLM-5.3.
  • Anthropic’s safeguard figures have not been replicated. CAISI and AISI did not repeat its safeguard tests, and The Decoder calls GLM-5.3 a direct competitor of Anthropic. Its capability findings, it says, broadly match CAISI’s.
  • Six questions sort most headlines: who measured, with what method, with which safeguards, what was left out, who gains from the result, and what can be compared.
  • No source I read measures the effect on marketing work or on an ordinary company. Figures for real attacks with GLM are not available in these reports.

What the Tests Measure, in Plain Words

A vulnerability is a flaw in software that an attacker can abuse, such as a bug in a web browser that lets a malicious page read files from a visitor’s computer. An exploit is the code, or the sequence of steps, that turns such a flaw into a break-in.

A benchmark is a fixed set of tasks given to many models, so the scores can be compared, the way an exam gives every student the same questions. Cyber benchmarks differ in what they ask. Some ask the model to find a flaw nobody pointed out, some hand over a known flaw and ask for the exploit, and some measure whether a model can carry out a long attack step by step in a practice network.

Two more terms come up throughout. Safeguards are the refusals and filters a developer builds into a model so that it declines harmful requests. Open-weight means the maker published the trained files, so anyone can run the model on their own servers and nobody can monitor how it is used afterward; I cover the business side in my article on Chinese AI models for business.

A clay caricature leans with crossed arms beside a plinth that holds one glowing cube, while three framed charts on the wall (a bar chart, a dotted line chart and a ring chart) each describe that same cube in a different way

Source One: CAISI, the US Government Tester

CAISI is the Center for AI Standards and Innovation, part of NIST, the US standards agency. It tests models from major developers, and it published its assessment of GLM-5.3 on September 17, 2026, about a month after the release.

Its summary calls GLM-5.3 “the most cyber-capable open-weight model released to date”. The same summary finds its cyber capabilities “significantly lower” than those of current US frontier models, with a lag of about four months on an aggregate measure. For most of the evaluations, Kimi K3, from Moonshot AI, had been the best Chinese model and the best open-weight model CAISI had evaluated before it.

The four benchmarks

CAISI used four benchmarks, and each asks a different question:

BenchmarkTasksWhat the model must do
SEC-Bench Pro183Find a vulnerability in the V8 or SpiderMonkey browser engines and write code that crashes it
ExploitBench41Exploit known flaws in V8; each task is scored out of 16, best of three attempts
ExploitGym (userspace)502Exploit real bugs in open-source software; the crash input is already known
CAISI OSS-Fuzz297Private benchmark on open-source projects; no description, crash or fix is given

V8 is the engine that runs JavaScript in Chrome, and SpiderMonkey is its counterpart in Firefox. A private benchmark keeps its tasks unpublished, so a model is less likely to have seen them during training.

The results

Here are CAISI’s results, with the task counts behind the percentages as CAISI prints them:

BenchmarkGLM-5.3Best US model testedBest other Chinese model tested
SEC-Bench Pro40.4% (74 of 183)90.2% (165 of 183)27.3% (50 of 183)
ExploitBench61.1% (9.8 of 16)100% (16.0 of 16)32.2% (5.1 of 16)
ExploitGym9.4% (47 of 498)44.4% (223 of 502)2.6% (13 of 502)
CAISI OSS-Fuzz7.7% (23 of 297)23.2% (69 of 297)2.4% (7 of 297)

Where the number comes from: CAISI ran every model on the same tasks, and the percentage is the share of tasks solved, or on ExploitBench the share of the 16 points available per task. What it means: GLM-5.3 is above the best other Chinese model on all four benchmarks, clearly on ExploitGym and OSS-Fuzz and with overlapping ranges on the other two, and it is well below the best US model on all four. What changes: “the most cyber-capable open-weight model” and “significantly lower” than the US frontier are both CAISI’s statements and both hold, so a headline that quotes only one gives the wrong picture.

The error bars matter too. CAISI reports 95% confidence intervals, the range in which the true score most likely falls. On SEC-Bench Pro, GLM-5.3’s 40.4% could sit anywhere from 33.6% to 47.7%, and on ExploitBench its 61.1% anywhere from 45.9% to 74.5%.

The best other Chinese model’s ranges on those two benchmarks, 21.4% to 34.2% and 19.9% to 47.4%, overlap with these, which is why I do not call the lead clear there.

CAISI does not say why GLM-5.3 was scored on 498 ExploitGym tasks and the other models on 502.

A clay caricature points at a board titled "CAISI: four cyber benchmarks" with four pairs of bars, a short lime bar for GLM-5.3 and a longer white bar for the best US model tested: SEC-Bench Pro 40.4% against 90.2%, ExploitBench score 61.1% against 100%, ExploitGym 9.4% against 44.4%, CAISI OSS-Fuzz 7.7% against 23.2%

The method

CAISI ran each model as an agent: the model reasons, runs a command in a terminal, reads the output and tries again, up to 200 or 300 turns depending on the benchmark. US models were tested with their cyber safeguards disabled where applicable, so the numbers show what the model can do, and not what it would agree to do for a stranger. Models that were not yet released were left out, and CAISI notes that they could be stronger.

The “about four months” comes from a single index. CAISI folds the four benchmarks into one “cyber capability index” with a method borrowed from exam testing, Item Response Theory: harder tasks count for more, and every model gets a place on one scale. A model 400 points higher on that scale has ten times the odds of solving a task.

GLM-5.3’s index sits above Kimi K3 and below the US frontier, by more than the error bars.

The model itself is the subject of my article on GLM 5.3 and the best open source LLM, so here I stay with the tests.

Source Two: The UK AI Security Institute

The UK AI Security Institute (AISI) is the British government’s testing body for advanced AI. Its analysis of the gap between open-weight and closed models appeared in July. It tested GLM-5.2 and DeepSeek V4-Pro, so it says nothing directly about GLM-5.3.

Its conclusion is that recent open-weight models trail the closed cyber frontier by four to seven months. That is narrower than the six to ten months AISI measured in internal evaluations through most of 2025.

Two kinds of test

The first kind is narrow tasks, each one a specific cyber skill. AISI used 70 tasks from its suite of 96, spread over four difficulty levels from “technical non-expert”, someone like a data analyst, up to “expert”: 18 tasks at the first level, 25 at the second, 19 at the third and 8 at the fourth.

Each task got five attempts. Each attempt could use at most 2.5 million tokens, small pieces of text, so the limit caps how long a model can work.

On these tasks GLM-5.2 performed comparably to the most cyber-capable models released four months before it, Opus 4.6 and GPT-5.3-Codex. AISI writes that this holds across all four difficulty levels. DeepSeek V4-Pro compared with Opus 4.5, released five months before it.

The second kind is cyber ranges, simulated company networks built by experts. One of them, “The Last Ones”, is a 32-step attack across 4 subnets and about 20 hosts, which AISI estimates would take a human expert roughly 20 hours. Each model ran it 10 times, except where AISI marks a best attempt, with a limit of 100 million tokens per run.

On that range, GLM-5.2 reached as far as Opus 4.5, a model released less than seven months before it. DeepSeek V4-Pro fell below Sonnet 4.5.

AISI treats the range comparison as weaker evidence, because it draws on fewer ranges than the narrow tasks do. A stall could come from too little hacking skill or from too little ability to keep a long plan going, and the trajectories do not say which. The ranges also lack the active defenders and defensive tools of a real company network.

A clay caricature kneels on a bridge made of calendar pages with a tape measure, between a locked vault on one cliff and an open glowing box on the other, under the title "UK AISI: the gap" and the figure "4 to 7 months", with the line "open-weight models behind the closed cyber frontier"

Safeguards and limits

AISI’s tests were largely unimpeded by safeguards. DeepSeek V4-Pro occasionally refused narrow tasks, mostly in reverse engineering, and AISI got past that with a small number of repeat attempts. AISI explains the reason: once the weights are public, monitoring, classifiers and user bans cannot be universally applied, and refusal training, the technique that remains, is “often easily reversible with access to the weights”.

AISI states its limits plainly. The setup likely slightly underestimates what the open-weight models can do, because the team did not pursue special optimizations, and the post only tests cyber capability: “inferences to other capabilities cannot be drawn”. The measured gap is also “not predictive of whether future open weight models will replicate the more recent jumps” that two closed models delivered in April 2026.

AISI’s practical reading is a window for defenders: capabilities that sit behind safeguards today may become available without them later.

Source Three: Anthropic, an AI Company

Anthropic makes Claude. Its post on GLM-5.3 appeared after CAISI’s report and says: “Our capability findings broadly match CAISI’s.” It has three parts: benchmarks, human-led sessions and safeguard tests.

Its conclusion is that GLM-5.3 is “a meaningful step change in the cyber capabilities available to attackers”, and it thinks state and non-state actors will likely use models like it to cause real-world harm.

Benchmarks

On ExploitBench, Anthropic counts how often a model builds a complete, working exploit from start to finish. GLM-5.3 did so in 50 of 410 attempts, and Claude Mythos Preview in 56 of 410.

On Anthropic’s internal Binary Exploitation benchmark, built from open-source projects, full credit goes to a “full control-flow hijack”, where the attacker takes over what the program executes. On 100 randomly chosen tasks, GLM-5.3 managed it in 4% of trials and Mythos Preview in 6%.

Anthropic adds that “a meaningful threshold has clearly been crossed”: earlier models like Opus 4.6 and GLM-5.2 succeed in none of them. The Claude models in these charts ran with safeguards disabled.

Human-led sessions

In the first session, a researcher used GLM-5.3 on a sandboxed machine with a local Linux build of a popular browser. Over about a day, with limited human attention, the model found several previously unknown vulnerabilities in the browser’s JavaScript engine and chained them into an exploit: a web page that, when visited, reads arbitrary files from the visitor’s computer. Anthropic says it disclosed the flaws to the maintainer.

In the second session, a researcher gave GLM-5.3-Flash, the smaller model, the public details of a recently disclosed Chrome flaw and another known flaw. The model chained the two into a reliable exploit for an ARM64 target, bypassing a security feature built into the processor chip. Anthropic reports 20 minutes of human attention plus eight hours of model work, and says that at Zhipu’s API prices the effort would have cost $20.40.

These are two sessions that Anthropic describes, so they show what is possible and say nothing about how often it happens.

Safeguard tests

Anthropic placed the model in a simulated world and gave it overtly malicious requests to attack critical systems. No code was run, and a second model approximated what each command would return, so the test cannot show that an attack would have succeeded. CAISI and AISI have not repeated it.

In this simulation, “engaged” means the model tried to connect to a remote target system after the harmful request. Each cell of the results table holds 50 samples: five attack orders, two targets, five attempts each. Anthropic reports that GLM-5.3 engaged in these shares of the samples:

  • A direct request: 0%
  • A deceptive prompt, meaning a false cover story: 64%
  • Prefilled reasoning, where the attacker writes the first lines of the model’s own thinking: 92%
  • An abliterated copy, one whose refusals were edited out of its weights: 100%

The Claude models stayed at 0% in every condition that applies to them. Claude’s safeguards blocked the deceptive prompts, the Anthropic API offers no way to prefill Claude’s thinking, and Claude’s weights are not provided, so they cannot be abliterated.

Anthropic also measured the abliterated copies on three public benchmarks of harmful requests. The mean refusal rate fell from 95% to 6% for GLM-5.3 and to 14% for GLM-5.3-Flash, with capability scores nearly unchanged. The technique and what it means for choosing a model are the subject of my article on abliteration explained.

The conflict of interest

Two outlets raised it. The Decoder headed a section “Anthropic’s warning also serves its business”: Anthropic does not release its model weights, and a cheap Chinese open-weight model near the frontier is a direct competitor.

The same piece adds that the warning is not only self-interest: “CAISI’s independent assessment backs up the capability numbers”.

Trending Topics notes that Anthropic is a competitor of Z.ai, quotes a Hacker News reader asking about a “research conflict of interest with a direct competitor” and adds that Z.ai had not yet responded to Anthropic’s analysis. The Cloud Security Alliance’s AI Safety Initiative, in an October 4 research note, records that Anthropic’s evaluation has not been replicated by CAISI or AISI.

One voice disagrees on the threat itself. Jake Williams of IANS Research, quoted by Trending Topics from The New Stack before the release, said he expects threat actors to use the model but answered “Absolutely not” when asked whether it will be “a significant change in the threat landscape”.

My reading is that the capability level has a second source, CAISI. The safeguard percentages rest on Anthropic’s own simulation, so I read them as a claim that still waits for a second test. The direction has partial support: CAISI’s earlier page on GLM-5.2 states that the model’s safeguards “allow assistance with agentic cyber exploit development”.

Why the Numbers Do Not Line Up

Put the three sources in one table and the differences are visible before any number appears:

CAISIUK AISIAnthropic
WhoUS government institute at NISTUK government instituteA company that competes with Z.ai
Model testedGLM-5.3GLM-5.2 and DeepSeek V4-ProGLM-5.3 and GLM-5.3-Flash
What counts as successShare of tasks solved; on ExploitBench a graded score out of 16 per taskShare of 70 narrow tasks solved; steps completed on a practice networkNumber of complete end-to-end exploits out of 410 attempts; share of trials on 100 tasks
AttemptsBest of three on ExploitBenchFive per narrow task; 10 runs on The Last OnesCounted over 410 attempts
Main findingMost capable open-weight model; significantly below the US frontier; about four months behindOpen-weight models 4 to 7 months behindCapability findings broadly match CAISI’s; in its own simulation, not repeated by CAISI or AISI, safeguards bypassed in 64% to 100% of samples

Take ExploitBench, the one benchmark that both CAISI and Anthropic ran. CAISI gives each task a graded score out of 16 and keeps the best of three attempts, so a model that gets part of the way earns part of the points. Anthropic counts only the complete result, an exploit that works from end to end.

The two numbers answer different questions, and for that reason I do not put them in the same row. One measures how far a model gets on average, the other how often it arrives.

The yardsticks differ as well. CAISI compares GLM-5.3 with the best US model it tested on each benchmark, a group that includes trusted-access releases as well as public ones, while Anthropic compares it with its own Claude Mythos Preview. The Cloud Security Alliance’s note sees a tension between Anthropic’s framing and CAISI’s “significantly lower” assessment, and says the difference depends heavily on which benchmark and which US comparator is used.

A clay caricature squints with a puzzled half smile while he holds three different rulers (a fine one, a coarse one and a curved one) against the same small glowing cube on a table

Six Questions for Any AI Cyber Headline

These are the questions I now ask before I repeat a number.

  1. Who measured it? A government institute, a lab, a vendor or a model maker? CAISI and AISI are government bodies; Anthropic is a company, and Z.ai’s own benchmark table is the maker’s claim.
  2. With what method? Look for the task type, the unit of the score and the number of attempts. A graded score out of 16 and a count of complete exploits are different things.
  3. With which safeguards? CAISI tested US models with safeguards disabled where applicable, and Anthropic ran its Claude models without safeguards for the capability charts. A result with safeguards off shows what a model can do and says little about what it will do.
  4. What was left out? CAISI excluded unreleased models, AISI’s ranges have no active defenders, and AISI tested a predecessor of the model in the headline.
  5. Who gains from the answer? A company that sells closed models gains when open-weight models look risky. A maker gains when its model looks strong. A government institute has a mandate too: CAISI’s report compares US and PRC models by design, and part of its method, the OSS-Fuzz benchmark, is private.
  6. What can be compared? Only numbers from the same tasks, the same attempts and the same safeguards. When the units differ, compare the direction of the finding and leave the percentages alone.

A clay caricature leans on a clipped card titled "6 questions for any AI cyber headline" with six numbered lines: Who measured it? With what method? With which safeguards? What was left out? Who gains from the answer? What can be compared?

What the Tests Do Not Say

The reports leave three things open.

  • Real attacks. I found no figures in these reports for real-world attacks that used GLM. The tests run in labs, sandboxes and simulations.
  • The maker’s own numbers. Z.ai publishes a benchmark table for GLM-5.3, including cyber scores. I treat it as the maker’s claim and do not use its numbers here.
  • The future. AISI’s gap is not predictive of the jumps that closed frontier models made in April 2026, so the four to seven months it measured, like CAISI’s four, describe the past and are not a forecast.

What This Means for a Company Outside Security

An invented example: a company of 40 people has a website, an online shop and a customer list in a CRM, the tool that stores customer records. A headline says an AI can now attack browsers on its own.

What the reports support for that company is modest. AISI’s conclusion is a window for defenders, and the NCSC advice it cites is to invest in cybersecurity baselines, the minimum measures every organization should have, and in AI-enhanced defences.

The rest is my inference and not a finding of the reports. Anthropic’s second session turned the public details of a disclosed flaw, plus one other known flaw, into a working exploit in a short time, which makes me ask how fast updates are installed. I suggest four small steps, and the reader decides which fit:

  1. Ask your IT provider how quickly software updates are installed, and how you would know if they were late.
  2. List the AI tools and agents that have access to your accounts, and what each one can do with that access. My article on AI marketing agents covers the setup I use.
  3. Keep a person in the approval path for anything an agent sends outside the company. I describe that practice in human-in-the-loop AI verification, and the controls around an agent in AI agent security.
  4. Ask vendors for evidence in the form of the six questions above, and treat a claim without a method as marketing.

What It Costs

The reports contain a few cost figures, and each one needs its source.

  • AISI’s cost estimate for open-weight models. A 100-million-token cyber range run cost roughly 85 dollars for Opus 4.5 and Opus 4.6, an estimated 46 dollars for GLM-5.2 and 1.19 dollars for DeepSeek V4-Pro. These are advertised prices, so real compute costs may vary.
  • Anthropic’s abliteration estimate. Anthropic reports about 2,200 GPU hours, roughly 4,400 dollars, for its first time using the technique, and estimates closer to 600 GPU hours, 1,200 dollars, for a team experienced with it. Both figures are Anthropic’s own, and no independent test has confirmed them.
  • The cost of a real attack. Not available. No source I read gives it.

For a marketing team, the figure that matters most, in my view, is none of these. It is the cost of one wrong approval by an agent that has access to your accounts, and that depends on the controls you set up.

A Reading Table for the Next Headline

This table applies the six questions to the GLM-5.3 case, and you can reuse it on the next headline.

QuestionWhere to find the answerExample from GLM-5.3
Who measured it?The first paragraph and the “about” pageCAISI (US government), AISI (UK government), Anthropic (competitor of Z.ai)
With what method?The method notes and the figure captionsAgent with bash and python; graded score out of 16 vs complete exploits out of 410
With which safeguards?The setup sectionUS models tested with cyber safeguards disabled where applicable
What was left out?The limitations sectionUnreleased models excluded; ranges without active defenders; GLM-5.2 tested by AISI
Who gains?Who publishes, who sells, who is namedAnthropic does not release its weights; Z.ai sells GLM; CAISI’s report compares US and PRC models
What can be compared?The unit of each numberCompare the finding that GLM-5.3 is the strongest open-weight model, and keep the percentages apart

A clay caricature smirks as he pins a news clipping onto the first row of a large board with six rows, each ending in a lime check box, in front of a wall of small pinned clippings

Frequently Asked Questions

What is CAISI?

CAISI is the Center for AI Standards and Innovation, a US government body inside NIST. It evaluates advanced AI models, including Chinese ones, and publishes its findings.

What is the UK AI Security Institute?

It is the British government’s body for testing advanced AI. It runs cyber evaluations on narrow tasks and on simulated networks, and its July analysis estimated the gap between open-weight and closed models.

What is AI in cybersecurity?

It covers two uses: models that help defenders, for example by reading logs or finding flaws before attackers do, and models that help attackers do the same work faster. The tests in this article measure the second use, as a way of judging risk.

Can an AI carry out a cyberattack on its own?

In AISI’s 32-step practice network, GLM-5.2 progressed to about step 11 before stalling, and the network had no active defenders. Anthropic describes two sessions in which GLM models built working exploits with limited human attention. I found no figures for real-world attacks.

Is GLM-5.3 more dangerous than the US models?

CAISI finds its cyber capabilities significantly lower than the US frontier, and that frontier includes trusted-access releases, not only public models. The difference is in deployment: AISI explains that once weights are public, monitoring cannot be universally applied and refusal training is often easily reversible.

What is the difference between a benchmark and a real attack?

A benchmark is a fixed set of tasks scored in a lab. A real attack faces monitoring, updates and people who react, which the practice networks in these reports do not include.

Why do different tests give different numbers?

They use different tasks, different units for the score, different numbers of attempts and different safeguard settings. Numbers can be compared only when these match.

Should a marketing team worry about this?

I suggest a calm response: check how fast your updates are installed, know which AI tools have access to your accounts, and keep a person approving what leaves the company. The reports measure cyber capability and say nothing about marketing work.

My Take

A test result is a measurement made by someone, with a method, under conditions. Reading the method tells you more than the headline does, and the six questions are quick to apply.

On GLM-5.3, the picture I take from these reports is a model that leads the open-weight field and trails the closed frontier by months, with safeguards that depend on how it is deployed. The capability level has two sources, CAISI and Anthropic, who describe the distance from the frontier differently. The safeguard percentages have one, and they come from a company with an interest in the result.

For a business that uses AI agents, I would put the attention on what the agents can reach and who approves their actions, since that part is yours to control.

Tags: ai cyberattacksai securityCAISIUK AISIGLM-5.3open weight modelsai benchmarks

Keep reading

All writing →
Newsletter

Stay ahead.

Occasional notes on AI marketing, growth, and speaking. No spam.