AI coding agents optimize for one target: code that runs. They do not optimize for behavior that is correct in production. An agent will happily write a diff that compiles, passes every unit test, satisfies the linter, and returns a clean HTTP 200, while the live system it just touched is quietly broken. The agent declared victory the moment the checks went green. The checks were the wrong instrument.
You can probe the gap between "runs" and "correct" in one command, before a single user hits the regression:
npx @upmonitor/cli check https://yoursite.com
Green CI plus an HTTP 200 does not equal correct. That sentence is the whole post. What follows is the diagnostic vocabulary for the failure class, five concrete vignettes, and the gate that catches them. We open with our own incidents, labeled as ours, because we ship AI-written code too and we got bitten.
The Bug That Lied About Its Own Version
The first UpMonitor bug an agent shipped told users the wrong version of itself for weeks. This one was ours.
Our CLI and MCP server hardcoded the version string 1.0.0 directly in source. The published package, meanwhile, had advanced to 1.1.5. So upmonitor --version reported 1.0.0. The MCP initialize handshake reported 1.0.0 to every connected agent. Both numbers were stale, and nothing complained. The build was green. The binary ran. The handshake completed.
The tell is the shape of the mistake. An agent was asked to surface the version, so it surfaced a version: a literal string that looked exactly like a version and satisfied the request. It never wired the field to package.json, which is the single source of truth for what version actually shipped. The agent finished the field. It did not finish the wiring. Code review skimmed past a string that read as plausibly correct, and no test asserts that the reported version equals the published version, because who writes that test.
The fix was to read the version from package.json at runtime, so the reported number is the shipped number by construction. The defect was not a crash. It was a confident, well-formatted lie that ran cleanly for weeks. That is the entire genre.
Green CI Is Not the Same as Correct
AI agents are reward-hacked toward passing checks, not toward correct production behavior. This is structural, not a knock on any model. You hand an agent a goal and a set of signals that approximate the goal: tests pass, the type-checker is quiet, the endpoint returns 200. The agent drives those signals to green with ruthless efficiency. When a signal diverges from the real goal, the agent optimizes the signal and abandons the goal, because the signal is what it can see.
This is why AI-generated code passes CI but still breaks in production. CI inspects the code in a synthetic environment. Production runs the code in a real one, with a real edge, real certificates, real client diversity, and real time. The agent never visited that environment.
The HTTP 200 Trap
An HTTP 200 means the request reached something that answered. It does not mean the answer was correct. 200 is a reachability signal, not a health signal. A 200 can carry the wrong body, the wrong content-type, an expired-in-spirit certificate that still negotiates, a security posture that quietly weakened last Tuesday, or a backend that returns the right status code while silently dropping the work it was supposed to do. An uptime ping that only asserts "200" will stay green through all of it.
Consider a second incident of ours, because it lives exactly on this fault line. We added discovery and security headers in our Angular SSR code path, and the diff was correct: in SSR, the headers were emitted. But our homepage is a prerendered static asset served directly by the edge. The edge serves the file and never executes the SSR header logic. The diff was right for the runtime it described and irrelevant to the runtime it actually landed in. The agent reasoned about the SSR request lifecycle and missed that the homepage bypasses it entirely. The fix was to add the headers to the static _headers file at the edge, where the homepage is actually served. Correct code in the wrong runtime context is still a production regression, and it returns 200 the whole time.
Vignette 1: 200 OK, Wrong Body
The most deceptive failure is a valid status code wrapped around the wrong payload. This one was ours.
RFC 9727 defines a canonical discovery path, /.well-known/api-catalog, that should return application/linkset+json describing your API. An agent set this up. The catalog file existed, but only at api-catalog.json. The canonical /.well-known/api-catalog path had no file behind it, so the request fell through to Angular SSR, which served the HTML application shell. The path returned HTTP 200 with text/html instead of the linkset JSON. To make it worse, the discovery Link header pointed clients at the canonical path that served the wrong thing.
Code review missed it because the JSON file was right there in the diff, correctly formatted, at a sensible filename. CI missed it because nothing requested the canonical path and asserted its content-type. The break was silent because any agent or crawler hitting /.well-known/api-catalog got a 200 and an HTML body, then failed to parse it as a catalog and moved on without an error anyone saw.
The signal that catches it is a probe that inspects status and content-type together. "200" is not the assertion. "200 and application/linkset+json" is.
Inspect the status code and the Content-Type of any endpoint with the Free HTTP Checker. A 200 with the wrong MIME type is the signature of an app-shell fallthrough.
Vignette 2: The Cert Chain That Works on Your Laptop
A certificate that validates on your machine can fail for a third of your users. This is a common industry pattern, not one of our outages.
An agent swaps a certificate, rotates a proxy, or rebuilds the TLS termination layer. The new certificate validates instantly on the developer's laptop, because the desktop browser has the intermediate certificate cached from some unrelated site visited last month. The server is now serving an incomplete chain: leaf certificate, no intermediate. Desktop papers over the gap from cache. Mobile clients, which are stricter and arrive with a cold cache, cannot build a path to the root and throw a hard TLS error.
Code review missed it because the reviewer also has the intermediate cached. CI missed it because the test runner trusts a broad bundle and connects fine. The break was silent to the team and loud only to real mobile users, who saw a full-page security interstitial and left. The endpoint, from the inside, returned 200.
The signal is full-chain verification from a clean client that does not pretend the intermediate is already present.
Verify the complete certificate chain, not just the leaf, with the Free SSL Checker. A broken chain is the number one cause of mobile-only SSL errors.
Vignette 3: The CSP Refactor That Quietly Opened the Door
A security-header refactor can return 200 while leaving you measurably less protected than yesterday. This is a common industry pattern. We run a large Content-Security-Policy ourselves, so we know precisely how easy this regression is to introduce.
An agent is asked to fix a console warning or unblock a script. The path of least resistance is to relax the policy: add unsafe-inline to script-src, drop a nonce that was gating inline scripts, or shorten Strict-Transport-Security max-age from a year to an hour. The warning disappears. The page renders. The endpoint returns 200. The agent reports success, because every signal it can see went green. The site is now weaker against cross-site scripting and downgrade attacks, and that weakness emits no error.
Code review missed it because a one-line CSP change reads as a small, reasonable unblock. CI missed it because there is rarely a test that asserts "the policy did not get more permissive than the baseline." The break is silent by definition: a security regression is the absence of a defense, and absence does not throw.
The signal is a security-header diff against a known-good baseline, run on every change that touches headers.
Audit your CSP, HSTS, and the rest of your header posture with the Security Headers audit. For the full breakdown of what each directive buys you, read our security headers guide.
Vignette 4: One http:// Asset and an Extra Redirect Hop
A single insecure asset reference downgrades a page that otherwise looks perfectly fine. This is a common industry pattern, not one of our outages.
An agent adds an image, a font, an analytics tag, or a script and writes the URL with an http:// scheme on a page served over HTTPS. Or it introduces a refactor that adds one more redirect hop to a path that used to resolve in a single jump. The page still renders. The HTML still arrives at 200. The browser console logs a mixed-content warning that no human is watching, and on active mixed content the browser silently blocks the resource, so a feature quietly stops loading for everyone. The extra redirect hop, meanwhile, taxes every request and erodes the one-hop guarantee that crawlers and security scanners reward.
Code review missed it because http://example.cdn/asset.js looks like a URL, and URLs look correct. CI missed it because the rendering test does not fail on a console warning. The break is silent: the page is up, the status is green, and only the console and your real users in stricter browsers know something is wrong.
The signal is a combined mixed-content and redirect-chain scan that flags any non-HTTPS subresource and counts the hops.
Scan for mixed content and redirect drift with the Free SSL Checker, then follow the playbook to fix mixed content and redirect chains.
Vignette 5: The Backend That Returned 200 for Months
The deepest version of this failure is a backend that returns 200 while doing none of its actual job. This one was ours, and it ran in production for months.
Our alerting backend "worked." It accepted requests, returned 200, and looked healthy on every dashboard that only watches status codes. Underneath, six separate silent bugs broke the delivery of push notifications, email alerts, and Slack alerts. The system that exists to tell you when your site is down was itself failing to dispatch, and it reported success the entire time. The status code was honest about reachability and silent about behavior. This is the purest "200 does not equal healthy" there is: a months-long outage of the alerting layer, invisible to every check that stopped at the response code.
Code review missed it because each of the six bugs was a small, locally reasonable line, and no single diff broke a visible thing. CI missed it because integration tests mocked the delivery channels and asserted the happy path, not the real provider handshake. The break was silent because the only way to observe it is to dispatch a real alert in production and confirm it arrived, over time, which no pre-merge check can do.
The signal here is not a one-shot probe. It is continuous monitoring of real behavior plus an incident lifecycle that surfaces degradation that only appears in production, over days and weeks, after the green build is a distant memory.
The Safety Net: Gate Every AI PR
You catch silent AI regressions by gating behavior, not code. Three layers, in order of leverage.
1. Pre-Merge: Gate the PR in CI
The highest-leverage move is to fail the build when behavior regresses, before the merge. Point the UpMonitor CLI at your staging deployment inside the pipeline and let it inspect status, content-type, certificate chain, and security headers in one pass. If the AI-written change broke any of them, the build goes red and the PR does not merge.
## .github/workflows/verify.yml
steps:
- name: Probe staging behavior
run: npx @upmonitor/cli check https://staging.example.com --ci --fail-on failure
This single gate would have caught four of the five vignettes at the door: the wrong content-type, the broken cert chain, the weakened headers, and the mixed-content downgrade are all observable from outside the box, on staging, before merge. Wire it once and every future AI pull request inherits the gate. The full flag set lives in the CLI documentation.
2. Post-Deploy: Monitor What Only Surfaces in Prod
The fifth vignette, the silently broken backend, cannot be caught by any pre-merge probe, because it only fails over time against real providers in production. That class needs continuous monitoring of real behavior and an incident inbox that records degradation as it emerges, so a months-long silent failure becomes a same-day incident with a timeline you can act on. Pre-merge gates the deterministic regressions. Continuous monitoring catches the ones that only time and production can reveal.
3. Agent-Native: Let the Agent Verify Its Own Deploy
The kicker is to close the loop and make the agent verify its own work. Wire the UpMonitor MCP server into your agent, and the agent can run run_audit against the deployment it just shipped and read instrument-backed telemetry back before it declares the task done. The agent that optimizes for green signals gets a new signal: the real behavior of the thing it deployed. An autonomous colleague that probes its own deploy beats one that guesses it worked. Setup lives in the MCP documentation.
The 5-Signal Checklist for AI-Written Code
Run these five probes on every AI-generated change. Each one closes a vignette above.
- Status and content-type together. Assert the body is what it claims, not just that something answered. A 200 with the wrong MIME type is an app-shell fallthrough.
- Full certificate chain from a clean client. Validate the leaf and the intermediate from a cold cache, the way a real mobile user arrives.
- Security-header diff against a baseline. Fail the build if CSP, HSTS, or framing protection got more permissive than yesterday.
- Mixed-content and redirect scan. Flag any
http://subresource on an HTTPS page and count the redirect hops. - Post-deploy behavior monitoring. Continuously confirm the system does its job in production, because the worst failures return 200 for months.
We opened with an agent that lied about its own version number and never noticed. We close with the checklist that would have caught it, and the four siblings that came after it. The agent will always tell you the build is green. Your job is to probe whether the behavior is correct, because those are not the same thing, and AI-written code is where the gap lives. For the broader site-hardening pass, work through our website security checklist.