Your deploy went green and changed nothing

A green pipeline is a claim about a script's exit code. It is not a claim about production. The gap between those two statements is where a surprising number of outages live, because nothing looks less alarming than a checkmark.

Here are four ways a deploy reports success without changing anything, and the checks that catch each one.

The health check that proves nothing

The obvious way to verify a deploy is to request the site afterwards. If it answers, the deploy worked.

It does not. A health check cannot tell the difference between "the new version is serving traffic" and "the old process never stopped, and it is still answering beautifully." Both return 200. If the build failed half way through, the previous release keeps serving, and your verification step congratulates it.

A health check measures whether something is up. It says nothing about what.

The build that was killed by a hangup

If your pipeline deploys over SSH, the connection is part of your build. When it drops, the signal reaches whatever it started, the remote work dies partway through, and the runner can sit waiting on a socket with nothing behind it. The job shows as still running. Nothing deployed, and nothing failed either, in the sense that CI understands failure.

Two fixes, and you need both:

• Start the remote work detached, with the process ignoring hangups and stdin closed, so a dropped connection cannot signal it.
• Have the runner poll for a completion signal over fresh connections, rather than holding one connection open and trusting it.

The second half is the part people skip. If the runner's only link to the deploy is the connection it opened, that connection is a single point of failure for your knowledge of what happened.

The marker that names the commit

Have the deploy write a completion marker as the very last thing it does, and have it record which commit it deployed. The pipeline then asserts that the marker names the commit it asked for. Not "a marker exists". Not "the site is up". The exact revision.

This single assertion catches the entire class of problems where something deployed, just not what you wanted: a stale checkout, a reset that did not apply, a build served from the wrong cache layer, a second deploy that overtook the first. Without it, "the script finished" is the strongest statement available, and it is far weaker than it sounds.

The two merges that raced

Two merges landing a minute apart start two deploys. Both pull, both build, both bring services up, and they interleave on the same machine. The result is not the second release. It is a deployment rebuilt on top of itself, in whatever order the two runs happened to reach each step.

The fix is boring: serialize deploys per target, and let an in-flight deploy finish rather than cancelling it. Cancelling a deploy mid-build leaves exactly the inconsistent state you were trying to avoid.

The job that could run forever

Give the deploy job a hard timeout. This is not an optimization, it is an admission. Your keepalive and polling logic represents your current understanding of how a remote deploy can hang, and that understanding has a history of being incomplete. A timeout is the backstop for the failure mode you have not met yet.

Verify from inside

When checking that a deploy worked, request the real hostname but pin it to the machine's own loopback address.

Requesting from outside tests a chain: DNS, whatever proxy or CDN sits in front, certificate validity, then finally the application. Any link can serve a cached success for an application that is down, or report failure for one that is healthy behind an expired certificate. Pinning the request to loopback tests the single thing you are actually asking about.

Measure the component you care about, not the pipeline in front of it.

The checklist

1. Tests pass, and the deploy job declares them as dependencies so a red build cannot deploy.
2. Deploys run on pushes to real branches only, never on pull requests.
3. Remote work runs detached, so the network cannot kill it.
4. The runner polls for a marker written on the script's last line.
5. The marker must name the expected commit, or the job fails loudly.
6. The post-deploy request is pinned to the machine's loopback interface.
7. One deploy per target at a time, with a hard job timeout.

None of this is clever. Every item on it exists because somebody once spent an afternoon staring at a green pipeline and a production machine that disagreed with it.

The general rule

A check is worth exactly what it can disprove. "Did the script exit zero" disproves very little. "Is the revision I asked for the revision now running, verified from inside the machine" disproves nearly everything that goes wrong in practice.

Write the check that can fail. A check that cannot fail is decoration.