The test suite that tested nothing

Here is a test that has never caught a bug and never will, and it sits in a great many repositories:

if TEST_DATABASE_URL is not set: skip this test

There is nothing wrong with the line. It is there on purpose, for a good reason. The problem is what it does to your signal when nobody is watching.

Why the skip exists

Integration tests need a real database. Unit tests do not. If every test in the suite demands one, then anyone with no database running cannot run anything, so they stop running tests at all. Guarding the database tests with a skip keeps the fast suite usable in a repository checked out thirty seconds ago.

That is a good trade. The cost is that a skipped test reports as success.

The failure mode

Set up CI without that environment variable and watch what happens. Every integration test skips. The suite is green. The badge is green. Coverage barely moves, and nobody reads a coverage delta on a green build.

You now have a repository where the data layer, the migrations, the transaction boundaries and the constraint violations are completely untested, and every signal available says things are fine.

It is worse than having no tests, because having no tests is at least honest about it.

A skip is a conditional, so give it two branches

The fix is to stop treating "no database available" as one situation. It is two situations with opposite correct behaviours:

• On a developer machine, a missing database means skip. The developer is working on something unrelated and should not be blocked.
• In CI, a missing database means the pipeline is misconfigured, and the correct response is to fail.

Nearly every CI provider sets a CI environment variable, so the branch is one line: if the database URL is missing and CI is set, fail with a message naming the variable that should have been there. Otherwise, skip.

Put that helper in one place, have every integration test call it, and the guarantee covers tests nobody has written yet.

Count your skips

Even with the guard, make the number visible. Most test runners print a single success line for a suite whose every test skipped, which is the root of the whole problem. Verbose output names each skip, and the count belongs in the CI log deliberately rather than buried.

The useful question is not "did the suite pass". It is "how many tests ran, and is that the number I expected". A suite that quietly drops from four hundred tests to forty is the highest value alert available in a test pipeline, and almost nobody has it wired up.

The same shape, elsewhere

Once you see this pattern you find it everywhere, and it is always the same bug: a fallback designed for convenience that is indistinguishable from success.

• A fixture helper that returns an empty collection instead of failing, so assertions compare nothing to nothing.
• A permissive mock that accepts any call and returns a zero value, so a renamed method silently stops being exercised.
• A try block wrapped around an assertion, swallowing the failure it was meant to report.
• A conditional that disables a check in precisely the environment where the check mattered.
• A retry loop that hides a dependency which is down, until the day it is down for longer than the retries last.

Each one is a green light wired to a disconnected sensor.

What to hold yourself to

Coverage is a smoke detector. It reliably tells you what is untested, and very little about what is tested well. A hundred percent of happy paths is worth less than seventy percent that reaches the error branches, the boundaries and the concurrent case.

So make the rules concrete:

• Anything touching money, authentication, state transitions, concurrency or data migration has a test, whatever the coverage number says.
• Every bug you have actually hit keeps a regression test forever. That is how a bug you have hit once does not come back.
• Integration tests run against the real dependency, not a stand-in. A suite that only ever talks to fakes proves the fakes work.
• Reproduce before you fix. A failing test that demonstrates the bug is worth more than a debugging session, because the test stays.

And never weaken a test to get a green build. Deleting the assertion, loosening the matcher or adding a skip to make a pipeline pass is not fixing the build, it is falsifying the result. The cost comes due later, at a worse time, with less context.

If a test cannot pass honestly, that is information. Keep it.