The test that failed every morning and passed every afternoon

The test that failed every morning and passed every afternoon

Originally published on the WatchNext blog.

While making an unrelated change - adding a processor to a privacy page and a link to a footer - the test suite came back with eleven passes and one failure:

• 1 + 2 ❯ tests/airDate.test.ts:125:47
  125| expect(getAirDateDaysDiff(plus(1), "US")).toBe(1); | ^
  An episode airing tomorrow was being counted as two days away. That is about as load-bearing as a bug gets in an app whose entire purpose is telling you when the next episode airs. It turned out the countdown was fine. The test was wrong, in a way that made it fail for roughly a third of every day and pass for the rest.

First: prove it isn't your change

The edited files were a privacy page, a footer and a markdown document. The failing test covers air-date arithmetic. Those are obviously unrelated - but "obviously unrelated" is a hypothesis, and the cost of checking it is one command:

git stash && npx vitest run
# same failure
git stash pop

Identical failure with the changes removed. That single result reframes everything that follows: this is not a regression being debugged, it is an existing defect being discovered. Without it, the natural next move is to start reading your own diff for a cause that was never in it - and the worst outcome of that search is "fixing" code of your own that was correct.

Then: rule out the machine

A test about day counting failing intermittently points at timezones, so the first suspect is the machine's own zone. That is easy to test by simply telling the process it lives somewhere else:

TZ=Europe/Dublin npx vitest run
# 1 failed
TZ=America/New_York npx vitest run
# 1 failed

Both fail, identically. That is a useful negative: the host timezone changes what the process calls local time, but it does not change the instant at which the test runs. So the hidden dependency is not where the test runs. It is when.

Two calendars, one comparison

Here is the test as written. It builds "tomorrow" and "yesterday" by adding and subtracting a day in milliseconds, then trimming the result to a date:

const today = new Date();
const iso = (d: Date) => d.toISOString().slice(0, 10);
const plus = (n: number) => iso(new Date(today.getTime() + n * 86400000));
expect(getAirDateDaysDiff(plus(1), "US")).toBe(1);
expect(getAirDateDaysDiff(plus(-1), "US")).toBe(-1);

That looks unimpeachable, and it contains the whole bug. toISOString() always formats in UTC. So the date handed to the function is "tomorrow according to UTC". The function, though, deliberately does not count days in UTC. It counts them in the show's own country's timezone, and for a US show that means America/Los_Angeles. That choice is not an accident or an oversight - it is the fix for an earlier bug where a show's card and its notification disagreed about whether the same episode aired today or tomorrow, because one of them mixed the viewer's timezone into the comparison.

So the test measures from one calendar and the function measures from another. For most of the day they agree. For the hours when UTC has already rolled over to a new date and Los Angeles has not, they do not:

At 06:39 UTC on 12 September it is still 23:39 on 11 September in Los Angeles. UTC's "tomorrow" is 13 September. Los Angeles's "today" is the 11th. Two days apart, and the assertion wanted one. Which means it fails on a timetable

The failure window

The failure window is exactly the offset between the two zones: seven hours while Los Angeles is on daylight time, eight hours when it isn't. Midnight UTC to around 07:00 UTC, every single day. After that it goes green again on its own.

Two things about that are worse than an ordinary broken test

The first is that it presents as a feature bug - the assertion says an episode airing tomorrow is being counted as two days out, which is precisely what a real off-by-one in the countdown would look like. The second is that re-running it later makes it pass, which quietly teaches everyone that it is "flaky" and can be re-run rather than read.

The fix, and what it deliberately doesn't cover

The repair is to build the fixture in the same frame the function measures in, rather than in UTC:

const zone = timeZoneData.find((e) => e.iso_3166_1 === "US")?.ianaTimeZone;
const plus = (n: number) => DateTime.now().setZone(zone).plus({ days: n }).toISODate() as string;

The zone is looked up from the same table the application uses, so the test does not hardcode a mapping that could later drift out of step with the code. That has a real cost worth naming: because the fixture now derives its zone from the same table as the code under test, this test can no longer catch a wrong entry in that table. It is not trying to. Its job is the day arithmetic, and covering the mapping is a different test with fixed, hand-written dates.

A test that quietly tests two things is how you end up with one that fails for reasons its name doesn't explain

Checking the fix didn't just silence it The easiest way to make a failing test pass is to adjust it until it agrees with whatever the code currently does. That is not a fix; it is a deletion with extra steps. So the fixed test was checked the same way the original air-date work was - by breaking the code on purpose and confirming the test notices. The mutation was to reintroduce exactly the bug this block of tests is named after, replacing the zone-aware "now" with a plain local one:

- const now = zone ? DateTime.now().setZone(zone) : DateTime.now();
+ const now = DateTime.now();

Red again, as it should be. The test still guards the thing it was written to guard; it simply no longer reports a failure that depends on what time you asked.

The general version

Any test that calls new Date() has an input that isn't written down anywhere in it. Most of the time that input is harmless. It stops being harmless the moment the code under test cares about which calendar it is using - and if you are dealing with schedules, billing periods, delivery estimates or anything else with a day boundary in it, your code cares.

There are two honest ways out. Freeze the clock, so "now" is a fixed value you chose. Or, as here, build your fixtures in the same timezone the code measures in, so there is only one calendar in the comparison. What does not work is the thing that feels most neutral: reaching for UTC in a test because UTC feels like the absence of a timezone. It isn't. In a codebase that reasons in a show's local calendar, UTC is simply a third timezone that nobody asked for - and it will agree with the right answer often enough, and for long enough, to get committed.

WatchNext is a deliberately simple TV tracker: it tells you when the next episode of your favourite shows airs, for any show, in any country - and nothing else.