Tips / Tip

Compare two environments' config and ask which differences matter

By Michael Shreeves · 9 August 2026 · IT & Systems · 2 min read

It works in test and not on live. There's a difference somewhere that nobody wrote down.

A diff tool will find every difference, and it won't miss one. What it can't do is tell you which one matters, because in two config files of any size most of the differences are fine. Ranking them is the part an AI tool can add.

So find the differences with the tool, and ask the AI to rank them.

  1. Run a plain diff. Your editor's compare, or git diff --no-index test.conf live.conf.
  2. Take the secrets out of the diff, but keep what you know about them. Replace each value with [SECRET: differs], [SECRET: same] or [SECRET: missing on live]. A wrong or missing secret is a very common cause, and a bare [REDACTED] on both sides would hide it.
  3. Paste the diff, not the whole files, with the symptom:
Below is a diff between the config for TEST (works) and LIVE (fails with:
[SYMPTOM]). Secrets are replaced by labels that say whether they differ.
Rank the differences by how likely each is to cause the symptom, with a
sentence on why.
- Separate differences that are expected (hostnames, environment names)
  from ones that look accidental.
- Don't suggest changes yet. Tell me what to check first.
- Say what you can't judge without more information.

Diff: [PASTE]

The expected differences are noise that hides the one that isn't. Think of a cache setting, a timeout, a missing feature flag, or a file path that's the wrong case on Linux when it worked fine on Windows.

Then change one thing at a time, in test first if you can, and write down what you changed. Keep the diff and the ranking next to the change record and the next one takes five minutes.

The same idea works for two versions of a document: let the tool find the differences, and use the AI to say what they mean.

← All tips