Measured Forward, or It Is Not Evidence

There are no client results on this website. No testimonials, no case studies, no percentages, no names. Prospects notice, and occasionally ask, and it is a fair question to ask of anyone selling anything.

The short answer is that FGN has not produced a client outcome it has permission to publish. The longer answer is a standard I now work to, which came out of getting a number badly wrong in my own business and believing it for a while.

The mistake

I wanted to know my reply rate on outbound messages. I had a table of logged conversations, so I queried it, divided replies by contacts, and got 17%. That is a good number. I reported it to myself as a good number and started reasoning from it.

It was nonsense. The table I queried holds hand-logged conversations. A conversation gets logged because somebody replied. I had computed a reply rate from a table that exists as a consequence of replies, which is a bit like measuring the survival rate of a hospital by surveying people in the waiting room.

The real figure, pulled from the send log rather than the conversation log, was 2.1% across 767 sends at the time. Today the same query against the same table returns 3.78% across 899 sends. Not 17%.

The problem was not the arithmetic. The arithmetic was fine. The problem was that I had answered the question "what does the data say" when the question was "what is actually true", and those come apart whenever the data was generated by the thing you are trying to measure.

What makes this failure mode dangerous is that a wrong number does not feel like uncertainty. It feels like knowledge. So it gets built on, and everything above it inherits the flaw invisibly.

The four rules

These are on the Practice Operations Review page as the standard the work is judged by. This is where they came from.

Measured forward. The measurement is defined before the work starts, not after it. The baseline gets agreed in writing in the first week and the fields are fixed before anything runs. Going back through the data afterwards looking for a number that supports the argument is not evidence gathering, and it is how a great deal of reported marketing performance is produced. Not by lying. By searching.

Reported honestly. Some measurements move between runs. Page performance scores are the obvious example: run the same test three times on the same unchanged page and you can get three different numbers. When that happens the honest response is to say the measure is unstable and quote the stable one instead, not to publish the run that flatters. If I ever hand you a number without telling you it moved, I have chosen the best of three and hidden it.

Input with output. A result produced from a file of twelve hundred past contacts is not a result someone with forty contacts can expect. The ratio might transfer. The absolute number never does. Publishing the output without the input is the most common way a true statistic is used to mislead, and it sets the next person up to fail against a benchmark that was never available to them.

Permission first. Nothing gets claimed on a client's behalf without their permission. That is the direct reason there are no names on this site, and it is not going to change when the results arrive. A client's business performance is their information, not marketing material I happen to have access to.

Why I am publishing the absence

There is a commercial argument for staying quiet about this. Every competitor page has a wall of numbers on it, and mine does not, and a visitor comparing the two pages quickly will read mine as the less proven option.

I would rather take that hit, for a reason that is about this specific audience.

If you are an established practitioner, you have been marketed to by people quoting results you cannot check, about clients you cannot contact, using methods nobody will describe. You have probably bought from one of them. The category has a credibility problem it earned.

Against that background, an unattributed number does not build confidence. It is one more claim in a genre of claims. Whereas a description of exactly how a number would be produced, and what would disqualify it, is something you can actually evaluate.

So: when there are results here, they will arrive with the input alongside the output, a baseline that was agreed before the work started, and the client's permission on the record. Until then this page stays empty, and I will keep publishing my own numbers, including the 3.78% and including the eight-fold error that preceded it.

A number you cannot audit is not proof. It is decoration.