What to ask your agency for, in one page

You do not need to become a marketing analyst to know whether your marketing reporting is any good. You need nine questions and the willingness to ask for the answers in writing.

None of these are traps. A good agency will answer them in one sitting and several will be pleased you asked, because the questions distinguish good work from work that merely looks good, and that distinction usually favours them.

On the numbers

1. What is your definition of a lead?

Specifically: is it deduplicated to distinct people, and does that deduplication run across channels?

Somebody who calls twice is one lead. Somebody who fills in a form and then phones about the same job is one lead. If the reporting counts events, your lead volume is roughly a third larger than reality, and every derived figure inherits that inflation.

2. What is the denominator in cost per lead?

All leads, or only leads attributable to paid?

Total spend divided by total leads blends your organic performance into your paid efficiency. It is the single most common way marketing reporting misleads without anybody lying, and the error can be very large. On one real account, dividing by all leads understated true cost per paid lead by 61%.

3. Which attribution model, and what lookback window?

If nobody knows, the answer is last touch, because that is the default nearly everywhere.

Last touch credits whatever somebody clicked immediately before converting, which for any considered purchase is very often branded search. It systematically flatters demand harvesting and starves demand creation.

4. Are inbound sales solicitations excluded, and how many were excluded this period?

Both halves matter. Excluding them is table stakes. Telling you the count is what makes the filter auditable rather than something you have to trust.

5. Was ad spend blended or prorated by actual run dates?

A campaign that ran for nine days is not one third of a month. If spend is smoothed into a monthly average, every date-range comparison you make has had the variable you were comparing averaged out of it.

On what is missing

6. What did we not measure this period, and why?

The answer should be specific. A data source failed, a platform changed an export, a tag was broken for six days.

The answer you do not want is “nothing.” Something is always partially missing, and a report that never acknowledges a gap is a report that fills gaps quietly.

7. Which of these numbers are measured and which are estimated?

Every report contains both. The problem is when they are formatted identically, because then a modelled figure inherits the credibility of a counted one.

Ask for the estimated ones to be marked. If nobody can tell you which are which, that is the answer.

On the market

8. Who are we losing deals to, and what did they do this quarter?

This is the question most reporting cannot answer at all, and it is the one your competitors’ behaviour actually turns on.

Your analytics can only describe people who found you. It has nothing to say about the buyer who compared you to two other companies and chose one of them, which is most of the decisions being made about you.

If the answer is anecdotal, that is not a criticism of your agency. It is a description of what conventional marketing reporting covers. It is also the largest blind spot you have.

9. If our marketing were underperforming, how would this report show it?

The best question on the list, and the one that tells you most about the reporting rather than the marketing.

A report that can only render good news is not a measurement instrument. Ask what a bad quarter looks like in this format, and whether any of the metrics currently shown can go down in a way that is visible.

If every chart is designed so that up is the only direction with a story, you are looking at a presentation.

How to use the answers

Ask in writing and ask once. This is not a quarterly ritual and it is not a performance review.

What you are looking for is whether the definitions exist. An agency that can answer all nine crisply has thought about measurement, which usually correlates with thinking carefully about everything else. An agency that has to go and find out is not doing anything wrong, and now they will have written the definitions down, which is a genuine improvement.

The answer that should concern you is a defensive one. These questions have factual answers. Treating them as an accusation is itself informative.

What to do with the definitions once you have them

Write them into your reporting agreement, and require that a change to a definition is called out when it happens.

That single clause removes most of the risk. Definitions drift, usually gradually and usually in the flattering direction, and drift is invisible unless somebody has to announce it.

A report where the definitions are fixed and published, applied identically in a bad month and a good one, is worth more than a more sophisticated report that can be quietly re-cut.

Our own definitions are published in full, including the awkward ones, for exactly this reason. If a definition is doing quiet work in somebody’s favour, it should be possible to catch it.