An agency sends you three references. You call them, they say nice things, you sign. Two months later you learn the reference clients were the three happiest customers the agency has, hand-picked and lightly coached, and the reason the agency was comfortable giving them to you is that they would never say anything sharp. A reference list an agency chooses is a marketing asset, not a due-diligence one. If you want a reference check that changes your decision, you have to run it differently.
This matters more with agencies than with individual hires, because the money is bigger and the failure modes are quieter. A bad first engineer is visible in weeks. A bad agency can send confident status updates for a quarter while very little ships, and you will not know until a deadline you cannot move arrives. The reference call is one of the few places you can catch that before you have spent the money.
Find your own references, not just theirs
Use the agency's list, but treat it as the floor, not the ceiling. The references that matter are the ones the agency did not choose.
Finding them is easier than it sounds. Look at the agency's public case studies and portfolio, then find the person who would have owned that project on the client side, usually a founder, a head of product, or the first engineer who was there when the work happened. Their name is often on LinkedIn, sometimes in the case study itself. Reach out for a short call and say plainly why you are asking. Most people will take twenty minutes to help a stranger avoid a mistake they already made.
Pay special attention to clients whose engagement clearly ended a while ago. A reference who is still mid-project has every incentive to stay diplomatic, because they still need the agency's goodwill. Someone who finished a year ago, and especially someone who has since moved the work in-house or to another vendor, has the distance to be honest about what they would do differently. Those are the calls that are worth an hour of your time.
The questions that get real answers
The difference between a useless reference call and a decisive one is the questions. Yes-or-no questions get you yes. Open questions about specifics get you the story.
Skip "were you happy with them?" Nobody says no to that after the fact. Ask instead:
- "Walk me through what the team looked like on day one versus month three. Were the people who pitched you the people who built it?" This is how you catch the bait-and-switch where senior staff pitch and juniors build, which is one of the most common and most expensive agency patterns.
- "What did the first serious disagreement look like, and how did they handle it?" Every real engagement has one. How the agency behaved when the client pushed back tells you more than any success story.
- "When something slipped, how did you find out? Did they tell you early, or did you discover it?" Agencies that surface bad news late are the ones that blow up your launch.
- "What did the final bill look like against the original number?" You are listening for the gap between the quote and the reality, the same gap that turns a fixed bid into a 60 percent overrun.
- "If you were starting this project again, would you hire them, and what would you do differently in the contract?" The second half of that question is where people tell you the truth they were too polite to lead with.
Keep the tone conversational and let silences sit. The most useful sentence in a reference call almost always comes after a pause, when the person decides to tell you the thing they were not sure they should.
Reading the answers
A reference call is not scored on how positive it is. It is scored on how specific it is. A reference who gives you concrete detail, names, timelines, a real disagreement and a real resolution, is telling you the truth whether the detail is flattering or not. A reference who only offers warm generalities is either coached, disengaged, or hiding something, and any of those means you learned nothing.
Watch for the things people soften rather than say outright. "They were a little slow to communicate sometimes" from a polite founder often means the agency went dark for weeks. "We had some scope conversations" often means a bruising fight about change orders. When you hear a softened phrase, ask a follow-up that makes it concrete: how slow, how often, what did the scope conversation cost you.
One genuinely bad data point from a reference you found yourself is worth more than three glowing ones from the list you were handed. Weight them accordingly.
When you cannot read the technical answers
Some of what a reference tells you is business, and you can judge it yourself: communication, honesty, billing surprises, whether they hit dates. Some of it is technical, and here a non-technical founder is at a disadvantage, because a reference might say "the code was fine" and have no way of knowing whether that is true. If the reference was also non-technical, their verdict on code quality is worth very little.
That is the gap worth closing with an outside opinion. If the engagement is large, it is worth having someone technical join one reference call, or review the agency's proposed approach, so the reference's business signals and a real read on the technical risk land in the same place. A short book a call to structure the reference questions and sit in on the calls that matter is cheap relative to a six-figure build with the wrong shop.
FAQ
How many references should I actually check?
Two or three the agency gives you, and at least one or two you find yourself. The self-sourced ones carry most of the weight, so if you can only do a few, prioritize finding your own over working through the agency's list.
What if the agency will not give references at all?
That is itself a strong signal. A reputable agency has happy clients who will vouch for it and expects to be asked. Reluctance, or references who turn out to be friends and partners rather than paying clients, is a reason to slow down, not push through.
Is a case study on their site good enough?
No. A case study is written by the agency and approved for publication, so it is closer to an ad than a reference. Use it to find the person who lived the project, then call that person and ask what the case study left out.
Should I check references before or after I get the proposal?
After you have a proposal but before you sign anything. You want to know what to ask about, and a specific proposal gives you concrete things to probe, such as whether the team, the timeline, and the price in the proposal match what past clients actually experienced.