Most founders evaluate a development agency the way they would evaluate a car they cannot test-drive: a sales call, a portfolio of finished work, a confident quote, and a leap of faith. Then they sign a six- or twelve-month contract and discover what the team is actually like to work with on someone else's clock. The fix is simple and almost nobody uses it. Before the big contract, run a small paid pilot.
Why the sales call tells you nothing
A portfolio shows you a team's best work after months of iteration, polished for display. The sales call shows you their best communicator, who may never touch your project. The quote shows you a number, not whether the number is real. None of it tells you the thing you actually need to know: what is this team like to work with when the requirements are messy and the deadline is close?
That information only exists on the other side of a commitment. A pilot lets you buy it cheaply.
A four-to-six week paid engagement against one clear, shippable deliverable surfaces everything the sales process hides:
- How they communicate when it's real. Do they ask sharp questions, or nod and disappear? Do they surface problems early, or let them arrive as surprises?
- Whether they push back. A team that says yes to every request is not being agreeable; it is not engaging. The pilot shows you whether anyone is thinking past the ticket.
- Whether the work survives contact with reality. A demo can be staged. A shipped feature you use yourself, on your own data, cannot. The pilot is the first time you see the difference.
- How they handle a change mid-stream. Throw one realistic scope change at them and watch. You will learn more in that one moment than in the entire sales cycle.
How to structure a pilot that actually tests them
A pilot only works if it is a real slice of the work, not a toy. Get the structure right.
Pick a deliverable that ships
Choose something small enough to finish in a month but real enough to go to production: a single user-facing feature, a working integration, a contained piece of the product. "Build a prototype" is too soft; prototypes hide the discipline that production demands. You want to see whether they can ship something a real user touches.
Define success as an outcome, not a task list
Before the pilot starts, write down what "done" means in terms a non-engineer can verify: a user can do X, the page loads in under Y, the integration handles Z. If you define the pilot as a list of technical activities, you will get the activities and learn nothing about whether they can deliver an outcome. This is the same discipline that separates motion from progress once the real engagement begins, the same lens you would apply to the agency invoice nobody reads.
Pay properly and keep the exit clean
Pay a fair rate. Underpaying tells the team this is not serious, and you will get B-team effort. But keep the pilot legally and practically separable from the main contract: clear deliverable, fixed window, no automatic rollover. The entire value of a pilot is that walking away costs you a month, not a year.
The decision the pilot makes for you
At the end of the pilot you will know things no reference call could have told you. You will know if they communicate like partners or vendors. You will know if their code holds up when you use it. You will know if they pushed back when you were about to make a mistake, or just billed the hours. And you will have spent a small fraction of the annual contract to learn it.
If the question underneath all this is whether you should hire an agency at all versus your first senior engineer, the pilot helps there too. Running the same outcome-based test against a contractor tells you which model your stage actually needs. When you want a second opinion on what the pilot revealed before you commit, that is a short conversation worth having.
FAQ
How long should a pilot be?
Four to six weeks is the sweet spot. Short enough that a bad fit costs you a month, long enough to include a real deliverable, a change of direction, and the moment the initial polish wears off and you see how the team actually works.
Will good agencies refuse to do a pilot?
The good ones rarely do. They are confident in their work and would rather start small than oversell. Be wary of a team that insists on a long commitment up front and will not prove itself on a contained piece first. That insistence is information.
What should the pilot cost?
Pay your normal rate for the scope; do not ask for a discount. A discounted pilot invites discounted effort, and the point is to see their real work. The cost is small next to a year-long contract signed on faith.
What if the pilot goes well but the full quote is high?
Then you are negotiating from strength. You have seen the work, they have seen you are serious, and both sides want to continue. That is a far better position than negotiating a big contract against a portfolio and a promise.