Vapi vs. Retell vs. building your own voice AI agent

What the choice actually is

A voice AI agent is not one thing. It is a pipeline: audio comes off a phone call or a browser session, a speech-to-text model turns it into text, a language model decides what to say and which tools to call, a text-to-speech model turns that back into audio, and the audio goes back down the same call — continuously, in both directions, while a human is waiting.

Every part of that has to be streamed rather than batched, because a person hears any pause longer than a beat as the system being broken. Around the pipeline sits a second layer of problems nobody puts on the whiteboard: deciding when the caller has stopped talking, handling them talking over the agent, keeping state coherent across turns, and connecting all of it to real telephony with DTMF, transfers, and voicemail detection.

Vapi and Retell are platforms that own that orchestration for you. You bring prompts, a tool schema, and a phone number; they run the pipeline. The alternative is to build the orchestration yourself, directly on top of a CPaaS provider for the telephony and the speech and model APIs for the intelligence.

That is the real fork. Not "which vendor," but whether the pipeline is something you operate or something you rent.

Vapi and Retell differ from each other in the usual ways competing products do — the shape of the configuration model, how workflows are expressed, which providers are swappable at which stage. Those differences are best settled by building the same small agent twice, an afternoon each, against your real use case. What they do not differ on is the thing that decides the architecture: both abstract the pipeline, and both trade control for speed. That trade is what this page is about.

The comparison

	Voice AI platform (Vapi, Retell)	Building your own pipeline
Time to a working prototype	Hours	Weeks
Who owns latency tuning	The platform, within the knobs it exposes	You, at every stage
Model choice	Whatever the platform supports, when it supports it	Anything with a streaming API
Infrastructure to operate	None	Real-time audio, stateful and long-lived
Cost shape	Per-minute, including the platform's margin	Fixed build cost, then underlying API costs only
Cheaper at low volume	Yes, clearly	No
Cheaper at high volume	No	Yes, and the gap widens
Debuggability when quality drops	Whatever the platform surfaces	Full trace across every stage
Exposure to pricing changes	Yours to absorb	Only the underlying providers'
Who you call when it breaks at 2am	Support	You

The right-hand column is not free and the last row is the honest reminder of that. Owning the pipeline means owning the pager.

What a platform buys you

A working prototype in a day rather than a fortnight. This is the honest headline benefit and it should not be undersold. Getting a voice agent that answers a real phone number and holds a coherent conversation is a same-afternoon exercise on a platform. Built from scratch, the same milestone is a couple of weeks of work before you learn anything at all about whether your use case makes sense.

Latency work you did not have to do. Turn detection, streaming boundaries between stages, warm connections to model providers, filler-word handling while a tool call resolves — this is unglamorous, fiddly engineering, and a platform has already invested in it. A platform's out-of-the-box responsiveness is typically better than a first-attempt custom pipeline, and often better than a second attempt too.

Less infrastructure to own. Real-time audio is stateful, long-lived, and does not fit the request-response shapes most teams' infrastructure is built around. A platform means no WebSocket fleet to scale, no media servers, no audio-buffer memory leaks discovered on a Friday, no on-call rotation for a system that fails audibly in front of customers.

A shorter path through the boring parts. Recordings, transcripts, call logs, basic analytics, a dashboard non-engineers can look at. All buildable, none interesting, all included.

If your question is "does a voice agent work for this workflow at all," a platform answers it faster and cheaper than any custom build. That is not a small thing — it is often the only question worth answering in month one.

What you give up

Latency control past a point. You can tune what the platform exposes. You cannot tune what it does not. When you have set every knob available and the agent still feels a half-beat slow, you are out of moves — you cannot move a stage of the pipeline closer to the caller, swap in a faster model at one specific stage, change how endpointing decides a turn ended, or keep a connection warm the way you want. If the difference between "good" and "uncannily good" for your product lives in that last stretch of tuning, the abstraction is now the ceiling. The mechanics of what you would tune are covered in the voice agent latency guide.

Dependency on someone else's roadmap. The features you need arrive when they arrive. A capability that would take you a day to build yourself might sit in a vendor's backlog indefinitely, and you have no lever beyond asking.

Pricing risk you do not control. Platform pricing in this category has moved repeatedly as the underlying model costs have moved. Sometimes that works in your favour. But your unit economics are downstream of a pricing page you do not own, and if you have built a business model on today's rate, a change to it is a change to your margin — arriving with whatever notice the vendor decides to give.

Customization that fights the grain. Every platform has an implicit model of what a voice agent is. Inside that model, everything is easy. Outside it, things get hard quickly. Anything unusual — a custom audio processing step, a non-standard telephony flow, an agent that hands off mid-call in a particular way, a compliance requirement about where audio is processed — either has a supported path or does not, and when it does not, the workaround is usually worse than the from-scratch version would have been.

An extra layer between you and the failure. When quality degrades, the platform sits between you and every underlying provider. Was it the speech model, the LLM, the TTS, the carrier, or the orchestration? You get whatever observability the platform surfaces, and debugging is a support ticket rather than a trace you can read yourself.

The cost curve

This is the part most comparisons handle badly, usually by quoting per-minute prices that are stale within a quarter. The useful thing to understand is the shape rather than the numbers.

A platform charges you a rate that must cover the underlying speech, model, and telephony costs plus its own margin plus the engineering you did not do. A custom build pays the underlying costs directly, with no margin, but adds a fixed cost up front — the build — and an ongoing cost of maintaining it.

So the curve has two regions:

At low volume, the platform wins, usually by a lot. When you are running hundreds of calls a month, the per-minute margin you are paying is trivial compared to the cost of the engineering weeks you skipped. Buying is straightforwardly cheaper.

At meaningful volume, the custom build wins. The margin scales linearly with your minutes; the build cost does not scale at all. There is a crossover, and past it the gap widens every month.

Three things move that crossover point:

• Call volume and length. More minutes, earlier crossover. A high-volume agent handling long calls reaches it far sooner than a low-volume one.
• How much of the build you actually need. If you only need one narrow call flow rather than a general orchestration layer, the build is much cheaper than the generic estimate suggests.
• Whether you would have needed the expertise anyway. If your team already runs real-time infrastructure, the build's true marginal cost is much lower than it looks on paper.

Two things get missed in these comparisons. First, a custom build is not zero-maintenance — models change, providers deprecate endpoints, and someone has to own the pipeline forever. Second, migrating later is a real project, not a flip of a switch. That means the decision has a timing component: the cheapest path is often a platform now and a build later, but only if you have kept your business logic on your own side of the boundary so that "later" is a migration rather than a rewrite. What actually drives the numbers is broken down in the voice agent development cost guide.

When to build your own

The voice agent is the product. If voice is what customers are paying for, the pipeline is your core competency and outsourcing it means outsourcing the thing you compete on. Every rival on the same platform has the same ceiling as you. The differentiated version of the product lives below the abstraction line.

You need control the platform does not expose. Latency past the exposed knobs. A specific model at a specific stage. Custom audio handling. Region-pinned processing for a data residency requirement. Deterministic behaviour around interruption. If your requirement list has items no platform config screen has a field for, you have already answered the question.

Volume is past the crossover, or clearly heading there. If you can see the month where the platform margin exceeds what the build would have cost, and the product is proven, building is the cheaper path from that month onward.

You need the infrastructure anyway. If you are already running real-time media for other reasons, the marginal cost of adding voice AI orchestration to it is much smaller than a greenfield estimate implies. LastWorker is a build in this shape — the voice layer sits inside a system whose behaviour, integrations, and call handling are the product, not a bolt-on feature.

Your compliance posture requires it. Adding a vendor between you and the underlying providers means one more processor holding call audio and transcripts. Sometimes that is fine and the vendor's certifications are enough. Sometimes an auditor or a customer's security review wants a shorter chain of custody, and the fastest way to give them one is to not have the extra layer.

What "build your own" actually involves

It is worth being concrete about the work, because "build the pipeline yourself" is said casually and is not a casual amount of engineering.

You need a telephony leg streaming media to your service over a WebSocket rather than handled by the carrier's IVR primitives. You need a session layer that holds a call's state for its full duration, survives a reconnect, and cleans up when someone hangs up mid-sentence. You need streaming speech-to-text with partial results and a turn-detection policy on top of it, because the transcript alone will not reliably tell you whether a caller finished a thought or paused to breathe. You need the model response streaming before the turn is fully confirmed, and cancellable the instant the caller interrupts. You need text-to-speech chunked small enough that first audio arrives fast, with a barge-in path that stops playback immediately. And you need tool calls that resolve without dead air, which usually means saying something plausible while the request to your backend is in flight.

Then the operational layer: recording and transcript storage under whatever retention policy your compliance posture demands, per-call cost accounting, an evaluation harness so you can tell whether a prompt change helped, and enough tracing to see which stage cost the extra 400 milliseconds. None of it is exotic, and a team that has done it before moves quickly — but it is the reason platforms exist, and estimating the build without pricing that list is how people end up wrong by a wide margin.

When to use a platform

You are still validating. Nobody should build orchestration infrastructure to find out whether customers will talk to a machine about this workflow. Ship the platform version, put it in front of real callers, learn what the conversation actually needs to do. That knowledge is portable; the orchestration you would have built is not.

Voice is a feature, not the product. If it is one channel among several — a booking line beside your app, a callback flow, an after-hours fallback — then deep specialization in real-time audio is attention spent in the wrong place. Rent the pipeline and put the engineering into the thing customers actually chose you for.

The team does not have the expertise and should not acquire it. Real-time audio is a specialty. Learning it well takes months, and if the roadmap does not have other work that would reuse that knowledge, the learning is a sunk cost. A platform is a legitimate way to skip a discipline you do not need to own.

Volume is low and likely to stay low. An internal agent handling modest call volume will never approach the crossover. Building it would be an expensive way to save nothing.

How to decide

Four questions, in order:

1. Is the voice agent your product, or a feature of it? If it is the product, expect to build eventually, and plan the boundary now even if you start on a platform.
2. Do you have a hard requirement no platform exposes? A specific latency target you have already failed to hit, a data residency rule, a non-standard call flow. One genuine item here decides it.
3. Where is your volume relative to the crossover? Model it with your own call volume and length against current platform pricing. If you are not close, the platform is cheaper and it is not a close call.
4. Have you proven the use case? If not, none of the above matters yet. Ship the fastest working version and come back with real call data.

The most common mistake is not picking wrong — it is picking permanently. Starting on a platform is almost always correct, and it stays correct until one of the first three questions changes its answer. What makes that transition cheap is a boundary you drew early: business logic, tool implementations, and conversation state living in your own backend, with the platform holding only the pipeline. Teams that do that migrate in weeks. Teams that let platform-shaped abstractions leak through their whole product rewrite instead.

If you want a second opinion on which side of that line your product sits, the AI agents service page covers how these builds are scoped in practice.