How to Scale LiveKit-Based AI Avatars While Keeping Lip-Sync Under 200ms
How to scale LiveKit AI avatars with streaming TTS, audio-driven sync, and latency tactics to keep lip-sync under 200ms.
Introduction
Scaling a live avatar is not the same problem as scaling a static video stream. Once you put an AI agent behind a talking face, your latency budget becomes shared across ASR, LLM inference, TTS, video synthesis, transport, and playback. If any one of those stages drifts, the face starts talking over the user, the mouth loses alignment, or the interaction feels “sticky” even if the text response is correct.
This post is about keeping lip-sync under 200ms end-to-end for realtime avatar interactions. By the end, you should have a clear mental model for where the latency comes from, what to optimize first, and how to structure your agent and media pipeline so the avatar stays responsive under load.
What “under 200ms” actually means
People often say “lip-sync under 200ms” as if it were a single number, but in practice you care about offset and jitter separately:
Offset: how far the visual mouth motion trails the audio.
Jitter: how much that offset varies from frame to frame or utterance to utterance.
A constant 120ms offset is usually acceptable. A variable 60ms-to-250ms offset is much worse, because viewers perceive it as instability even if the average is lower.
For realtime avatars, the practical latency budget is usually consumed in this order:
Speech recognition or turn detection
LLM token generation
TTS first-audio latency
Avatar mouth/video generation
Transport and client playout
If you want the face to feel synchronized, you cannot treat the avatar as an independent “render after speech” step. It has to be coupled to the same streaming speech timeline that the audio uses.
Keep the media path streaming, not batch-oriented
The easiest way to blow your latency budget is to wait for the full sentence before sending anything to TTS or the avatar renderer. That creates unnecessary head-of-line blocking. For conversational agents, the correct model is incremental:
Detect the end of user speech quickly.
Start LLM generation immediately.
Stream partial text to TTS or a speech renderer as soon as you have stable tokens.
Begin avatar synthesis from the same audio timeline, not from a separate “finished text” event.
In practice, you also want short utterance chunking. If your agent produces long monologues, split them into smaller semantic chunks so the first audio frame and the first mouth movement happen quickly. That improves perceived responsiveness more than shaving a few milliseconds off the LLM.
Control the sources of jitter
At scale, average latency is not the hardest problem. Tail latency is. A few common causes:
Cold starts: model or worker startup delays on the first request.
Queue buildup: too many concurrent sessions routed to one worker or region.
Cross-region hops: audio, agent, and avatar services not colocated.
Buffer bloat: over-aggressive client or server buffering to “stabilize” media.
The fix is not to add more buffering. It is to make the system deterministic and capacity-aware:
Pin a session to a region/worker so audio packets and synthesis requests do not bounce around.
Warm capacity for your expected concurrency, especially at peak times.
Use bounded queues and fail fast when overloaded rather than letting sessions accumulate delay silently.
Measure p95 and p99 separately for ASR, LLM, TTS, and video generation.
If you only watch “average response time,” you will miss the exact conditions that make lip-sync feel off. Users notice the worst 1%.
Prefer audio-driven sync over text-driven rendering
For avatars, the audio timeline is the source of truth. If the renderer advances mouth shapes from text alone, it will often be wrong during pauses, rewrites, and token revisions. Audio-driven sync gives you a stable clock:
When the first audio frame is ready, the avatar can begin speaking immediately.
Mouth shapes can be aligned to phoneme timing or viseme timing derived from audio.
If the agent pauses or corrects itself, the video follows the actual spoken stream instead of an earlier draft.
This also means you should be careful about mixing transport layers. If audio goes through one path and video through another with different buffering behavior, the client will see drift. Keep the streaming relationship explicit and observable.
Operational tactics that keep lip-sync stable
Once the media path is streaming, the remaining work is operational. A few patterns matter a lot:
Use a single timebase per session. Do not let the avatar renderer, TTS, and transport layer each invent their own notion of “now.”
Prefer small, frequent updates over large buffered batches. This reduces visible step changes in mouth motion.
Instrument first-audio latency and first-frame latency separately. They are not the same metric.
Budget for browser decode time. Even if your server is fast, the client still needs to decode and schedule playback.
Backpressure the agent when downstream media generation lags. Otherwise the text stream gets ahead of the avatar and sync suffers.
One practical rule: if your system cannot maintain sub-200ms offset under load, reduce scope before adding more parallelism. For example, a shorter answer with slightly less verbose phrasing often feels much better than a richer answer that starts late.
A minimal LiveKit integration pattern
If you are building a voice agent on LiveKit, the most important design choice is to attach the avatar to the agent pipeline instead of bolting it on afterward. The LiveKit Agents plugin for Protoface does exactly that: it turns the agent’s spoken output into a synchronized talking face in the same realtime session.
In a typical setup, you initialize the agent, configure the avatar/video service, and let the plugin consume the same speech stream that drives audio. The exact configuration fields vary by stack version, but the shape is straightforward:
If you are using Pipecat instead of LiveKit directly, the integration is similarly stream-oriented; the plugin wraps the video side so your pipeline can remain focused on agent logic. The main point is that video should follow the same realtime event stream as speech, not a separate postprocessing job.
For examples and plugin-specific setup, the repo and docs are the right place to start: repository examples and documentation.
How Protoface fits into the architecture
Protoface is useful here because it gives you a developer-facing avatar layer that can be driven from realtime agent output instead of forcing you to build and operate the lip-sync stack yourself. In the LiveKit path, the plugin is the cleanest integration point: your agent keeps ownership of conversation state, while the avatar service handles synchronized talking video.
For teams that need to provision avatars or sessions dynamically, the REST API is the other important surface. You can create and manage avatars and realtime sessions server-side with standard bearer authentication:
That pattern matters operationally because you keep API keys out of the browser and can control session lifetime, rate limits, and allocation centrally. Exact request fields and response shapes live in the docs.
Common mistakes that break sync at scale
There are a few recurring failure modes I see in realtime avatar systems:
Starting TTS too late: waiting for a full completion before synthesizing speech.
Over-buffering the client: a large playout buffer hides jitter briefly, then creates a visible lag spike.
Using separate queues for audio and video: the two paths drift under load.
Ignoring concurrency limits: one noisy tenant can starve other sessions if you do not isolate capacity.
Failing to load test with realistic turn-taking: short user interruptions and barge-in behavior are where sync bugs show up first.
The easiest way to validate your system is to test with real conversational patterns: interruptions, backchannels, quick clarifications, and overlapping speech. If the avatar can survive that, it will probably handle polite one-turn QA just fine.
Conclusion
Keeping lip-sync under 200ms is mostly an architecture problem, not a rendering trick. Stream early, keep a single realtime timeline, colocate the media path, and watch tail latency instead of averages. If you do those things, the avatar feels attached to the conversation instead of lagging behind it.
If you want to implement this in a LiveKit-based agent, start with the plugin integration, then instrument first-audio and first-frame latency separately. For API-driven provisioning and session management, check the docs at docs.protoface.com. If you want a working reference, the quickstarts in the GitHub org are the fastest way to see the pieces wired together.