Case study 04 of 07 · Realtime voice · RAG · Windows automation
AI Receptionist
A real-time voice AI receptionist for live streams, built for TikTok LIVE and usable on YouTube too. Guests join a stream, speak to an animated avatar and get spoken answers about a business, inside hard safety, time and spend limits.
In 20 seconds
- What it is
- Built on OpenAI’s Realtime speech-to-speech API with a RAG knowledge base, a FastAPI control panel and a Windows layer that works with TikTok LIVE Studio even though TikTok offers no guest API.
- My part
- Product owner and director. I defined the requirements and constraints, made the stack and cost decisions, and ran all the real-world testing against TikTok LIVE Studio and OpenAI. Claude Code wrote the code.
- Built with
- Claude Code, a pinned virtual environment, pytest and httpx, headless-browser checks
- AI in the product
- gpt-realtime-2.1-mini holds the spoken conversation. text-embedding-3-small powers the knowledge index.
- Status
- Working system (local). Stack chosen 12 Nov 2026.
Straight answer: It runs on my own Windows PC and has been through real OpenAI conversations and live tests. It is not a launched public product yet, and the auto-disconnect feature has only been tested against fakes.
- 36 ms to hand a ready connection to the next guest verified
- 1.0 s for the AI to stop after a guest starts speaking verified
- 73% of the cost was the AI's voice verified
- 178 test functions in 3,175 lines of tests verified
Overview
A solo practitioner can’t talk to every curious viewer on a live stream. AI Receptionist answers guests one at a time for a few minutes each, by voice, inside strict safety, cost and time limits, without the owner needing to speak. Guests join the stream as a guest, talk to an animated on-screen avatar and are sent to the website. The first business profile is the AI Hypnotherapist receptionist, but the system is built to be universal: one profile per business, so other businesses can be added without code changes.
Not tied to one platform. The voice engine, knowledge base, safety layer, cost controls and avatar don’t depend on TikTok. TikTok LIVE Studio is the first integration, and the same setup can be used on YouTube reported
Who it is for. The business owner, who runs a local admin panel, and guests or viewers on a live stream such as TikTok LIVE or YouTube, who talk to the AI by voice.
What makes it different
- Real-time speech to speech with OpenAI Realtime, not a typed chatbot.
- A fresh AI session per guest, closed afterwards.
- A locked safety layer in code: AI disclosure, no medical claims, crisis handling and a UK health-advertising rule pack that no setting can remove.
- Cost controls built in: spend limits, a time limit, answer caps and a usage log, because voice is billed per response.
- No TikTok API needed, since none exists for guests. It reads LIVE Studio’s own log files and uses Windows UI Automation.
- It runs on a weak PC (an Intel i7-4510U with 2 cores), because nothing heavy runs locally.
How it works
How the pieces connect
- Client
- Service
- Data store
- AI model
- External service
Read this diagram as a list
The boxes
- Admin panel: browser on localhost: start, end, settings
- Avatar window: captured by LIVE Studio
- FastAPI engine: state machine, timers, tools, usage log
- Profiles and knowledge: YAML, Markdown, vector index
- OpenAI Realtime: one WebSocket session per guest
- Audio in and out: sounddevice, WASAPI, 24 kHz
- TikTok LIVE Studio: no API: logs and UI Automation
The connections
- Admin panel → FastAPI engine (control)
- FastAPI engine → Avatar window (state, voice level)
- FastAPI engine ⇄ OpenAI Realtime (Realtime)
- FastAPI engine → Profiles and knowledge
- FastAPI engine ⇄ Audio in and out (PCM)
- FastAPI engine → TikTok LIVE Studio (reads logs, presses Disconnect)
- Audio in and out ⇄ TikTok LIVE Studio (voice via BEACN)
Runs without me during a LIVE
- Auto-start: a listener measures the guest’s input level and starts a chat after about 0.8 s of voice. It is off at every launch, switches itself off after 2 silent auto-chats in a row, and is capped at 30 starts an hour.
- Time and turn limits: 3 minutes with a wrap-up nudge at 2:30, an answer-turn limit, and an end after 10 s of guest silence (‘still there?’ at 5 s).
- Cleanup: closes the session, wipes guest data in memory and writes a usage row with durations, tokens and cost.
- A self-healing avatar window: if it is minimised (which freezes the stream capture) it is restored behind other windows within about 1 s, without taking focus.
- Ready-connection renewal with a 25-minute maximum age and retries with back-off.
- Spend limits checked at every start. I set $10 a day and $100 a month.
Needs a person
- I run the admin panel: Start and End guest, Silence AI now, settings, and approval of knowledge documents. New documents default to off.
- I host and moderate the TikTok LIVE itself.
- I set up the BEACN audio routing and the LIVE Studio configuration.
- Auto-disconnect of the TikTok guest is a switch (Off, Test or On).
Stack
AI models and APIs
- OpenAI Realtimegpt-realtime-2.1-mini, speech to speech
- OpenAI embeddingstext-embedding-3-small
- gpt-transcribeoptional, off by default
- LangChaincore, OpenAI and text splitters, no LangGraph
AI techniques
- Real-time voice (speech to speech)
- Retrieval: embeddings and vector index
- Tool calling3 tools
- Deterministic state machine
- Prompt cachingcache-aware prompt layout
- Safety classifier and guardrailslocked rules in code
- Prompt-injection resistanceguest speech treated as untrusted
- Cost and token trackingper-response log and spend limits
Languages
- Python3.14
- JavaScriptES modules
- GLSLavatar shaders
- HTML and CSS
- PowerShell and batch
Backend and APIs
- FastAPI0.162.5
- uvicorn
- WebSockets
- Pydantic v2
- numpy
Frontend
- Vanilla JavaScript (no framework)admin panel and avatar pages
- three.js r186vendored, avatar picture
Audio and media
- sounddevicePortAudio, WASAPI, 24 kHz
Data and storage
- SQLiteusage log
- YAMLprofiles and settings
- Markdown with YAML headersknowledge documents
- JSON vector index file
Automation
- pywinautoUI Automation of TikTok LIVE Studio
- pywin32 and ctypeswindow handling
- Log-file watchingLIVE Studio connect and leave events
Cloud and infrastructure
- Local Windows 10 PC127.0.0.1, not deployed
- BEACN audio interfacechannel routing
Testing and tooling
- pytest and httpx
- Claude Code
- Headless-browser checks
AI and machine learning
Models and what they do
- gpt-realtime-2.1-miniThe live voice model. Chosen as the cheapest Realtime model after reading pricing and deprecation pages (audio in $10 per million tokens, out $20). The older gpt-realtime-mini was rejected as deprecated.
- text-embedding-3-smallEmbeddings for the knowledge index ($0.02 per million tokens).
- gpt-transcribeOptional guest speech to text. Off by default and not used in the live tests.
Techniques actually used
- One Realtime WebSocket session per guest, closed afterwards, so one guest can never hear about another. It is configured with instructions, voice, server turn detection with interruption allowed, tools and an answer token cap.
- Three tools: lookup_knowledge (RAG), safety_alert (always on, returns an approved crisis message) and finish_conversation (the AI can end the chat).
- A prompt layered per guest from the business profile, behaviour settings, live prices, conversation skills, approved documents, locked rules, a rule pack and my extra rules. A test proves my rules can’t override the locked ones.
- Prompt-caching-aware layout: stable text sits early so repeated responses are billed at the cached rate.
- Length control in seconds of speech, silent tool calls, and a rule that guest speech is untrusted (prompt-injection resistance by instruction, not by a classifier).
- A ready connection is kept open and handed to the next guest, because OpenAI bills per response, not per connection.
- RAG with a LangChain splitter, OpenAI embeddings and a saved in-memory vector store, filtered by switched-on documents and categories with a relevance cut-off. Since 15 Nov the AI is given all switched-on documents directly (about 2,800 tokens for 14 documents) when they fit under 6,000 tokens, with RAG lookups above that. This change has not been tried in a real chat yet.
- A test enforces that the langchain meta-package, which would pull in LangGraph, is not installed.
Not in this project
- LangGraph or multi-agent orchestration
- A planner
- Fine-tuning
- Custom or cloned voice (researched, not built)
A single-agent system. The engine is a deterministic state machine (idle, guest accepted, active, ending, cleanup) with timers. The model only decides whether to call one of its three tools.
Features
- Admin panel with tabs for Live control, Business profiles (create, duplicate, delete to trash), Behaviour, Voice (with generated samples), Avatar, Knowledge, Functions, Usage and costs, and Settings.
- Live control: Start and End guest, Silence AI now, a countdown, an auto-start switch, a ready check, spend today and this month, red safety alerts, an avatar-frozen warning and the LIVE Studio guest status.
- Knowledge manager: documents with on and off switches, categories, rebuilding the index with the cost shown first, and a “Test a question” box.
- Live prices fetched from the business’s public endpoint and cached.
- Avatars: an SVG 2D robot and a 3D-bent photographic “green robot” whose jaw and head are shader-driven by voice level, plus an on-screen label, a services list, a guest countdown and a web-address highlight.
- Safety: AI disclosure, a crisis message copied from a verified GB resource file, an optional crisis keyword net and a UK health rule pack.
- Usage log with per-response token and cost accounting, and audio device tools including a test beep and tolerance for Windows renaming a device.
- Admin security: a scrypt password hash, an HttpOnly cookie, localhost only.
My role
What I did. I conceived the product and its business purpose and chose the stack (Python, LangChain, RAG and OpenAI Realtime, no LangGraph) on 12 Nov 2026 over an earlier Claude Haiku plan. I questioned design decisions, such as why the AI couldn’t be interrupted and whether Haiku would be cheaper. I own all the wording: the assistant’s name, greeting, rules, knowledge documents and on-screen texts. I did all the hands-on testing: LIVE Studio multi-guest behaviour, BEACN audio routing, phone-as-guest tests and YouTube and auto-start tests. My precise observations drove fixes (“the avatar freezes when minimised”, “End needs two clicks”, “the AI hears its own voice”). I set the limits: 3-minute guests, no captions, the voice “ash”, spend limits, and the rule that closing the black window must stop everything. I also run the OpenAI and TikTok accounts.
What Claude Code did. All the code, tests and documentation, plus research into TikTok, OpenAI and LIVE Studio behaviour, diagnosis from logs, and design proposals that I approved or rejected.
Engineering challenges
No TikTok API for guests
TikTok offers no API for guests, yet the system needs to know who is connected.
How it was solvedHide
LIVE Studio’s logs contain UserLinkStateChanged and AllLinkStateChanged lines. A tail-reader tracks who is connected, validated by replaying real logs: it showed a guest who left at 20:24:03 while the AI kept talking until 20:24:47.
Audio routing so the AI hears only the guest
The AI must hear the guest, not itself.
How it was solvedHide
Found by playing soft tones into every BEACN channel and reading every input. LIVE Studio plays guests on the Windows default device, the AI listens on a mix that carries only that channel, and the AI speaks on another channel muted from that mix. A BEACN update broke it: the AI answered itself 27 times in 46 s ($0.108). The fix came from re-measuring and moving to the “Audience Mix” input. The lesson is recorded: run the beep test after any audio change.
Interruptions and echo
Interruptions were disabled at first.
How it was solvedHide
They were enabled after rerouting the AI’s output away from the listening channel. Measured: the AI stopped 1.0 s after a guest started speaking.
Latency
The time from guest speaking to greeting was about 7.5 s in a self-test.
How it was solvedHide
An always-ready OpenAI connection and cached prices cut the handover to 36 ms. The connection itself takes about 4 to 5 s and is now opened while waiting.
Cost
Voice was 73% of the cost, with answers averaging 10 s.
How it was solvedHide
Prompt changes cut a 3-minute guest from about $0.065 to about $0.04, estimated from three real chats.
Windows quirks
Windows hid several traps.
How it was solvedHide
WASAPI fails on worker threads without COM initialisation (fixed). A minimised Chrome window stops painting and no flag helps, so a self-healing window keeper restores it in 0.1 to 0.5 s. A driver update silently renamed devices, so matching is now tolerant, with tests. LIVE Studio runs as administrator, so UI Automation saw no windows until the program also ran elevated. With elevation, 220 controls were visible in a read-only check.
“End needs two clicks”
A UI bug where End needed two clicks.
How it was solvedHide
It was traced to the Live card being rebuilt 4 times a second, which dropped clicks that straddled a rebuild. Reproduced with human-length clicks: 7 of 12 worked before the fix and 12 of 12 after.
An avatar on a weak PC
A 2-core i7-4510U has to animate a talking avatar.
How it was solvedHide
Shader-based head and jaw bending of a single photo, a frame-rate cap and a transparent WebGL canvas. After I rejected the first cartoon version, the moves became eased paths with small angles.
Mistakes handled openly
An unintended real chat started while testing auto-start.
How it was solvedHide
It cost $0, and a rule was added never to switch the real auto-start on during a check. A wrong claim of “I found the cause” was corrected in the project log.
Status
Working, and running locally on my own PC. It is not deployed to a server. Real OpenAI conversations have run: 10 guest records in the usage database, $0.2742 in total and 542 s in all, including test chats. Payment systems: none in this project.
Complete
- Engine and state machine, admin panel, business profiles, knowledge manager and RAG.
- Usage log, spend limits, ready connection, auto-start, avatar pages, the safety layer and audio device handling.
- A platform-independent voice core: it only needs audio in and out, so it can be used on YouTube as well as TikTok reported
Partial or still open
- Auto-disconnect of the TikTok guest: built and tested with fakes. The one real attempt (15 Nov) found no guest Disconnect button, so a diagnostic dump was added.
- The 15 Nov conversation upgrade and the 48 natural avatar movements have not yet been tried in a real chat.
- The administrator-rights launch (start.bat self-elevates because LIVE Studio runs as administrator) is unverified.
Not built, and not claimed
- Welcome-line recordings, mid-guest spend-limit enforcement and a cost-per-LIVE-hour report.
- Ending the AI chat automatically when the TikTok guest leaves, a self-echo detector and a Browser-volume warning.
- Other channels (website, phone) and custom or cloned voice.
- A git repository and any server deployment. Payments: none in this project.
Evidence
- 36 ms to hand a ready connection to the next guest Measured in a real chat. Opening the connection itself takes about 4 to 5 s, done while waiting.verified
- 1.0 s for the AI to stop after a guest starts speaking Self-test on 12 Nov 2026.verified
- 73% of the cost was the AI's voice Measured, with answers averaging 10 s. Prompt changes then cut a 3-minute guest from about $0.065 to about $0.04, estimated from three real chats.verified
- 178 test functions in 3,175 lines of tests More cases through parametrisation. 35 Python modules, 5,056 lines.verified
- $0.2742 total spent across 10 logged guest chats Including test chats: 542 s in all, $0.0274 on average.verified
- $0.000057 to build the knowledge index (27 passages) verified
- to the cent usage balance matched OpenAI's dashboard On 12 Nov 2026.reported
Code worth reading
app/engine/engine.pyState machine, timers and the ready connection.app/engine/instructions.pyThe layered prompt with locked rules.app/knowledge/index.py and documents.pyRAG with filters and the inline-documents fallback.app/tiktok/studio_log.py, studio_ui.py and guest_disconnect.pyIntegration without an API: log reading and UI Automation.app/avatar/keeper.py and packs/android/avatar_image.jsThe self-healing window and the avatar shader.app/audio/devices.pyDevice handling that survives driver renames.
What I can't verify, so I don't claim
- That auto-disconnect works on a real guest. It has only run against fakes.
- Any public deployment or user numbers. There are none.
verified checked against the code, tests, logs or the live system. reported from my own notes or hands-on use. unverified could not be checked, so it is not claimed.