Convonet Voice AI Productivity System

The Convonet Voice AI Productivity System is an enterprise-grade platform running on Google Cloud Run as five FastAPI microservices. It combines multi-LLM AI (Claude, Gemini, OpenAI), WebSocket-based voice (no LiveKit on GCP), domain-specific agents (Productivity, Mortgage, Healthcare, Restaurant/Hanok), and intelligent call center integration. Voice flows through the voice-gateway WebSocket at /webrtc/ws: PIN authentication (PostgreSQL or env), batch STT/TTS (Deepgram), and HTTP to agent-llm for LangGraph and MCP tools.

The system enables teams to manage todos, mortgage workflows, healthcare intakes, and Hanok restaurant reservations via voice and web dashboards, with seamless transfer to human agents (Twilio → FusionPBX). Agent Monitor shows tool calls and voice response timing. Services are exposed under v2.convonetai.com (and optionally hanok.convonetai.com) with path-based routing/custom-domain mapping; Redis and PostgreSQL (e.g. Render) back session and application data.

Used by agent-llm-service (e.g. Render.com PostgreSQL). Connection via DB_URI; optional RENDER_POSTGRES_HOST_SUFFIX for Render host normalization.

Full schema and migrations are in convonet/models/ and migrations/.

MCP provides a standardized way for AI agents to interact with external tools and services. On GCP, agent-llm-service uses MCP to expose domain tools: database operations (todos, reminders, calendar), team management, mortgage/healthcare, Hanok reservation operations (via hanok-table-service), call transfer to FusionPBX, and Google Calendar. STT/TTS are handled by voice-gateway (Deepgram), not MCP.

{
  "mcpServers": {
    "db": {
      "command": "python",
      "args": ["./convonet/mcps/local_servers/db_todo.py"],
      "transport": "stdio",
      "env": {
        "DB_URI": "${DB_URI}",
        "GOOGLE_OAUTH2_TOKEN_B64": "${GOOGLE_OAUTH2_TOKEN_B64}",
        "GOOGLE_CLIENT_ID": "${GOOGLE_CLIENT_ID}",
        "GOOGLE_CLIENT_SECRET": "${GOOGLE_CLIENT_SECRET}"
      }
    }
  }
}

The LangGraph implementation provides intelligent tool calling capabilities with dynamic tool selection and error handling. The agent automatically chooses appropriate tools based on user intent and maintains conversation context for seamless interactions.

Tool Calls Flow Diagram

Tool Calls Flow: LangGraph implementation showing dynamic tool selection and intelligent orchestration of MCP tools.

Paths are served via path-based routing on v2.convonetai.com, with optional custom-domain mapping for hanok-table-service (hanok.convonetai.com).

Browser voice: Use /voice_assistant → WebSocket /webrtc/ws (voice-gateway-service). PIN auth (PostgreSQL or env), Deepgram STT/TTS, greeting after login, “please wait” + elapsed timer during processing, barge-in (Start stops playback and starts recording).

Phone (Twilio): Twilio webhooks target voice-gateway-service at POST /twilio/process_audio. STT/TTS are Deepgram (not Twilio/Polly). On transfer intent, agent-llm returns transfer_marker; voice-gateway returns TwiML <Dial><Sip> to FusionPBX (e.g. extension 2001). Config: VOICE_GATEWAY_PUBLIC_URL, FUSIONPBX_SIP_DOMAIN, TRANSFER_TIMEOUT. Callback /twilio/transfer_callback handles dial status.

Transfer context (conversation_history, user_id) is cached in Redis for the call-center UI. See docs/CLOUD_RUN_ENV.md for Twilio/FusionPBX env vars.

Validate the GCP deployment via /voice_assistant (WebSocket voice, PIN, greeting, barge-in), /agent-monitor (tool calls and voice timing), /call-center (SIP dashboard and customer popup from Redis), and /mortgage_dashboard. For phone flows, configure Twilio webhooks to VOICE_GATEWAY_PUBLIC_URL/twilio/process_audio and test transfer to FusionPBX.