System Overview
This Unity-based VR system supports configuring and interacting with groups of embodied conversational agents. It combines speech, hand gestures, gaze, simulated touch, and spatial layout control in one real-time interaction loop for multi-agent group communication.
Architecture
VR Client
A Unity VR scene presents embodied agents, captures user speech and hand interactions, manages agent placement, and plays synchronized speech, gestures, gaze, and facial animation.
Backend and AI Integration
A Python/FastAPI backend communicates with the Unity/C# client over WebSockets. The interaction pipeline integrates automatic speech recognition (ASR), LLM orchestration, text-to-speech synthesis (TTS), and generated humanoid motion. My work combines client and backend development with multimodal interaction and multi-agent coordination.
Input Router
Speech, gesture, touch, gaze, and spatial manipulation events are routed into a shared interaction state so verbal and non-verbal signals can affect the same group conversation.
Group Orchestrator
A turn-state layer handles speaker selection, interruption, buffered non-verbal inputs, and routing between the user and multiple agents.
Per-Agent State
Each agent maintains personality and emotion state that conditions response content and delivery properties such as tone, tempo, and loudness.
Embodied Output
Generated responses are converted into speech, facial animation, full-body gesture, gaze behavior, and touch-triggered reactions for playback in VR.
Core Features
- Multimodal input: Users can communicate through speech, hand gestures, simulated physical contact, gaze, and spatial repositioning instead of relying on voice-only commands.
- Multi-agent group coordination: The system supports multiple embodied agents in the same conversation, including turn routing, next-speaker cues, interruption, and group-level state updates.
- Stateful agent behavior: Agents update personality and emotion state over time, allowing each character to respond with distinct content and paralinguistic delivery.
- Embodied response pipeline: Speech output is synchronized with generated full-body gestures, facial animation, gaze shifts, and contact reactions so agent responses are visible as well as audible.
Interaction Pipeline
- The user speaks, gestures, touches an agent, looks toward an agent, or changes the group layout in VR.
- The VR client captures the event and sends the relevant verbal or non-verbal signal into the interaction state.
- The group orchestrator selects target agents, handles interruption or buffering, and updates turn state.
- Per-agent state updates condition the generated response, including content, emotion, tone, speed, and loudness.
- The output pipeline plays synchronized speech, facial animation, gesture, gaze, and contact reactions back in the VR scene.
Technical and Design Challenges
Coordinating group conversation
The system separates group-level turn orchestration from per-agent state so speaker selection, interruption, and agent-specific behavior can be handled together without collapsing the interaction into one-on-one dialogue.
Combining verbal and non-verbal signals
Speech, gestures, touch, gaze, and spatial movement are routed through a shared interaction state, allowing non-verbal events to influence timing and response behavior.
Keeping agent responses embodied
The output pipeline synchronizes synthesized speech with full-body motion, facial animation, gaze, and touch reactions so the response remains spatial and visible in VR.