Knee Surgery VR Training System

System Overview

This Unity-based VR training system supports a specialized knee surgery procedure through a guided virtual operating room. It was developed with orthopedic experts to help trainees practice procedural sequencing, instrument recognition, and decision-making in a controlled environment before working in clinical settings.

Architecture

VR Client

The trainee enters a virtual operation room with surgical instruments, an anatomy model, contextual panels, and step-based guidance. The scene is built in Unity and uses customized 3D medical assets modeled in Blender.

Training Guidance Layer

The system coordinates procedure videos, automatic pauses, instrument highlighting, and contextual explanations so trainees can connect each surgical action with the relevant clinical knowledge.

Interaction Layer

Users can manually interact with instruments and the anatomy model while receiving visual and haptic cues. The training flow tracks the current learning step and exposes relevant guidance at each point.

Evaluation Context

The prototype was developed with two orthopedic experts and evaluated by 6 surgeons. This was a prototype evaluation; it does not establish improved clinical outcomes.

Core Features

• Guided procedure playback: Training videos pause automatically at 8 critical learning points so the system can surface targeted explanations during the procedure.
• Interactive surgical scene: A virtual operation room lets trainees inspect instruments, anatomy, and procedure context in VR rather than only watching flat video.
• Custom medical assets: The system includes 12+ customized 3D medical instrument models built in Blender for the specialized procedure.
• Multimodal cues: Highlighted instruments, visual panels, and haptic cues guide attention during important procedural steps.

Interaction Pipeline

1. The trainee starts a procedure step in the virtual operating room.
2. The system plays the relevant surgical training video segment.
3. Playback pauses at a critical learning point.
4. Contextual panels summarize the knowledge point and highlight relevant instruments or anatomy.
5. The trainee inspects and interacts with the virtual instruments before continuing to the next step.

Technical and Design Challenges

Translating clinical workflow into VR

The system needed to preserve the logic of a specialized surgical procedure while making it learnable in an interactive VR scene. The training flow breaks the procedure into explicit learning points and pairs each one with the relevant instruments, anatomy, and explanation.

Modeling specialized instruments

Commercial assets were not sufficient for the procedure, so the project used 12+ customized 3D medical instrument models to support realistic practice and recognition.

Combining passive video and active practice

The prototype connects video instruction with manual interaction by pausing at key moments and prompting trainees to inspect or manipulate the relevant virtual objects before moving forward.