Get a VRM Avatar

VRM is an open standard for 3D humanoid avatars. You can download free models from VRoid Hub. Here's how:

🧑‍🎤

Recommended: Free Sample Model

This character is free to download with "Allow" characterization. Click the link, then press the "Download" button on the model page to get the .vrm file.

Download this VRM →

📥

How to Download from VRoid Hub

1. Visit a model page → 2. Click "Download" (agree to terms) → 3. Save the .vrm file → 4. Drag it into the viewport in Step 4 below.

Browse more free VRMs →

2

Set Up the Project

Add the dependencies via a CDN import map — no bundler needed. This loads Three.js for 3D rendering, @pixiv/three-vrm for VRM support, ONNX Runtime for neural inference, and AnimaSync V1 for lip sync.

HTML — CDN Import Map

<script type="importmap">
{ "imports": {
  "three":            "https://cdn.jsdelivr.net/npm/three@0.179.1/build/three.module.js",
  "three/addons/":    "https://cdn.jsdelivr.net/npm/three@0.179.1/examples/jsm/",
  "@pixiv/three-vrm": "https://cdn.jsdelivr.net/npm/@pixiv/three-vrm@3.4.5/...module.min.js"
}}
</script>

<!-- ONNX Runtime (required by AnimaSync) -->
<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.17.0/dist/ort.min.js"></script>

Or install via npm: npm install three @pixiv/three-vrm onnxruntime-web @goodganglabs/lipsync-wasm-v1

3

Initialize AnimaSync

Create a LipSyncWasmWrapper, then call init(). This loads the Rust/WASM module, validates the license (30-day free trial with no signup), and decrypts the ONNX model.

JavaScript

const CDN = 'https://cdn.jsdelivr.net/npm/@goodganglabs/lipsync-wasm-v1@0.4.4';

const { LipSyncWasmWrapper } = await import(`${CDN}/lipsync-wasm-wrapper.js`);
const lipsync = new LipSyncWasmWrapper({ wasmPath: `${CDN}/lipsync_wasm_v1.js` });

await lipsync.init({
  onProgress: (stage, pct) => console.log(`${stage}: ${pct}%`)
});
// Stages: wasm → license → decrypt → onnx

4

Load VRM into Three.js

Create a scene with a camera, orbit controls, and lighting. Then load the VRM using GLTFLoader with the VRMLoaderPlugin. AnimaSync also provides embedded VRMA bone animation clips for idle breathing and speaking gestures.

JavaScript — Scene + VRM Loading

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
import { VRMLoaderPlugin } from '@pixiv/three-vrm';

// Scene
const renderer = new THREE.WebGLRenderer({ canvas, antialias: true });
const scene    = new THREE.Scene();
const camera   = new THREE.PerspectiveCamera(30, w/h, 0.1, 100);
camera.position.set(0, 1.25, 1.5);

// Load VRM
const loader = new GLTFLoader();
loader.register(p => new VRMLoaderPlugin(p));

const gltf = await loader.loadAsync(vrmUrl);
const vrm  = gltf.userData.vrm;
scene.add(vrm.scene);
vrm.scene.rotation.y = Math.PI;

5

Apply Lip Sync

Process an audio file to get blendshape frames, then apply each frame's 52 ARKit values to the VRM expressionManager inside the render loop. AnimaSync outputs at 30 fps.

JavaScript — Process + Render Loop

const ARKIT_52 = ['browDownLeft', 'browDownRight', 'browInnerUp', /* ...49 more */];

// Process audio file
const result = await lipsync.processFile(audioFile);

// Queue all frames
const queue = [];
for (let i = 0; i < result.frame_count; i++)
  queue.push(lipsync.getFrame(result, i));

// Apply inside render loop at 30fps
function render() {
  requestAnimationFrame(render);
  if (queue.length > 0) {
    const frame = queue.shift();
    for (let i = 0; i < 52; i++)
      vrm.expressionManager.setValue(ARKIT_52[i], frame[i]);
  }
  vrm.update(dt);
  renderer.render(scene, camera);
}

6

Add Real-time Microphone

For real-time lip sync, capture microphone audio with an AudioWorklet at 16 kHz, then feed 100 ms chunks to processAudioChunk(). Pipeline latency is ~130-300 ms.

JavaScript — Mic Streaming

const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const ctx    = new AudioContext({ sampleRate: 16000 });
const source = ctx.createMediaStreamSource(stream);

// AudioWorklet captures 1600 samples (100ms at 16kHz)
worklet.port.onmessage = async (e) => {
  const result = await lipsync.processAudioChunk(e.data);
  if (result)
    for (let i = 0; i < result.frame_count; i++)
      queue.push(lipsync.getFrame(result, i));
};

Build Your Own AI Talking Avatar