Build Your Own 3D Smart Home App With an AI Agent

What it does

• A 3D model of the house (three.js) that shows every light, fan, TV and the thermostat live, in the right room.
• Tap a light in the model to toggle it, drag to dim it.
• An Ask bar: plain-English commands ("movie mode in the basement", "is anything on upstairs?"), handled on the phone for simple requests and by an AI model for the rest.
• Scenes (Morning, Night) with undo, a home-screen widget, and a Google TV remote over Wi-Fi.

Requirements

• Your devices already in Google Home. The app doesn't replace Google Home; it uses Google's Home APIs to control what's already set up there.
• A Google account and a free Google Cloud project. The Home APIs need an OAuth consent screen and an Android OAuth client registered with your app's package name and signing certificate (SHA-1). Leave the consent screen in "Testing" and add the Google accounts of everyone in your house as test users. No Play Store listing needed.
• Android. The Home APIs are Android-only (iOS has its own SDK). There's no web or server API for your lights, so the phone is where control happens.
• A floor plan, plus photos or a walkthrough video. Room sizes from a plan or tape measure, and then either a photo of each room or one slow video walking through the house. With a video, the agent pulls still frames out of it and uses those as its photos. Either is enough for a believable model.
• Optional: an AI API key (Anthropic, OpenAI or Gemini) for the smarter Ask bar. Simple commands work without one.

Apps and tools

Hardware

• An Android phone on Android 10 or newer. A full 3D house is heavy for a phone, so expect some performance work.
• A Google Home hub (Nest Hub, Google TV Streamer or similar) if you have Matter devices; it's what lets Google control them locally.
• A computer to build on. Any recent Mac, Windows or Linux machine that can run Android Studio. 16 GB of RAM is comfortable.
• The devices themselves. Anything that works with Google Home: smart bulbs, switches, plugs, fans, a thermostat, Google TVs.

How the pieces talk

The phone owns the devices; the 3D page just draws them. They swap small JSON messages through a WebView message channel:

// Phone -> page: the current state of every device
{ "type": "state", "devices": [
  { "id": "light-1", "name": "Desk Lamp", "room": "Office", "kind": "light", "on": true, "level": 70, "color": "#ffcf87" },
  { "id": "fan-1",   "name": "Ceiling Fan", "room": "Bedroom", "kind": "fan", "on": true, "fanPct": 50 },
  { "id": "thermo",  "name": "Thermostat", "room": "Hall", "kind": "thermostat", "mode": "Heat", "running": "Heat", "tempF": 68 }
]}

// Page -> phone: the user tapped or dragged something in the model
{ "type": "toggle", "id": "light-1", "on": false }
{ "type": "level",  "id": "light-1", "level": 30 }

On the page, each device gets a glow that follows that state:

HomeBridge.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type !== 'state') return;
  for (const d of msg.devices) {
    const glow = glows.get(d.id);          // a THREE.PointLight placed in the device's room
    if (!glow) continue;
    glow.visible = d.on && d.online !== false;
    glow.intensity = (d.level ?? 100) / 100 * MAX_INTENSITY;
    if (d.color) glow.color.set(d.color);
  }
};

The Ask bar gives the AI model a short list of tools instead of letting it do anything it wants. The model proposes calls; the app shows the plan and asks before big changes:

{
  "name": "set_light",
  "description": "Turn lights on or off, set brightness, or set a color. Give device_ids, or a room to change every light in it.",
  "input_schema": {
    "type": "object",
    "properties": {
      "device_ids": { "type": "array", "items": { "type": "string" } },
      "room":       { "type": "string" },
      "on":         { "type": "boolean" },
      "brightness": { "type": "integer", "minimum": 1, "maximum": 100 },
      "color":      { "type": "string" }
    }
  }
}

The others follow the same pattern: list_rooms, list_devices, set_fan, set_thermostat, tv_control, run_scene. The system prompt is one paragraph plus a line per device (id | name | room | kind | state), so a cheap, fast model handles it well.

Two rules worth copying: the API key is encrypted with a key in the Android Keystore and never leaves the phone except to its own provider, and simple commands ("kitchen off") are parsed on the phone so they're instant and free.

Prompts to hand your agent

Run these one at a time, in order. Check each step on your phone before moving on.

1. Set up the project

Fork the Google Home APIs Android sample app (github.com/google-home/google-home-api-sample-app-android) into a new repo. Rename the package to com.yourname.home and the app to "My Home". Walk me through creating the Google Cloud project, the OAuth consent screen in Testing mode, and the Android OAuth client with this app's package name and debug and release SHA-1s. Then build a signed APK I can install, and keep the signing key safe and out of git.

2. Build the 3D house

Here's my floor plan and either a folder of room photos or a walkthrough video of the house [attach]. If it's a video, pull still frames from it (ffmpeg is fine), pick the clearest shot of each room and label them by room. Then build a three.js static site of the house: walls, floors, doors, windows, stairs and basic furniture at real dimensions, with an orbit camera, a whole-house view and a tap-to-focus view per room. No build step; three.js from a CDN via an import map. Take Playwright screenshots of each room next to my photos or frames and fix what doesn't match.

3. Connect the model to the real devices

Bundle the 3D site into the Android app's assets and show it in a WebView on the home screen. Add a JSON message bridge: the app sends {type:'state', devices:[...]} whenever a device changes, and the page sends toggle and level messages back when I tap or drag a light. Place each Google Home device in its room in the model (ask me where each one goes). Lights glow with their brightness and color, fans spin, TVs shimmer when playing.

4. Add the Ask bar

Add an Ask bar with a mic. Parse simple commands on the phone ("kitchen off", "bedroom 30%"). For anything else, call the AI provider I pick in Settings (Anthropic, OpenAI or Gemini) with tools: list_rooms, list_devices, set_light, set_fan, set_thermostat, tv_control, run_scene. Store the API key encrypted with the Android Keystore, never log it, and show me the plan before changing more than a few devices. Write unit tests against canned API responses, not the network.

5. Make it fast

Profile the app on my phone. The 3D view should hold 60 fps while moving and draw nothing when idle. The device list should scroll smoothly. Measure before and after each change.

6. Share it with the family

Add my family's Google accounts as test users, add a "Change home" and "Use another Google account" option in Settings, and set up a private download link for the APK so their phones can get updates.

What I'd tell you before starting

• Start from the sample app. Google's sign-in and permission flow is the hardest part; the sample has it working.
• Expect to tune performance. A full 3D house in a phone WebView is heavy. Idle rendering, fewer lights and smaller textures got mine smooth.
• Keep it private. Don't put your address in the app name, the model, or screenshots you share. The OAuth app stays in Testing, so only people you add can sign in.
• Let the agent check its own work. Screenshots and tests after every change beat me eyeballing every build.

The takeaway: you don't need to be an Android developer. You need your devices in Google Home, a floor plan, and an AI coding agent, and what you get is pretty incredible: a custom smart home app designed around your own house.