English | 中文

Provides integration of any AI agent (Gemini CLI, Claude Code, Codex CLI, etc.) with the Unity editor using Agent Client Protocol (ACP). AI agents can directly inspect, query, and modify your Unity project — scenes, components, assets, and more.

Note: Unity Agent Client now includes the ACP icon in the window title bar and native Markdown rendering for agent responses.

These recordings show the project in action:

• Install and use walkthrough — install the tool and get connected in the editor
• Generate a 3D scene with an agent — use the agent to create a 3D model and place it in the scene
• Switch scene lighting from noon to another look — adjust the scene lighting atmosphere live

Unity Agent Client is an editor extension that uses the Agent Client Protocol (ACP) to connect AI agents to the Unity editor. Instead of just chatting, agents can operate on your project through a comprehensive set of built-in MCP tools.

You:   "Set the Player's movement speed to 10"
Agent: 1. scene_get_hierarchy       → finds Player object
       2. scene_get_component_data  → reads PlayerMovement component
       3. scene_modify_component    → sets Speed = 10
       4. "Done. Player speed is now 10."

• Any AI agent — supports all ACP-compatible agents (Gemini CLI, Claude Code, Codex CLI, opencode, Goose, etc.)
• 60+ built-in MCP tools — grouped into meta-tools for efficient agent interaction
• Meta-tool architecture — agent sees ~13 tools instead of 60+, reducing context overhead by ~80%
• Multi-session support — session dropdown in toolbar lets you switch between conversations; ＋ button creates new sessions
• Elicitation forms — agents can request structured input via native UI controls (toggles, sliders, dropdowns, multi-select, color picker, Vector3, object picker); URL-mode opens the browser for OAuth flows
• Skill recipes — pre-defined step-by-step guides for common tasks (14 built-in)
• AI 3D asset generation — generate 3D models from text/image via Meshy AI
• Extensible tool system — add custom tools with a single class implementing IMcpTool
• Dual MCP transport — auto-selects HTTP (no Node.js) or stdio proxy based on agent capabilities
• Drag & drop context — attach assets to prompts for richer context
• Auto-reconnect — recovers from agent crashes with exponential backoff; auto-reconnects on config change

The demo recordings above highlight the install flow, agent-driven 3D generation, and lighting changes in a real Unity scene.

┌─────────────────────────────────────────────────────────────┐
│  Unity Editor                                               │
│                                                             │
│  ┌─ Open Source (GitHub) ─────────────────────────────────┐ │
│  │  AgentWindow ──ACP(stdio)──> Agent Process             │ │
│  │      │                        (Gemini/Claude)          │ │
│  │      │                             │                   │ │
│  │      ├── Elicitation Panel        │ MCP               │ │
│  │      └── Permission UI            ▼                   │ │
│  │  BuiltinMcpServer  <──── HTTP or stdio proxy          │ │
│  │      │                                                 │ │
│  │  80+ Tool Implementations (Mcp*Tools.cs)               │ │
│  └────────────────────────────────────────────────────────┘ │
│              │ references                                   │
│  ┌─ UnityAgentClient.Core.dll (Closed Source) ────────────┐ │
│  │  AgentClientProtocol  — ACP protocol types & transport │ │
│  │  McpToolRegistry      — IMcpTool, permissions, dispatch│ │
│  │  MetaToolRouter       — category routing, batch exec   │ │
│  │  SkillRegistry        — YAML skill recipes             │ │
│  │  SessionStore         — session persistence            │ │
│  └────────────────────────────────────────────────────────┘ │
│              │                                              │
│          Unity Editor API                                   │
└─────────────────────────────────────────────────────────────┘

This project uses a hybrid open/closed source model:

• Open source (this repository) — UI layer, tool implementations, settings, MCP server bootstrap
• Closed source (UnityAgentClient.Core) — Core engine distributed as a NuGet DLL, containing ACP protocol implementation, tool registry, meta-tool routing, and skill system

To add custom tools, implement the IMcpTool interface (exported by the Core DLL) and register in BuiltinMcpServer.cs.

The agent sees 13 tools instead of 62, organized by category with a fallback for direct access:

Agent sees 13 tools:
┌──────────────────────────────────────────────────────────────┐
│ unity_scene     — Scene & GameObject operations (22 actions) │
│ unity_editor    — Editor control & project info (12 actions) │
│ unity_asset     — Asset management (14 actions)              │
│ unity_material  — Material & rendering (5 actions)           │
│ unity_lighting  — Lighting & environment (4 actions)         │
│ unity_animation — Animation & navigation (6 actions)         │
│ unity_ui        — UI Canvas operations (6 actions)           │
│ unity_generate  — AI 3D asset generation (5 actions)         │
│ unity_spatial   — Spatial queries & 3D perception (6 actions)│
│ unity_particle  — Particle system editing (8 actions)        │
│ unity_terrain   — Terrain sculpting & painting (5 actions)   │
│ unity_skill     — Skill recipes for common tasks (14 skills) │
│ unity_tool      — Direct access fallback (any tool by name)  │
│ unity_batch     — Cross-category batch operations            │
└──────────────────────────────────────────────────────────────┘

Example:
  unity_scene(action="modify_component", gameObjectPath="Player", ...)
  unity_asset(action="find_references", assetPath="Assets/Materials/Wood.mat")
  unity_generate(action="text_to_3d", prompt="a medieval wooden chair")
  unity_tool(name="list")  → shows all 62 available tool names

See the lighting demo recording for a live example of scene lighting changes.

Generate game-ready 3D models directly inside Unity using Meshy AI. The agent handles the entire pipeline: prompt → API call → poll → download → import → place in scene.

See the 3D generation demo recording for a complete end-to-end example.

Setup: Set the MESHY_API_KEY environment variable in Project Settings > Unity Agent Client > Environment Variables, or save it to UserSettings/UnityAgentClient/meshy_api_key.txt.

Example — end-to-end generation flow:

You:   "Generate a low-poly medieval chair and place it at position 3,0,5"
Agent: 1. unity_generate(action="text_to_3d",
          prompt="a medieval wooden chair with ornate carvings",
          artStyle="low-poly", outputFormat="glb")
       2. (polls Meshy API for ~30-120 seconds until model is ready)
       3. unity_generate(action="instantiate_prefab",
          prefabPath="Assets/Generated/Meshy/a_medieval_wooden_chair_143022.glb",
          position="3,0,5")
       4. "Done. Medieval chair generated and placed at (3, 0, 5)."

Use Cases:

For when the agent knows the exact tool name or can't find the right category:

unity_tool(name="list")                           → list all 51 tools
unity_tool(name="physics_get_settings")           → call any tool directly
unity_tool(name="config_tags_and_layers")          → bypasses meta-tool routing

Skills are pre-defined step-by-step guides for common Unity tasks. The agent reads a skill recipe, then executes the steps using the other meta-tools.

unity_skill(action="list")                        → list all available skills
unity_skill(action="search", query="button")      → search by keyword
unity_skill(action="get", name="create_ui_button") → get full recipe

Built-in Skills:

Adding Custom Skills: Place .yaml files in UserSettings/UnityAgentClient/Skills/ in your project. See BuiltinSkills.cs for the format.

Implement IMcpTool and register in BuiltinMcpServer.RegisterTools():

public class MyTool : IMcpTool
{
    public string Name => "my_custom_tool";
    public string Description => "Does something useful";
    public bool RequiresMainThread => true; // true if using Unity API
    public JsonElement InputSchema => JsonDocument.Parse(@"{ ""type"": ""object"", ""properties"": {} }").RootElement;

    public McpToolResult Execute(JsonElement args)
    {
        // Your Unity API calls here
        return McpToolResult.Success("Result text");
    }
}

For the full step-by-step setup guide with troubleshooting, see docs/SETUP.md.

Quick start:

Unity Agent Client requires Unity 2021.3 or later. Node.js is optional (only needed for agents that don't support HTTP MCP transport).

git clone https://github.com/3DLabInstruments/UnityACPClient.git

Open the cloned folder in Unity Hub. All DLLs are bundled — no extra steps needed.

To add to an existing project, copy Assets/UnityAgentClient/ into your project's Assets/ folder (the DLLs are bundled inside Editor/Plugins/).

Open Project Settings > Unity Agent Client and fill in the settings according to the AI agent you want to use.

Open Window > Unity Agent Client > AI Agent to automatically connect to the session.

• Enter a prompt in the field and press Send (or Enter) to submit. Shift+Enter inserts a newline; the input field auto-grows as you type. Press Esc to cancel a running request.
• Drag and drop assets into the window to attach them as context. Attached items appear as compact chips with an icon; click a chip to ping the asset in the Project window, or press × to remove it.
• A colored dot in the toolbar reflects connection status (green = connected, yellow = connecting, red = failed); hover it for details.
• A "thinking…" indicator is shown while the agent is responding.
• Session management: A dropdown in the toolbar shows all sessions for the current agent. Click ＋ to create a new session. Sessions persist across domain reloads (up to 20 per agent config). There is no "Disconnect" button — sessions coexist and the agent process stays alive.
• Elicitation: When an agent needs structured input (e.g. choosing a strategy, configuring build settings), an inline form appears with native controls — dropdowns, toggles, sliders, multi-select checkboxes, and text fields with validation. Submit, Decline, or Cancel (Esc) to respond.
• If the connection fails, a Retry button and an Open Settings button appear with the error message. If it's still connecting, a Cancel button lets you abort and reconfigure.
• When executing tools, the agent may request permission (whether permission is requested depends on the agent's settings).
• If supported by the agent in use, you can switch modes or models in the toolbar. Mode names are shown as human-readable labels (e.g. "Agent", "Plan") rather than raw identifiers.
• Changing agent settings (command/arguments) while connected automatically triggers a reconnect — no need to close and reopen the window.

Unity Agent Client DOES NOT recommend using AI agents for coding within the editor. Due to Unity's constraints, editing C# scripts triggers a Domain Reload which restarts the agent process. The session list and last active session are preserved and automatically restored after reconnection.

Unity Agent Client is best suited for:

• Scene setup & iteration — "Add a directional light and set it to warm color"
• AI 3D asset generation — "Generate a low-poly medieval chair and place it at position 3,0,5"
• Project analysis — "Which assets reference the Wood material?"
• Configuration review — "Show me the import settings for all textures in Assets/UI"
• Quick prototyping — "Create a Cube named Enemy at position 5,0,3 with a Rigidbody"
• Debugging assistance — "Show me the recent console errors"

See ROADMAP.md for planned features and future direction.

This library is provided under the MIT LICENSE.