> ## Documentation Index
> Fetch the complete documentation index at: https://hanabiaiinc-agents-response-wait-settings.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Web SDK
> Voice sessions in the browser with @fishaudio/agent-client: options, events, transcripts, text input, audio controls, client tools, and errors
`@fishaudio/agent-client` runs a live voice conversation with your agent from any web page: open the microphone, stream audio both ways, and react to typed events for transcripts, agent state, and tool calls. The SDK handles the realtime transport (WebRTC) internally.
Using React? [`@fishaudio/agent-react`](/agents/deploy/react-sdk) wraps this SDK in hooks and a provider.
## Install
```bash npm theme={null}
npm install @fishaudio/agent-client
```
```bash pnpm theme={null}
pnpm add @fishaudio/agent-client
```
```bash yarn theme={null}
yarn add @fishaudio/agent-client
```
## Start a session
`AgentSession.start()` creates the session, connects, and opens the microphone in one call. Authenticate one of two ways:
* **`agentId`**: for [public agents](/agents/deploy/public-agents). The SDK creates the session directly from the browser; no backend needed.
* **`sessionToken`**: for private agents. Your backend calls `POST /v1/agent/sessions` with your API key and hands the JSON response to the browser; pass it through unchanged. See [Authenticated sessions](/agents/deploy/authenticated-sessions).
```javascript Public agent theme={null}
import { AgentSession } from "@fishaudio/agent-client";
const session = await AgentSession.start({
agentId: "YOUR_AGENT_ID",
});
```
```javascript Private agent theme={null}
import { AgentSession } from "@fishaudio/agent-client";
// Your backend calls POST /v1/agent/sessions and returns the
// response body. Pass it to the SDK as-is.
const resp = await fetch("/api/voice-session", { method: "POST" });
const sessionToken = await resp.json();
const session = await AgentSession.start({ sessionToken });
```
`start()` resolves once the realtime connection is up and rejects with a [`FishAgentError`](#errors) if session creation, the microphone permission, or the connection fails. Call it from a user gesture (a click handler): browsers only grant the microphone and audio playback inside one.
### Options
| Option | Type | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `agentId` / `sessionToken` | `string` / `SessionToken` | Exactly one is required: a public agent id, or the JSON your backend received from `POST /v1/agent/sessions` |
| `serverUrl` | `string` | API base used in `agentId` mode. Default `https://api.fish.audio` |
| `clientTools` | `Record` | Handlers for [client tools](/agents/build/client-tools) declared on the agent |
| `clientToolTimeoutMs` | `number` | Per-handler timeout for client tools, in milliseconds. Default `15000` |
| `overrides` | `SessionOverrides` | Per-session config [overrides](/agents/deploy/authenticated-sessions#overrides) (`agentId` mode only; with `sessionToken`, your backend sends them when creating the session). Keyless sessions accept only `voice_id` and `language`; the prompt-shaping fields are rejected with `400` |
| `dynamicVariables` | `Record` | Values for `{{placeholders}}` in the agent config; see [Dynamic variables](/agents/build/dynamic-variables) (`agentId` mode only) |
| `language` | `SessionLanguage`, the code of one of the [52 supported languages](/agents/build/voice-language#speaking-language) | Shorthand |
| for `overrides.language` (`agentId` mode only). Omit it to use the agent's configured [speaking language](/agents/build/voice-language#speaking-language); any other value is rejected with `422` | | |
| `toolEvents` | `boolean` | Whether tool lifecycle events reach this client (default `true`; `agentId` mode only, with `sessionToken` your backend sets `tool_events`) |
| `timezone` | `string` | IANA timezone for the agent's sense of local time: the top of the [resolution order](/agents/build/time-timezone), overriding the agent's configured timezone. When omitted, the SDK still sends the browser timezone as a lower-priority hint (`client_timezone`), which applies only if the agent has no timezone configured |
| `worldContext` | `boolean` | Whether the agent knows the current date and time (default `true`; `agentId` mode only, with `sessionToken` your backend sets `world_context`) |
| `endUserId` | `string` | Your identifier for the end user, stored on the session and echoed in [webhooks](/agents/monitor/webhooks) (`agentId` mode only; with `sessionToken` your backend sets `end_user_id`) |
| `metadata` | `object` | Your own key-values, stored on the session record and returned verbatim (`agentId` mode only; with `sessionToken` your backend sets `metadata`) |
| `microphone` | `boolean` | Capture the microphone on start (default `true`). `false` joins muted with no permission prompt, for text-first UIs; the first `setMicMuted(false)` captures it, so call that from a user gesture |
| `audio` | `{ inputDeviceId?, outputDeviceId? }` | Pick specific microphone and output devices. `start()` rejects with `device_change_failed` if the output device cannot be selected |
| `wakeLock` | `boolean` | Hold a screen wake lock while the session is live, so long calls survive the phone trying to sleep (default `true`). A denied or unsupported wake lock is silent |
| `callbacks` | `Partial` | Shorthand for `.on()`: each `onXxx` key subscribes the `xxx` [event](#events), so `onUserTranscript` subscribes `userTranscript`. From 0.2.1, a key that is not a known event name throws a `TypeError` |
```javascript Callbacks shorthand theme={null}
const session = await AgentSession.start({
sessionToken,
callbacks: {
onUserTranscript: ({ text, final }) => renderUserBubble(text, final),
onAgentResponseDelta: ({ text }) => renderAgentBubble(text),
onModeChange: mode => setOrbState(mode),
onDisconnect: ({ reason }) => showCallEnded(reason),
onError: error => console.error(error.code),
},
});
```
## Session lifecycle
A session moves through a fixed state machine, surfaced by the `statusChange` event and the `session.status` property:
```text States theme={null}
connecting → connected ⇄ reconnecting → ended(reason)
└───────────(failure)───────────→ ended
```
`start()` resolves once the realtime connection is up; the agent itself joins moments later. If it has not joined within 15 seconds, the session emits a `connection_failed` error and ends with reason `connection_lost` instead of idling on a dead call.
When the session ends, `disconnect` fires with a reason (also available as `session.endReason`):
| `EndReason` | Meaning |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user_hangup` | You called `session.end()`, the user left the page, or your backend [ended the session](/agents/deploy/authenticated-sessions#ending-sessions-from-your-backend) |
| `agent_hangup` | The agent ended the call: the hang-up [system tool](/agents/build/system-tools), a workflow end node, or an idle hangup |
| `conversation_timeout` | The session reached the agent's [maximum duration](/agents/build/configuration#call-duration) |
| `escalated` | The call was transferred to a human |
| `connection_lost` | The connection dropped and could not be recovered, the agent never joined, or the call ended before the server could say why |
Every reason except `connection_lost` comes from the server: it sends the protocol's [`session.ended` event](/agents/deploy/protocol#session-ended) just before tearing the call down, and the values match `end_reason` on the [session record](/agents/monitor/conversation-history#status-and-end-reason), so your client and your backend agree about how a session ended. This needs `@fishaudio/agent-client` 0.3.0 or later; older clients infer the reason from the disconnect and report every server-side end as `agent_hangup`.
Brief network drops don't end the session: the SDK moves to `reconnecting` and back to `connected` automatically, reusing the same session. It never creates a new session, so you never need a new token mid-call.
Events are not replayed after a reconnect. Anything emitted while you were in
`reconnecting` (transcript updates, tool events, errors) is dropped, not
resent. Design your UI to tolerate gaps: tool `toolCallCompleted` /
`toolCallFailed` events repeat the tool name and source, so a terminal event
still renders even if you missed `toolCallStarted`.
The session also exposes `sessionId` (use it to look up the [conversation record](/agents/monitor/conversation-history) later), `status`, `mode`, `isSpeaking`, `micMuted`, and `endReason` as properties.
## Events
The session is a typed event emitter: subscribe with `session.on(event, handler)`, remove with `off`, or use `once`.
| Event | Payload | Fires when |
| -------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------- |
| `connect` | `{ sessionId }` | The session is connected and live |
| `disconnect` | `{ reason: EndReason }` | The session has ended |
| `statusChange` | `SessionStatus` | Status transitions (`connecting` / `connected` / `reconnecting` / `ended`) |
| `modeChange` | `AgentMode` | The agent switches between `listening`, `thinking`, and `speaking` |
| `userTranscript` | `{ segmentId, text, final }` | The user's speech is transcribed; interim updates replace the whole segment |
| `agentResponseDelta` | `{ segmentId, delta, text }` | The agent speaks: `delta` is the new text, `text` the segment so far |
| `agentResponse` | `{ segmentId, text }` | An agent segment is finalized |
| `message` | `{ role: "user" \| "agent", text }` | A finalized message from either side, a ready-made chat feed |
| `toolCallStarted` | `{ callId, toolName, source, input, inputTruncated }` | A tool call begins; `input` is a JSON string (truncated at 4 KB) |
| `toolCallCompleted` | `{ callId, toolName, source, output, outputTruncated }` | A tool call succeeds |
| `toolCallFailed` | `{ callId, toolName, source, error }` | A tool call fails |
| `error` | `FishAgentError` | A session or tool error occurs; see [Errors](#errors) |
```javascript Subscribe to events theme={null}
session.on("userTranscript", ({ segmentId, text, final }) => {
upsertBubble("user", segmentId, text, final);
});
session.on("agentResponseDelta", ({ segmentId, text }) => {
upsertBubble("agent", segmentId, text, false);
});
session.on("modeChange", mode => setOrbState(mode));
session.on("disconnect", ({ reason }) => showCallEnded(reason));
```
### Transcript semantics
Transcripts on both sides arrive as **segments**, one segment per utterance or response, identified by `segmentId`:
* **User segments**: interim results **replace the entire segment text** (they never append). Render by upserting on `segmentId`; `final: true` marks the segment as finalized.
* **Agent segments**: text streams in sync with audio playback: what you display matches what the user has actually heard. If the agent is interrupted, the segment finalizes containing only the words that were spoken. A reply that spans several speech segments (for example around a tool call) arrives as several segments.
* The `message` event delivers only finalized messages from both sides, in order. Use it when you want a simple transcript list without handling interim updates.
A listener attached late misses the events fired before it subscribed. `session.getTranscript()` returns every segment so far, as `{ segmentId, role, text, final }` in conversation order: seed your UI from it, then keep applying live events by `segmentId`.
### Tool call lifecycle
Every tool the agent runs (webhook, client, or system tool) emits one `toolCallStarted`, resolved by exactly one `toolCallCompleted` or `toolCallFailed` with the same `callId`. `input` and `output` are JSON strings truncated at 4 KB; parse them only when the matching `*Truncated` flag is `false`. These events carry tool arguments and results into the end user's browser: to keep them off the client, start with `toolEvents: false`, or create the session with `tool_events: false` on your backend.
## Agent modes
`session.mode` (and the `modeChange` event) tracks what the agent is doing, for driving an orb or status indicator:
| Mode | Meaning |
| ----------- | ----------------------------------------------------------- |
| `listening` | Default state: the agent is waiting for or hearing the user |
| `thinking` | The agent is preparing a response |
| `speaking` | Agent audio is playing; ends when playback finishes |
`session.isSpeaking` is a convenience boolean for the `speaking` mode. Mode does not react to the user's own speech. For instant "the mic hears you" feedback, poll `getInputVolume()` locally.
## Send text
Users can type instead of talking, in the same session:
```javascript Text input theme={null}
// Send a typed user turn. There is no server echo; the SDK emits
// the finalized `message` event locally from the text you passed.
// By default the agent answers a typed turn in text only
// (agentResponseDelta / agentResponse) without speaking out loud.
session.sendUserMessage("Do you ship to Norway?");
// Pass `audio: true` to have the agent speak its reply for this turn.
session.sendUserMessage("What's my order status?", { audio: true });
// Signal typing so the agent doesn't talk over the user.
input.addEventListener("input", () => session.sendUserActivity());
// Stop the agent's current speech immediately (explicit barge-in).
session.interrupt();
```
Text-only replies are not paced to audio playback: the transcript streams as fast as it generates. Messages sent right after `start()` resolves, before the agent has finished joining, are held and delivered in order once it is ready.
## Audio controls
| Method / property | Description |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `setMicMuted(muted)` / `micMuted` | Mute or unmute the microphone. After a `microphone: false` start, the first unmute captures the microphone (permission prompt, so call it from a user gesture) and rejects with `mic_permission_denied` if refused |
| `setOutputVolume(v)` | Set playback volume, `0`–`1` |
| `getInputVolume()` / `getOutputVolume()` | Current mic / agent volume, `0`–`1`; poll per frame |
| `getInputFrequencyData()` / `getOutputFrequencyData()` | FFT data as `Uint8Array`, for visualizers |
| `startAudio()` | Unlock playback under browser autoplay policies; call inside a user gesture (for example the click that starts the call) |
| `setInputDevice(deviceId)` | Switch the microphone mid-call. Rejects with `device_change_failed` if the device cannot be activated, switching back to the previous microphone (best effort) |
| `setOutputDevice(deviceId)` | Route playback to another output device (`""` for the default). Rejects with `device_change_failed` where the browser does not support output selection (common on mobile); playback stays on the previous device |
Device ids come from `navigator.mediaDevices.enumerateDevices()`. To pick devices before the call starts, pass the `audio` option (`inputDeviceId` / `outputDeviceId`) to `start()` instead.
```javascript Audio visualizer theme={null}
function draw() {
const fft = session.getOutputFrequencyData(); // Uint8Array
renderBars(canvas, fft);
rafId = requestAnimationFrame(draw);
}
draw();
// Stop drawing when the session ends.
session.on("disconnect", () => cancelAnimationFrame(rafId));
```
## Client tools
Register handlers for tools of type `client` declared on the agent. The agent calls them mid-conversation and your return value goes back to the model:
```javascript Register a client tool theme={null}
const session = await AgentSession.start({
agentId: "YOUR_AGENT_ID",
clientTools: {
open_page: async params => {
showPanel(String(params.page));
return { opened: true };
},
},
});
// Or after start:
session.registerClientTool("highlight_product", handler);
```
Handlers can be sync or async; a thrown error or a timeout (default 15 s, set with `clientToolTimeoutMs`) is returned to the agent as a tool error. See [Client tools](/agents/build/client-tools) for declaration, naming rules, result size, and dispatch semantics.
## Errors
Failures surface as `FishAgentError`: thrown from `AgentSession.start()` when the session can't be created, emitted on the `error` event otherwise. Each carries a `code`, an optional `statusCode` (set on session-creation HTTP errors), and `cause`.
| Code | Meaning |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `session_request_failed` | Session creation failed; check `statusCode` |
| `agent_not_public` | `agentId` mode, but the agent is not enabled for public access |
| `origin_forbidden` | The page's origin is not in the agent's allowed origins |
| `unsupported_transport` | The session token uses a transport this SDK version doesn't know; upgrade the SDK |
| `mic_permission_denied` | The user denied microphone access; the SDK ends the session |
| `device_change_failed` | A requested audio device could not be activated, or the browser doesn't support selecting it (output selection is unsupported on some mobile browsers). Thrown from `start()` with `audio.outputDeviceId` set, or from device-switching calls |
| `connection_failed` | The realtime connection could not be established or recovered, a message to the agent could not be sent, the agent never joined (15 s), or the microphone could not be toggled |
| `session_expired` | The session token's join deadline passed before connecting |
| `tool_failed` / `tool_timeout` | A client tool handler threw, was not registered, or its result could not be delivered / a handler exceeded `clientToolTimeoutMs` |
| `provider_error` / `internal_error` | The session failed server-side: upstream model/voice provider vs. platform runtime. Emitted on `error`; the session may recover |
`provider_error` and `internal_error` carry only the category code. Raw
provider or infrastructure details are never sent to the browser.
```javascript Handle errors theme={null}
import { AgentSession, FishAgentError } from "@fishaudio/agent-client";
try {
const session = await AgentSession.start({ agentId: "YOUR_AGENT_ID" });
session.on("error", err => console.warn("session error:", err.code));
} catch (err) {
if (err instanceof FishAgentError && err.code === "mic_permission_denied") {
showMicHelp();
} else {
showRetry(err);
}
}
```
## End the session
```javascript Hang up theme={null}
await session.end(); // graceful hangup; disconnect fires with reason "user_hangup"
```
Calling `end()` more than once, or while a previous call is still finishing, is safe. `endReason` stays readable on the ended session.
## Underlying connection
The session API is transport-neutral: the server picks the transport when it creates the session, `livekit` (WebRTC) today. For needs the session API does not cover yet, such as connection-quality telemetry, `session.getRoom()` returns the live [livekit-client `Room`](https://docs.livekit.io/reference/client-sdk-js/), or `undefined` once the session has ended. Prefer the session API where one exists: code built on the `Room` couples to this SDK's transport choice and its pinned livekit-client major, with no compatibility promise across SDK versions. See [Wire protocol](/agents/deploy/protocol) for the messages underneath.
## Going further
Hooks, provider, and visualizer components on top of this SDK.
Create session tokens on your backend for private agents.
Backend-free sessions with origin allow-lists and rate limits.
The wire-level events underneath the SDK.