Higgs Audio v3 — API Reference

Self-hosted, OpenAI-compatible text-to-speech and speech-to-text powered by Boson AI's Higgs Audio v3 models, plus a realtime full-duplex voice WebSocket. The TTS API mirrors the Boson /v1/audio/speech spec; ASR mirrors the OpenAI /v1/audio/transcriptions spec. TTS supports both non-streaming (full file) and streaming (low-latency PCM); the voice socket chains STT → LLM → TTS for live conversation.

Authentication

This is a self-hosted internal deployment — endpoints are open on the service host, no API key required. (Boson's hosted API uses Authorization: Bearer <key>; clients written for it work here by simply omitting/ignoring the header.)

Base URL

All endpoints below are relative to this base URL.

Streaming vs non-streaming

The service exposes three endpoints. Two are request/response (the speech and transcription HTTP routes); the third is a full-duplex realtime voice socket. Whether you get audio incrementally or all-at-once depends on the endpoint and flags below.

Create speech POST

Generate expressive speech from text. The input may embed inline control tokens for emotion, prosody, speed, and sound effects.

Request body

Responses

200 — audio in the requested format (audio/mpeg, audio/wav, audio/ogg, audio/L16 for pcm, …).   400 — invalid/missing input.   502 — upstream engine error.

Non-streaming — default, returns the full file

Omit stream (or set it to false). The server synthesizes the whole utterance and returns it as a single response body in your chosen response_format.

# Emotional, slow, expressive — saved as MP3
curl $BASE/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "<|prosody:speed_slow|><|emotion:affection|>Hello little friend, you did great today!",
    "response_format": "mp3"
  }' --output hello.mp3

import requests
r = requests.post("$BASE" + "/v1/audio/speech", json={
    "input": "<|emotion:elation|><|sfx:laughter|>Haha, wonderful job!",
    "response_format": "wav",
})
open("out.wav", "wb").write(r.content)

# Works with the OpenAI SDK (api_key is ignored here)
from openai import OpenAI
client = OpenAI(base_url="$BASE" + "/v1", api_key="not-needed")
client.audio.speech.create(
    model="higgs-audio-v3-tts",
    voice="default",
    input="<|emotion:contentment|>Good night, sleep tight.",
).stream_to_file("out.mp3")

Streaming — low-latency raw PCM

Set stream:true and response_format:"pcm". The server forwards raw PCM chunks as the engine decodes them — first audio typically arrives in ~0.2–0.5 s, so you can start playback before synthesis completes. The stream is headerless 24 kHz mono Int16LE (audio/L16); concatenate all chunks to get the full waveform.

# -N disables curl buffering so PCM bytes land as they arrive
curl -N $BASE/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "<|emotion:enthusiasm|>Streaming straight to your speakers!",
    "stream": true,
    "response_format": "pcm"
  }' --output out.pcm

# Headerless 24kHz mono s16le — play it with ffplay:
ffplay -f s16le -ar 24000 -ac 1 out.pcm

import requests
# stream=True on the request so chunks are yielded as they decode
with requests.post("$BASE" + "/v1/audio/speech", stream=True, json={
    "input": "<|emotion:enthusiasm|>Streaming straight to your speakers!",
    "stream": True,
    "response_format": "pcm",        # required for streaming
}) as r, open("out.pcm", "wb") as f:
    for chunk in r.iter_content(chunk_size=4096):
        if chunk:
            f.write(chunk)        # 24kHz mono Int16LE samples
# play: ffplay -f s16le -ar 24000 -ac 1 out.pcm

Control tokens

Embed tags as <|category:value|>. Sentence-level tags (emotion, style, prosody speed/pitch/expressive) lead the line and color the whole sentence; inline tags (sfx, prosody pause) are placed at the exact spot. Only catalog values are recognized — anything else is read aloud literally.

Emotion — sentence-level, <|emotion:…|>

Prosody — <|prosody:…|>

Style — <|style:…|>   ·   Sound effects — inline, <|sfx:…|>

Voice cloning

Pass a short reference clip via ref_audio (+ ref_text) for one-shot zero-shot cloning. This service translates them to the engine's references field.

curl $BASE/v1/audio/speech -H "Content-Type: application/json" -d '{
  "input": "Have a wonderful day!",
  "ref_audio": "https://example.com/sample.wav",
  "ref_text": "Hi, this is a sample of my voice.",
  "temperature": 0.8, "top_k": 50, "max_new_tokens": 1024
}' --output cloned.mp3

Languages

Higgs Audio v3 TTS speaks 100+ languages and code-switches within a single utterance — write the input in any mix (e.g. "哇，你真棒！Let's try again, 准备好了吗?") and it is spoken naturally. Full list: Boson · Languages (102).

Transcriptions POST

Transcribe speech to text — OpenAI Whisper-compatible. Built on Higgs Audio v3 STT (Whisper-large-v3 encoder + Qwen3 decoder), with a "thinking" mode for accuracy.

Request (multipart/form-data)

Example

curl $BASE/v1/audio/transcriptions \
  -F "file=@speech.wav" \
  -F "model=higgs-stt"

import requests
r = requests.post("$BASE" + "/v1/audio/transcriptions",
    files={"file": open("speech.wav", "rb")},
    data={"model": "higgs-stt"})
print(r.json()["text"])

Realtime voice WS

A full-duplex realtime voice agent over a single WebSocket. You stream microphone PCM up; the server returns live transcript partials, a streamed LLM reply, and synthesized speech PCM back down — all on the same socket. It chains the three Higgs/LLM stages into one low-latency loop.

Pipeline — what is truly streaming

Message protocol

The socket carries both JSON text frames (control + events) and binary frames (raw PCM audio). Direction matters: you send mic audio up as binary, the server sends speech down as binary.

Client → server

Server → client

Python client

A minimal verifiable client using the websockets library (pip install websockets). It sends a typed turn (skipping STT), prints every event, and collects the streamed speech PCM into a file you can play with ffplay.

import asyncio, json, websockets

# wss:// for https hosts, ws:// for http
URL = "$BASE".replace("https://", "wss://").replace("http://", "ws://") + "/ws/voice"

async def main():
    async with websockets.connect(URL) as ws:
        # Typed turn — skips STT, goes straight to LLM → TTS
        await ws.send(json.dumps({
            "type": "text",
            "scenario_id": "daily",
            "text": "hello",
        }))

        audio = bytearray()
        async for msg in ws:
            if not isinstance(msg, str):
                audio += msg                 # binary frame = 24kHz mono Int16LE PCM
                continue
            evt = json.loads(msg)
            print(evt["type"], evt.get("text", ""))
            if evt["type"] == "reply_done":
                break

        open("reply.pcm", "wb").write(audio)
        print(f"got {len(audio)} bytes of audio")
        # play: ffplay -f s16le -ar 24000 -ac 1 reply.pcm

asyncio.run(main())

Original model documentation

• Boson AI — Documentation overview
• Higgs Audio v3 TTS — model overview · create-speech API · control tokens · languages
• HF · higgs-audio-v3-tts-4b · HF · higgs-audio-v3-stt
• ModelScope · TTS-4B · ModelScope · STT
• SGLang-Omni — Higgs TTS cookbook · GitHub · boson-ai/higgs-audio
• Boson · streaming TTS — the stream + PCM contract this service mirrors for streaming speech
• websockets (Python) — client library used in the realtime voice sample