# `ALLM.Providers.ElevenLabs.Speech` [πŸ”—](https://github.com/cykod/ALLM/blob/v0.6.1/lib/allm/providers/elevenlabs/speech.ex#L1) ElevenLabs text-to-speech adapter. Implements `ALLM.SpeechAdapter` against `POST /v1/text-to-speech/{voice_id}`. Layer B β€” runtime. Wire it with `ALLM.Engine.new(speech_adapter: ALLM.Providers.ElevenLabs.Speech)` and call it through `ALLM.synthesize/3`. The key resolves via `ALLM.Keys.fetch!(:elevenlabs, opts)` (environment variable `ELEVENLABS_API_KEY`) after the pre-flight gates, so no key ever lives on the engine. req = ALLM.SpeechRequest.new(input: "Hello.", format: :pcm) {:ok, resp} = ALLM.Providers.ElevenLabs.Speech.synthesize(req, api_key: "sk_...") {resp.sample_rate, resp.audio.mime_type} ## Wire-field map Observed live on **2026-09-26** by `scripts/record_elevenlabs_audio_fixtures.exs`, whose probe halts the recording pass on any mismatch. | Concern | ElevenLabs | |---------|------------| | Endpoint | `POST /v1/text-to-speech/?output_format=…`, JSON body. The host is `https://api.elevenlabs.io` unless `opts[:base_url]` or `adapter_opts[:base_url]` names another (e.g. a data-residency host) | | Auth | `xi-api-key: ` | | Voice | `:voice`, in the URL path, or `JBFqnCBsd6RMkjVDRZzb` when `nil` | | `text` | `ALLM.SpeechRequest.input` | | `model_id` | `:model`, or `eleven_flash_v2_5` when `nil` | | `output_format` | from `:format` and `:sample_rate` (table below) | | `speed` | sent as `voice_settings.speed` | | `instructions` | ElevenLabs has no such field; refused (see gates) | | Options | merged **under** the fields above into the JSON body; a `"voice_settings"` map is merged with `speed`, so both are sent. `options["query"]` (a map) adds URL query parameters under `output_format`. `output_format`, `model_id` and `text` are dropped | | 200 body | raw audio bytes; `content-type` names the format (`audio/mpeg`, `audio/pcm`, `audio/wav`, `audio/opus`, `audio/ulaw`, `audio/alaw`) | | Correlation | the `request-id` response header β†’ `response.id` | | Cost | the `character-cost` response header β†’ `response.raw` as `%{"character_cost" => n}` | | Usage | **none**: the body is raw audio, so `:usage` is an all-`nil` `%ALLM.Usage{}` | | Unknown fields | **ignored** (200), so a mistyped `:options` key does nothing and raises no error | ## Formats and sample rates | `:format` | `:sample_rate` accepted (`nil` β†’ default) | `output_format` | |-----------|--------------------------------------------|-----------------| | `:mp3` or `nil` | 22050, 24000, **44100** | `mp3_22050_32`, `mp3_24000_48`, `mp3_44100_128` | | `:opus` | **48000** | `opus_48000_64` | | `:pcm` | 8000, 16000, 22050, **24000**, 32000, 44100, 48000 | `pcm_` | | `:wav` | the same, **24000** by default | `wav_` | | `:ulaw` | **8000** | `ulaw_8000` | | `:alaw` | **8000** | `alaw_8000` | | `:aac`, `:flac` | β€” | refused, `:unsupported_feature` | `:ulaw` and `:alaw` are headerless 8 kHz G.711 telephony audio. ElevenLabs sends them as `audio/ulaw` and `audio/alaw`; the response's `:audio.mime_type` is the canonical `audio/basic` for Β΅-law and `audio/alaw` for A-law. Any other rate for either is refused with `:unsupported_feature` and `metadata.field: :sample_rate`. `pcm_44100` and `wav_44100` need ElevenLabs' Pro tier; a lower tier's 403 surfaces as `:unsupported_feature`. ## Adapter-injected defaults * `voice` defaults to `"JBFqnCBsd6RMkjVDRZzb"` when `:voice` is `nil`. ElevenLabs puts the voice in the URL path, so there is no request without one. * `model_id` defaults to `"eleven_flash_v2_5"`, ElevenLabs' low-latency model, when `:model` is `nil`. * `output_format` defaults to `mp3_44100_128` when `:format` is `nil`, and a `nil` `:sample_rate` takes the **bold** default of the table above. * The HTTP receive timeout defaults to 60 s when `opts[:request_timeout]` is absent. * On `/stream-input` only: `inactivity_timeout` is `min(180, ceil(stream_timeout / 1000))` seconds, and `auto_mode=true` unless `options["query"]` sets `auto_mode` (see `stream_synthesize_input/3` for the latency and quality trade-off). ## Pre-flight gates In this order, before any HTTP I/O and before `ALLM.Keys.fetch!/2`, so a request that is going to be rejected never needs a key: 1. **Input shape.** A non-binary, empty, or non-UTF-8 `:input` β†’ `:invalid_request` with `metadata.field: :input`. 2. **Instructions.** A non-`nil` `:instructions` β†’ `:unsupported_feature` with `metadata.field: :instructions`. 3. **Format and sample rate.** A format or rate outside the table above β†’ `:unsupported_feature` with `metadata.field` `:format` or `:sample_rate`. There is no local input-length gate: the limit differs per model. A provider 400 whose `detail` names `text_too_long` maps to `:context_length_exceeded`. ## Response `:audio` is `%ALLM.Audio{source: {:binary, bytes}}`. `:format` is derived from the response `content-type` through `ALLM.SpeechResponse.mime_to_format/1`, and `:audio.mime_type` is that format's canonical MIME type. A 200 whose content type is not `audio/*`, or whose body is empty, is `:malformed_response`. `:sample_rate` is the rate that was requested (the response does not state it). `:model` is the model that was sent. `opts[:request_id]` is reflected onto `response.request_id`, and `request.metadata` round-trips onto `response.metadata`. ## Streaming This module also implements `ALLM.SpeechStreamAdapter`, with both callbacks: * `stream_synthesize/2` sends the same request to `POST /v1/text-to-speech/{voice_id}/stream` with `Finch.async_request/3` (HTTP/1, the `ALLM.Finch` pool) and emits one `:audio_delta` per body chunk. The body is raw chunked audio (observed 2026-09-27: 44 characters of `pcm_24000` arrived in 26 chunks between 425 ms and 545 ms). `:speech_completed.id` is the `request-id` header. * `stream_synthesize_input/3` speaks an enumerable of text chunks over `wss:///v1/text-to-speech/{voice_id}/stream-input`, through `ALLM.Providers.Support.WebSocket`. The key goes in the upgrade request's `xi-api-key` header, never in the URL. | Concern | `/stream-input` (observed 2026-09-27) | |---------|------------------------------------------| | Query | `model_id`, `output_format`, `inactivity_timeout`, `auto_mode=true` by default; `options["query"]` adds parameters under the first three and may replace `auto_mode` | | Initial message | `{"text": " "}` plus `voice_settings` (with `speed`) and the other `options` keys. Unknown keys are ignored | | Text | `{"text": text}`, no space appended; whole words only under `auto_mode` (the adapter buffers to a word boundary), each non-empty chunk verbatim without it | | End of input | `{"text": "", "flush": true}`, then `{"text": ""}`; the server answers with its last audio, `isFinal: true` and a close 1000 | | Audio | `{"audio": base64, "alignment", "normalizedAlignment", "isFinal"}`; a `nil` or `""` audio is skipped | | Errors | a bad or missing key and an unknown voice still upgrade with 101, then send `{"error": code, "message", "code": 1008}` and close 1008. `invalid_api_key` and `authentication_required` are `:authentication_failed`, `voice_id_does_not_exist` is `:invalid_request` | | Upgrade errors | a model the endpoint does not serve is refused at the upgrade with an HTTP 400 `unsupported_model` (`eleven_v3`), `:invalid_request` | | Keep-alive | `{"text": " "}` after half of `inactivity_timeout` without a client frame; a keep-alive sent between `"Hi"` and `" there."` left the returned alignment `" Hi there."` | Neither stream is retried once it has been returned. ## Retry Each attempt runs inside `ALLM.Retry.run/3` under `opts[:retry]` (default `:default`). The attempt marks `:rate_limited` (honouring `Retry-After`), `:provider_unavailable`, `:timeout` and `:network_error` as retryable, and the policy decides. The default policy's `retry_on` is HTTP codes plus `:timeout`, so on its own this loop retries **`:timeout` only**. Through `ALLM.synthesize/3` the faΓ§ade retries all four reasons. An out-of-credit error is `:invalid_request` and is never retried. ## Error-struct hygiene No raw response body, request header or `xi-api-key` value is copied into `%ALLM.Error.SpeechAdapterError{}`. The provider's message and its `detail.code`, `detail.type` and `detail.status` (`metadata.code`, `metadata.type`, `metadata.provider_status`) pass a redactor that replaces ElevenLabs-shaped keys (`sk_…`) with `[REDACTED]`. ElevenLabs' invalid-key error does not echo the key (observed 2026-09-26); the redactor is defence in depth. ## Test-injection escape hatch `synthesize/2`, `stream_synthesize/2` and `stream_synthesize_input/3` honour `opts[:adapter_opts][:speech_script]`: when the key is present, the call is handed to the same-named `ALLM.Providers.FakeSpeech` function BEFORE any of this adapter's gates run. `prepare_request/2` returns a stub error under the same key. # `prepare_request` ```elixir @spec prepare_request( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, Req.Request.t()} | {:error, ALLM.Error.SpeechAdapterError.t()} ``` Return an unfired `Req.Request` configured exactly as `synthesize/2` would fire it, for callers who add headers, middleware or their own retry loop. The pre-flight gates run first, so this is defined only for a request that passes them. Under `opts[:adapter_opts][:speech_script]` it returns a stub error instead of delegating to `ALLM.Providers.FakeSpeech`. ## Examples iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :pcm) iex> {:ok, http} = ALLM.Providers.ElevenLabs.Speech.prepare_request(req, api_key: "sk_x") iex> URI.to_string(http.url) "https://api.elevenlabs.io/v1/text-to-speech/JBFqnCBsd6RMkjVDRZzb?output_format=pcm_24000" # `stream_synthesize` ```elixir @spec stream_synthesize( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, Enumerable.t(ALLM.SpeechEvent.t())} | {:error, ALLM.Error.SpeechAdapterError.t()} ``` Stream speech for `request.input` from ElevenLabs' HTTP stream endpoint (`POST /v1/text-to-speech/{voice_id}/stream`) as it is synthesized. Returns `{:ok, enumerable}` of `ALLM.SpeechEvent` values, or `{:error, %ALLM.Error.SpeechAdapterError{}}` from a pre-flight gate. The three gates of `synthesize/2` run first, then `ALLM.Keys.fetch!/2` (which raises `%ALLM.Error.EngineError{reason: :missing_key}` by design). No HTTP request is made until the enumerable is reduced. **Injected defaults:** the same as `synthesize/2`: `voice` is `"JBFqnCBsd6RMkjVDRZzb"` and `model_id` is `"eleven_flash_v2_5"` when the request leaves them `nil`; `output_format` is `mp3_44100_128` for a `nil` format, and a `nil` `sample_rate` takes the format's default. The URL and JSON body are `synthesize/2`'s, with `/stream` appended to the path. **Options** (each read from the top level of `opts`; `ALLM.stream_synthesize/3` hoists an engine's `adapter_opts:` transport keys there): * `:stream_timeout` β€” milliseconds of silence between two transport messages before the stream ends with `:timeout`. Default 60000. * `:receive_timeout`, `:request_timeout`, `:pool_timeout` β€” forwarded to `Finch.async_request/3`. * `:finch_name` (default `ALLM.Finch`) and `:finch_module` (default `Finch`; tests pass `ALLM.Test.FinchStub`). **Events:** `:speech_started` once the response headers arrive (`format` from the `content-type`, `sample_rate` the requested rate), one `:audio_delta` per non-empty body chunk, then `:speech_completed`, whose `id` is the `request-id` response header. Usage is all-`nil`. A failure ends the stream with one `{:error, _}`: an HTTP error status is classified from its body exactly as `synthesize/2` classifies it; a 200 that is not `audio/*` is `:malformed_response`; a 200 with no audio is `:invalid_request` with `metadata.cause: :empty_input`; silence past `:stream_timeout` is `:timeout`; a transport failure is `:network_error`. **Halting** the stream early (`Enum.take/2`) cancels the HTTP request and removes its already-queued messages from the calling process's mailbox (best-effort: a message the request process sends while it is being shut down can still arrive afterwards). A stream is never retried once it has been returned. `opts[:adapter_opts][:speech_script]` hands the call to `ALLM.Providers.FakeSpeech.stream_synthesize/2` before any gate runs. ## Examples iex> req = ALLM.SpeechRequest.new(input: "Hello.", format: :pcm) iex> opts = [adapter_opts: [speech_script: [{:ok, "PCM-bytes"}]]] iex> {:ok, events} = ALLM.Providers.ElevenLabs.Speech.stream_synthesize(req, opts) iex> for {:audio_delta, bytes} <- events, into: "", do: bytes "PCM-bytes" iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :flac) iex> {:error, err} = ALLM.Providers.ElevenLabs.Speech.stream_synthesize(req, []) iex> {err.reason, err.metadata.field} {:unsupported_feature, :format} # `stream_synthesize_input` ```elixir @spec stream_synthesize_input( ALLM.SpeechRequest.t(), Enumerable.t(String.t()), keyword() ) :: {:ok, Enumerable.t(ALLM.SpeechEvent.t())} | {:error, ALLM.Error.SpeechAdapterError.t()} ``` Speak text as it arrives: `input` is an enumerable of text chunks (for example `ALLM.AudioStream.text_deltas/1` over a chat stream), sent over ElevenLabs' WebSocket endpoint `wss:///v1/text-to-speech/{voice_id}/stream-input`. `request.input` is ignored. Returns `{:ok, enumerable}` of `ALLM.SpeechEvent` values, or `{:error, %ALLM.Error.SpeechAdapterError{}}` from a pre-flight gate. The gates run in this order, then `ALLM.Keys.fetch!/2`, before the enumerable is returned: 1. **Request shape.** `ALLM.Validate.speech_request(request, input: :streamed)`; a failure is `:invalid_request` with the validator's errors on `metadata.errors`. 2. **Instructions** and 3. **format and sample rate**, as in `synthesize/2` (`:unsupported_feature`). No socket is opened until the enumerable is reduced. The stream then connects, sends the initial message, emits `:speech_started`, and only then starts reducing `input` (in a helper process, see `ALLM.Providers.Support.InputPump`), so an upgrade the server refuses (any status other than 101, such as the HTTP 400 for `eleven_v3`) never reduces the input. **A bad key or an unknown voice is not refused at the upgrade:** ElevenLabs answers it with 101 and rejects it afterwards with an error frame and a close 1008. By then `:speech_started` has been emitted and the pump has started, so the input may already be reduced (up to the pump's credit window ahead of what was sent) before the stream ends with `{:error, %{reason: :authentication_failed}}` (or `:invalid_request` for the voice). When `input` is itself a paid stream, such as `ALLM.AudioStream.text_deltas/1` over a chat, that request is issued even though the key was never valid. The stream does not wait for a first server frame before reducing the input, because ElevenLabs sends nothing until it has received text. **Wire.** The key goes in the upgrade request's `xi-api-key` header, never in the URL. The URL carries `model_id`, `output_format` and `inactivity_timeout` (plus `options["query"]`, merged under them). The initial message is `{"text": " "}` plus `voice_settings` (with `speed`) and any other `request.options` keys (e.g. `generation_config`). Text is sent as `{"text": text}` with no space appended: under `auto_mode` (the default) it is buffered to whole words first (see below), and with `auto_mode` off each non-empty chunk is sent verbatim. The end of input sends any buffered text, then `{"text": "", "flush": true}` and then `{"text": ""}`. **Injected defaults:** `voice` is `"JBFqnCBsd6RMkjVDRZzb"` and `model_id` is `"eleven_flash_v2_5"` when the request leaves them `nil`; `output_format` is `mp3_44100_128` for a `nil` format and a `nil` `sample_rate` takes the format's default; `inactivity_timeout` is `min(180, ceil(stream_timeout / 1000))` seconds, and 180 when `:stream_timeout` is `:infinity`, so the server does not close a slow input's socket before this stream would; and `auto_mode=true`, unless `options["query"]` sets `auto_mode` itself. When no frame has been sent for half of `inactivity_timeout`, the stream sends ElevenLabs' documented keep-alive, `{"text": " "}`. **Latency against quality: `auto_mode` plus word buffering.** `auto_mode=true` is the default because it produced the first audio sooner (238 ms against 563 ms from the first text chunk, one probe of the four chunks `["Hel", "lo", " world", "."]` on 2026-09-27). Under `auto_mode` ElevenLabs voices every text frame as it arrives: in that probe `"Hel"` and `"lo"` were generated as separate clips (2.1 s against 1.0 s for the same text under the provider's default buffering), and ElevenLabs recommends `auto_mode` only for whole words or sentences. So while `auto_mode` is on, the adapter holds incoming text until a word boundary and sends only whole words: `["Hel", "lo", " world", "."]` goes out as `"Hello "` and, at the end of input, `"world."`. A boundary is whitespace or one of `! ? ;` and the CJK full-width marks; `.`, `,`, `:` and `'` are sent at the space that follows them, because they also occur inside words and numbers ("3.14", "don't"). `?`, `!` and `;` are boundaries even inside a token, so a URL's query string is split after its `?` (`"https://x.com/a?q=1"` goes out as `"https://x.com/a?"` then `"q=1"`). Whitespace includes the no-break space, so a number written with one ("10 000") can be split there when it straddles two chunks. Text with no boundary at all waits for the end of input. Buffering follows `auto_mode`: it is on when the query's `auto_mode` is `true` or a string equal to `"true"` ignoring case, and off otherwise. To turn both off, pass `options: %{"query" => %{"auto_mode" => false}}`: each chunk is then sent verbatim and ElevenLabs buffers by its `chunk_length_schedule` (tunable through `options: %{"generation_config" => %{"chunk_length_schedule" => [...]}}`), so the first audio waits for that schedule or the end of input. **Options:** `:stream_timeout` (default 60000) is the silence allowed between two messages, where both a server frame and an input chunk count, so a slow input does not time out a socket whose server is waiting for text. `:connect_timeout` bounds the upgrade. `:ws_module` is the `ALLM.Providers.Support.WebSocket` implementation (default `ALLM.Providers.Support.WebSocket.Mint`). `adapter_opts[:input_window]` is the input pump's credit window (default 8). **Events and failures:** `:speech_started` (format, MIME type and sample rate from the requested `output_format`), one `:audio_delta` per server audio frame, and `:speech_completed` on the server's `isFinal`. A failure ends the stream with one `{:error, _}`, `:invalid_request` for the input rules of `ALLM.SpeechStreamAdapter` (`metadata.cause` `:invalid_input_chunk`, `:empty_input`, `:input_raised` or `:input_crashed`), a classified server error frame (an invalid or missing key is `:authentication_failed`, an unknown voice `:invalid_request`), `:network_error` for a transport failure or a close before `isFinal`, and `:timeout`. **Halting** the stream closes the socket, stops the input pump, and removes the socket's and the pump's pending messages from the calling process's mailbox. `opts[:adapter_opts][:speech_script]` hands the call to `ALLM.Providers.FakeSpeech.stream_synthesize_input/3` before any gate runs. ## Examples iex> req = ALLM.SpeechRequest.new(input: "", format: :pcm) iex> opts = [adapter_opts: [speech_script: [{:ok, "PCM-bytes"}]]] iex> {:ok, events} = ALLM.Providers.ElevenLabs.Speech.stream_synthesize_input(req, ["Hel", "lo."], opts) iex> for {:audio_delta, bytes} <- events, into: "", do: bytes "PCM-bytes" iex> req = ALLM.SpeechRequest.new(input: "", instructions: "Whisper.") iex> {:error, err} = ALLM.Providers.ElevenLabs.Speech.stream_synthesize_input(req, ["Hi."], []) iex> {err.reason, err.metadata.field} {:unsupported_feature, :instructions} # `synthesize` ```elixir @spec synthesize( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, ALLM.SpeechResponse.t()} | {:error, ALLM.Error.SpeechAdapterError.t()} ``` Synthesize speech from `request.input` against ElevenLabs. Returns `{:ok, %ALLM.SpeechResponse{}}` or `{:error, %ALLM.Error.SpeechAdapterError{}}`. The one exception is `ALLM.Keys.fetch!/2`, which raises `%ALLM.Error.EngineError{reason: :missing_key}` by design; all three pre-flight gates run ahead of it. **Injected defaults:** `voice` is `"JBFqnCBsd6RMkjVDRZzb"` and `model_id` is `"eleven_flash_v2_5"` when the request leaves them `nil`; `output_format` is `mp3_44100_128` for a `nil` format, and a `nil` `sample_rate` takes the format's default (44,100 for mp3, 48,000 for opus, 24,000 for pcm and wav, 8,000 for ulaw and alaw); the receive timeout is 60 s when `opts[:request_timeout]` is absent. **Refused before any I/O** with `:unsupported_feature`: `instructions`, `format: :aac | :flac`, and a `sample_rate` outside the format's set. **Retry:** this adapter's own `ALLM.Retry.run/3` loop retries `:timeout` under the default policy. `:rate_limited`, `:provider_unavailable` and `:network_error` are retryable too, but only under a caller-supplied `opts[:retry]` that lists them (as `ALLM.synthesize/3`'s own loop does). See the module documentation for the wire-field map and the `adapter_opts[:speech_script]` test-injection short-circuit. ## Examples iex> req = ALLM.SpeechRequest.new(input: "Hello.") iex> opts = [adapter_opts: [speech_script: [{:ok, "ID3-bytes"}]]] iex> {:ok, resp} = ALLM.Providers.ElevenLabs.Speech.synthesize(req, opts) iex> ALLM.Audio.to_binary(resp.audio) {:ok, "ID3-bytes"} iex> req = ALLM.SpeechRequest.new(input: "Hi.", instructions: "Speak slowly.") iex> {:error, err} = ALLM.Providers.ElevenLabs.Speech.synthesize(req, []) iex> {err.reason, err.metadata.field} {:unsupported_feature, :instructions} --- *Consult [api-reference.md](api-reference.md) for complete listing*