# `ALLM.Providers.OpenAI.Speech`
[🔗](https://github.com/cykod/ALLM/blob/v0.6.1/lib/allm/providers/openai/speech.ex#L1)

OpenAI text-to-speech adapter. Implements `ALLM.SpeechAdapter` and
`ALLM.SpeechStreamAdapter` against `POST /v1/audio/speech`.

Layer B — runtime. Wire it with
`ALLM.Engine.new(speech_adapter: ALLM.Providers.OpenAI.Speech)` and call it
through `ALLM.synthesize/3`. The key resolves via
`ALLM.Keys.fetch!(:openai, opts)` at request-build time, so no key ever
lives on the engine.

    req = ALLM.SpeechRequest.new(input: "Hello.", voice: "coral", format: :wav)
    {:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, api_key: "sk-...")
    {:ok, wav_bytes} = ALLM.Audio.to_binary(resp.audio)

## Wire-field map

Observed live on **2026-09-24** by `scripts/record_openai_audio_fixtures.exs`,
whose probe halts the recording pass on any mismatch.

| Concern | OpenAI |
|---------|--------|
| Endpoint | `POST https://api.openai.com/v1/audio/speech`, JSON body |
| Auth | `authorization: Bearer <key>` |
| `input` | `ALLM.SpeechRequest.input`, at most 4096 code points |
| `model` | `:model`, or `gpt-4o-mini-tts` when `nil` |
| `voice` | `:voice`, or `alloy` when `nil`. Forwarded verbatim; OpenAI's valid set differs per model |
| `response_format` | `:format` as a string (`mp3`, `opus`, `aac`, `flac`, `wav`, `pcm`); omitted when `nil`, and OpenAI answers `mp3`. `:ulaw` and `:alaw` have no OpenAI value and are refused (see gates) |
| `instructions`, `speed` | forwarded when set, omitted when `nil`. OpenAI accepts `instructions` on `tts-1` with a 200 even though it documents that model as ignoring them |
| Options | `ALLM.SpeechRequest.options` merged **under** the fields above; `stream_format` is dropped |
| 200 body | raw audio bytes; `content-type` names the format |
| Streaming | the same request; the body arrives with `transfer-encoding: chunked`, and its first bytes arrive while the rest is still being synthesized (observed 2026-09-26 on `gpt-4o-mini-tts` and `tts-1`) |
| `sample_rate` | not sent. `pcm` and `wav` are always 24,000 Hz, so the response reports `24000` for them and `nil` otherwise |
| Usage | **none**: the body is raw audio, so `:usage` is an all-`nil` `%ALLM.Usage{}` |
| Correlation | `x-request-id` response header |
| Error envelope | `{"error": {"message", "type", "param", "code"}}`; the 401 is sent as `text/plain` |
| Unknown fields | **ignored** (200), so a mistyped `:options` key does nothing and raises no error |

## Adapter-injected defaults

  * `model` defaults to `"gpt-4o-mini-tts"` when `:model` is `nil`.
  * `voice` defaults to `"alloy"` when `:voice` is `nil`. OpenAI
    requires a voice, and `alloy` is accepted by every model probed.
  * The HTTP receive timeout defaults to 60 s
    when `opts[:request_timeout]` is absent.

## Pre-flight gates

Before any HTTP I/O and before `ALLM.Keys.fetch!/2`, so a request that is
going to be rejected never needs a key:

  1. **Input shape.** A non-binary, empty, or non-UTF-8 `:input` →
     `:invalid_request` with `metadata.field: :input`.
  2. **Input length.** More than 4096 **code points** →
     `:context_length_exceeded` with `metadata.count` and `metadata.max`.
     Code points, not graphemes and not bytes: an accented letter written
     as a base letter plus a combining mark counts twice, and a precomposed
     one counts once. The unit was settled by a live probe on 2026-09-24.
     A provider 400 whose message names `string_too_long` maps to the same
     reason, so a caller whose model has another limit sees one reason.
  3. **Format.** `:ulaw` and `:alaw` (8 kHz telephony audio) have no
     `response_format` value → `:unsupported_feature` with
     `metadata.field: :format`. This runs before the sample-rate gate, so
     a telephony request at 8,000 Hz names the format.
  4. **Sample rate.** OpenAI cannot produce a requested rate. For `:pcm`
     and `:wav` a `sample_rate` of `nil` or `24000` passes;
     for every other format (and for a `nil` format, which OpenAI answers
     as `mp3`) only `nil` passes. Anything else is `:unsupported_feature`
     with `metadata.field: :sample_rate`.

`synthesize/2`, `prepare_request/2` and `stream_synthesize/2` run the same
four gates.

## Response

`:audio` is `%ALLM.Audio{source: {:binary, bytes}}`. `:format` is derived
from the response `content-type` through `ALLM.SpeechResponse.mime_to_format/1`,
and `:audio.mime_type` is that format's canonical MIME type (or the raw
content type when it is an `audio/*` type outside the table). A 200 whose
content type is not `audio/*`, or whose body is empty, is
`:malformed_response`. `:raw` is always `nil`: the audio bytes live once,
in `:audio`. `:model` is the model that was sent.

`opts[:request_id]` is reflected onto `response.request_id`; when absent,
OpenAI's `x-request-id` header is used. `request.metadata` round-trips onto
`response.metadata`.

> #### The `x-request-id` fallback is unreachable through `ALLM.synthesize/3` {: .warning}
>
> The façade always supplies `opts[:request_id]` (it generates one when the
> caller does not), so on that path `response.request_id` is the façade's
> id and OpenAI's own correlation id is never observed. It surfaces only on
> a direct `synthesize/2` / `decode_response/4` call that omits
> `opts[:request_id]`. Identical to `ALLM.Providers.OpenAI.Moderation`.

## Streaming

`stream_synthesize/2` sends the same request with `Finch.async_request/3`
on the `ALLM.Finch` pool (HTTP/1) instead of `Req`, and returns a lazy
stream of `ALLM.SpeechEvent` values: `:speech_started` when the response
headers arrive, one `:audio_delta` per body chunk, then
`:speech_completed`. Usage is an all-`nil` `%ALLM.Usage{}`, as on
`synthesize/2`. `stream_synthesize_input/3` is not implemented: OpenAI's
text-in streaming is a different API.

An HTTP error status is not classified from the status alone. The error
body is collected until the response ends and then classified exactly as
`synthesize/2` classifies it, so the key redactor and the
`string_too_long` rule see the provider's message. A stream is never
retried once it has been returned.

## Retry

Each attempt runs inside `ALLM.Retry.run/3` under `opts[:retry]`
(default `:default`). The attempt marks `:rate_limited` (honouring
`Retry-After`), `:provider_unavailable`, `:timeout` and `:network_error`
as retryable, and the policy decides. The default policy's `retry_on` is
HTTP codes plus `:timeout`, so on its own this loop retries **`:timeout`
only**. Through `ALLM.synthesize/3` the façade retries all four reasons,
and `:timeout` is retried by both loops: up to 9 attempts at the default
policy, against 3 for the other three reasons.

## Error-struct hygiene

`%ALLM.Error.SpeechAdapterError{}` derives `Jason.Encoder`, so it is
often logged. No raw response body, request header or
`Authorization` value is copied into it, and there is no body preview.
Every provider-authored string it carries (`:message`,
`metadata.openai_code`, `metadata.openai_type`) passes a redactor that
replaces key-shaped tokens with `[REDACTED]`. OpenAI's real 401 masks the
key it echoes (`sk-proj-*****…9900`, observed 2026-09-24); the redactor is
defence in depth.

> #### Not every error is JSON-encodable {: .warning}
>
> The struct derives `Jason.Encoder`, but a `:timeout`, `:network_error`
> or invalid-JSON `:malformed_response` error carries an exception struct
> (`Req.TransportError`, `Jason.DecodeError`) on `:cause`, and
> `Jason.encode!/1` raises `Protocol.UndefinedError` on it. Drop or
> `Exception.message/1` the `:cause` before encoding. The image, embedding
> and moderation adapter errors share this limitation.

## Test-injection escape hatch

`synthesize/2` honours `opts[:adapter_opts][:speech_script]`: when the key
is present, the call is handed to `ALLM.Providers.FakeSpeech.synthesize/2`
BEFORE any of this adapter's gates run. This is what lets the
`ALLM.SpeechAdapter` conformance suite drive a real adapter without an
HTTP stub. `prepare_request/2` returns a stub error under the same key,
because a scripted response has no `Req.Request` analogue.

# `prepare_request`

```elixir
@spec prepare_request(
  ALLM.SpeechRequest.t(),
  keyword()
) :: {:ok, Req.Request.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}
```

Return an unfired `Req.Request` configured exactly as `synthesize/2` would
fire it, for callers who add headers, middleware or their own retry loop.

The pre-flight gates run first, so this is defined only for an input that
passes them. Under `opts[:adapter_opts][:speech_script]` it returns a stub
error instead of delegating to `ALLM.Providers.FakeSpeech`.

## Examples

    iex> req = ALLM.SpeechRequest.new(input: "Hi.")
    iex> {:ok, http} = ALLM.Providers.OpenAI.Speech.prepare_request(req, api_key: "sk-x")
    iex> URI.to_string(http.url)
    "https://api.openai.com/v1/audio/speech"

# `stream_synthesize`

```elixir
@spec stream_synthesize(
  ALLM.SpeechRequest.t(),
  keyword()
) ::
  {:ok, Enumerable.t(ALLM.SpeechEvent.t())}
  | {:error, ALLM.Error.SpeechAdapterError.t()}
```

Stream speech for `request.input` from OpenAI as it is synthesized.

Returns `{:ok, enumerable}` of `ALLM.SpeechEvent` values, or
`{:error, %ALLM.Error.SpeechAdapterError{}}` from a pre-flight gate. The
four gates of `synthesize/2` run first, then `ALLM.Keys.fetch!/2` (which
raises `%ALLM.Error.EngineError{reason: :missing_key}` by design). No
HTTP request is made until the enumerable is reduced.

**Injected defaults:** the same `model` (`"gpt-4o-mini-tts"`) and
`voice` (`"alloy"`) as `synthesize/2`, and the same JSON body.

**Options:**

  * `:stream_timeout` — milliseconds of silence between two transport
    messages before the stream ends with `{:error, %SpeechAdapterError{reason: :timeout}}`.
    Default 60000. The transport's own receive timeout
    defaults above it (see `ALLM.Providers.Support.Transport`).
  * `:receive_timeout`, `:request_timeout`, `:pool_timeout` — forwarded
    to `Finch.async_request/3`.
  * `:finch_name` — the Finch pool (default `ALLM.Finch`).
  * `:finch_module` — the module called for `async_request/3` and
    `cancel_async_request/1` (default `Finch`); tests pass
    `ALLM.Test.FinchStub`.

Each transport option is read from the top level of `opts`.
`ALLM.stream_synthesize/3` hoists an engine's `adapter_opts:` transport
keys (`ALLM.Adapter.transport_opts/0`) there, as the chat façades do; a
direct call passes them at the top level.

**Events:** `:speech_started` (with `sample_rate` 24000 for
`:pcm` and `:wav`, else `nil`) once the response headers arrive, one
`:audio_delta` per non-empty body chunk, and `:speech_completed` with an
all-`nil` usage. A failure ends the stream with one `{:error, _}`:

  * an HTTP error status: the error body is collected until the response
    ends, then classified as `synthesize/2` classifies it (a 401 is
    `:authentication_failed`, a `string_too_long` 400 is
    `:context_length_exceeded`), with provider strings redacted;
  * a 200 whose `content-type` is not `audio/*`: `:malformed_response`;
  * a 200 with no audio bytes: `:invalid_request` with
    `metadata.cause: :empty_input`;
  * no transport message within `:stream_timeout`: `:timeout`;
  * a transport failure: `:network_error`.

**Halting** the stream early (`Enum.take/2`) cancels the HTTP request and
removes the request's already-queued messages from the calling process's
mailbox. This is best-effort: a message the request process sends while
it is being shut down can still arrive afterwards.

`opts[:adapter_opts][:speech_script]` hands the call to
`ALLM.Providers.FakeSpeech.stream_synthesize/2` before any gate runs.

## Examples

    iex> req = ALLM.SpeechRequest.new(input: "Hello.", format: :pcm)
    iex> opts = [adapter_opts: [speech_script: [{:ok, "PCM-bytes"}]]]
    iex> {:ok, events} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, opts)
    iex> for {:audio_delta, bytes} <- events, into: "", do: bytes
    "PCM-bytes"

    iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :mp3, sample_rate: 24_000)
    iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
    iex> err.reason
    :unsupported_feature

    iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :ulaw)
    iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
    iex> {err.reason, err.metadata.field}
    {:unsupported_feature, :format}

# `synthesize`

```elixir
@spec synthesize(
  ALLM.SpeechRequest.t(),
  keyword()
) ::
  {:ok, ALLM.SpeechResponse.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}
```

Synthesize speech from `request.input` against OpenAI.

Returns `{:ok, %ALLM.SpeechResponse{}}` or
`{:error, %ALLM.Error.SpeechAdapterError{}}`. The one exception is
`ALLM.Keys.fetch!/2`, which raises `%ALLM.Error.EngineError{reason: :missing_key}`
by design; all four pre-flight gates run ahead of it.

**Sample rate:** `response.sample_rate` is `24000` for `:pcm`
and `:wav` and `nil` for every other format. A request `sample_rate` other
than `nil` (or `24000` for `:pcm` / `:wav`) is refused with
`:unsupported_feature` before any I/O.

**Formats:** `:ulaw` and `:alaw` are refused with `:unsupported_feature`
and `metadata.field: :format` before any I/O; OpenAI has no telephony
output.

**Injected defaults:** `model` is `"gpt-4o-mini-tts"` and `voice` is
`"alloy"` when the request leaves them `nil`, and the receive
timeout is 60 s when `opts[:request_timeout]`
is absent.

**Retry:** this adapter's own `ALLM.Retry.run/3` loop retries `:timeout`
under the default policy. `:rate_limited`, `:provider_unavailable` and
`:network_error` are retryable too, but only under a caller-supplied
`opts[:retry]` that lists them (as `ALLM.synthesize/3`'s own loop does).

See the module documentation for the gates, the wire-field map and the
`adapter_opts[:speech_script]` test-injection short-circuit.

## Examples

    iex> req = ALLM.SpeechRequest.new(input: "Hello.")
    iex> opts = [adapter_opts: [speech_script: [{:ok, "ID3-bytes"}]]]
    iex> {:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, opts)
    iex> ALLM.Audio.to_binary(resp.audio)
    {:ok, "ID3-bytes"}

    iex> req = ALLM.SpeechRequest.new(input: String.duplicate("a", 4097))
    iex> {:error, err} = ALLM.Providers.OpenAI.Speech.synthesize(req, [])
    iex> {err.reason, err.metadata}
    {:context_length_exceeded, %{count: 4097, max: 4096}}

---

*Consult [api-reference.md](api-reference.md) for complete listing*