TTS gRPC Integration

A speech synthesis integration is judged on two things callers notice: how long they wait before they hear the first word, and whether the audio stays in step with the rest of the call. Over gRPC, the platform returns a timing map with every synthesis and lets you pull the audio in pieces as it is produced, so you can begin playback early and sync text, highlights, or other events to the speech. This article explains the call sequence, how to retrieve audio, and how to use the timing information, with the mistakes that cost integrators the most time.

Before you start. A TTS interaction uses the same session lifecycle as any other speech interaction, described in ASR gRPC Integration. Every setting on the request is described in the TTS Settings Reference. For voices and languages, see Neural TTS Languages. For writing SSML, see Introduction to SSML.

The call sequence

You send text or SSML, the platform synthesizes it, and you retrieve the audio. Processing starts automatically when the interaction is created.

Step	Your client sends	What happens
1	`session_create`	Creates the session and returns the `session_id`.
2	`interaction_create_tts`	Creates the TTS interaction with the text, language, and audio format, and returns an `interaction_id`. Synthesis starts immediately.
3	(wait for the result)	A `final_result` arrives with status `FINAL_RESULT_STATUS_TTS_READY`, meaning the audio is available to pull. It carries the timing information. If you miss it, call `interaction_request_results`.
4	`audio_pull`	Retrieves the synthesized audio. Set `audio_id` to the interaction ID.
5	`interaction_close`	Ends the interaction when you have the audio.
6	`session_close`	Ends the session when all interactions are done.

A session that only synthesizes speech has no inbound audio. In that case, set the session audio format to STANDARD_AUDIO_FORMAT_NO_AUDIO_RESOURCE, which explicitly indicates that no audio resource should be allocated. A session can run many interactions, including several at once, so one session can synthesize multiple prompts during a call.

Creating the interaction

A TTS interaction needs four things: the language, the text, the audio format, and, optionally, a few behaviors. These go in interaction_create_tts.

Field	Purpose
`language`	The synthesis language, for example `en-US` or `de-DE`.
`inline_request`	The text to synthesize, as plain text or SSML, with optional voice and delivery settings.
`ssml_request`	A URL from which the platform fetches the SSML to synthesize. Use either this or `inline_request`, not both.
`audio_format`	The audio format to generate. It is required on every TTS interaction.
`synthesis_timeout_ms`	The longest time allowed for a synthesis. The default is 5000 ms.
`enable_partial_results`	When true, partial results are delivered while synthesis is still running, so you can pull audio early. The default is false.
`engine`	Optionally, the engine to use. By default, the platform routes by the voice you select.

All of these, with their defaults and the voice and delivery options, are in the TTS Settings Reference. For example, to synthesize a short prompt as 16 kHz linear PCM with a named voice:

{
  "interaction_request": {
    "interaction_create_tts": {
      "language": "en-US",
      "inline_request": {
        "text": "Thank you for calling. How can I help you today?",
        "tts_inline_synthesis_settings": {
          "voice": "Chris"
        }
      },
      "audio_format": {
        "standard_audio_format": "STANDARD_AUDIO_FORMAT_LINEAR16",
        "sample_rate_hertz": 16000
      }
    }
  }
}

Field names are shown as they appear in the API reference. The voice shown is illustrative; choose one from the voices available in your deployment.

Getting the audio

The final result tells you the audio is ready, but the audio itself is not in the result. You retrieve it with an audio_pull request.

Field	Purpose
`audio_id`	The audio to retrieve. For synthesized speech, use the `interaction_id` of the TTS interaction.
`audio_channel`	For multi-channel audio, the channel to return. Channel 0 is used if you do not specify one.
`audio_start`	The number of milliseconds from the beginning of the audio to start from. The default is the beginning.
`audio_length`	The maximum number of milliseconds to return. A value of 0, or no value, returns all available audio from the start point.

The platform answers with one or more audio_pull responses. Each carries audio_data (binary audio) and the audio_channel. Large audio is split across several responses, and the last one has final_data_chunk set to true. Concatenate the chunks in order to build the audio. Set a correlation_id on your pull request, and the responses carry it, so you can match them to the request when several pulls are in flight.

Starting playback early with partial results

For long prompts, waiting for the whole synthesis delays the first word. With enable_partial_results set to true, the platform sends partial_result messages while it is still synthesizing, and each one reports in audio_length_ms how much audio exists so far. Pull only what is new:

Create the interaction with enable_partial_results set to true, and keep a counter of the milliseconds you have already pulled, starting at 0.
On each partial_result, read audio_length_ms. The new audio is that value minus your counter.
Send an audio_pull with audio_start set to your counter and audio_length set to the new milliseconds, send the audio to the caller, and add the new milliseconds to your counter.
When the final_result arrives, pull any audio beyond your counter. Its audio_length_ms is the total length.

To measure how much partial results help, use the tts_first_result_time metrics, which record the time between a synthesis request and the first audio packet. See Prometheus Metrics Available.

What the result tells you

The TTS result (tts_interaction_result) describes the audio and maps it to the text, which is what lets you synchronize other things to the speech.

Field	What it holds	Use it to
`audio_format`	The format of the returned audio.	Configure your player or media path correctly.
`audio_length_ms`	The length of the generated audio in milliseconds.	Size buffers, schedule the next step, or track partial pulls.
`sentence_offsets_ms`	The offset in the audio where each synthesized sentence begins.	Jump to or highlight a sentence.
`word_offsets_ms`	The offset in the audio where each synthesized word begins.	Highlight words as they are spoken, or build captions.
`ssml_mark_offsets`	The offset of each SSML mark, as a name and an `offset_ms`.	Trigger an event, such as a screen change, at an exact point in the speech.
`voice_offsets`	The offsets at which each voice begins, as a name and an `offset_ms`.	Know where a voice change occurs when the SSML switches voices.
`synthesis_warnings`	Warnings from the synthesizer, each with a `message` and, optionally, the `line` where the issue was detected.	Find and fix problems in your SSML. Log them in development.

The offsets are in milliseconds from the start of the synthesized audio, so they line up with the audio_start and audio_length values you use when pulling.

Troubleshooting

Symptom	Likely cause and fix
No audio arrives with the final result	The result reports that the audio is ready, but the audio is retrieved with `audio_pull`. Send a pull request with the `interaction_id` as `audio_id`.
The pulled audio is cut off	Large audio is split across several `audio_pull` responses. Keep reading until you receive the response with `final_data_chunk` set to true.
The pull returns nothing or fails	Check that `audio_id` is the `interaction_id` of the TTS interaction. Pull all the audio before you close the interaction.
The first word is slow to start on long text	Enable `enable_partial_results` and pull audio as it is produced.
Synthesis times out	The default `synthesis_timeout_ms` is 5000 ms. Raise it for long text.
The request is rejected or the audio is unusable	The audio format is required. Set `standard_audio_format`, and `sample_rate_hertz` for raw PCM.
Timing offsets do not match the audio you play	The offsets are relative to the synthesized audio. If you pulled a segment with `audio_start`, subtract that start from the offsets.
SSML is partly ignored	Check `synthesis_warnings` in the result for the message and line.
A voice name from an older application is not found	Add a voice mapping. See the TTS Settings Reference.

For more on common synthesis problems, see Troubleshooting Common TTS Problems.

Was this article helpful?