Platform Concepts: Sessions, Interactions, Settings and Audio
Most integration problems on the speech platform trace back to a missing mental model, not a missing feature. Once you see how the platform is organized, such as what a session holds, how an interaction relates to it, where settings come from, and how one stream of audio can serve several services, the individual API calls fall into place. This article describes that model: sessions, interactions, settings, audio, and the callbacks that carry results back. It applies to the speech gRPC API, and it is the foundation for the product-specific integration articles.
The model at a glance
| Object | What it is | How long it lasts |
|---|---|---|
| Session | A set of API calls for one audio conversation, typically one phone call. It owns the audio stream and the settings that apply to the work done on it. | From session_create to session_close. |
| Interaction | One product-specific service request within a session, such as one recognition, one synthesis, or one call classification. | From its create request to interaction_close. |
| Settings | The configuration values that control how interactions behave. They are inherited from the cluster, deployment, session, and interaction. | Resolved when an interaction is created. |
| Audio | The binary audio you push into a session, which any interaction that needs audio can use. | The life of the session. |
| Callbacks | The messages the platform returns on the session stream: results, voice activity events, and errors. | The life of the session. |
Sessions
A session is a set of API calls carried out for a specific audio conversation, for example one call in a contact center. To use the speech services, you start a session, keep the session_id it returns, and use that ID on every later call in the session. You can supply your own session ID (a UUID) or let the platform generate one.
A session is tied to a single audio stream, and that stream can be used by many interactions. This is the central idea: you push the audio once, and every interaction that needs audio reads it. A session is the natural unit to match to a call, because it carries everything the call needs and ends when the call ends.
Interactions
An interaction is a product-specific service request, usually a transaction between your client and the speech services. A session can run any number of interactions, of different products or of the same one. For example, an IVR can run a fresh recognition for every question it asks during one call, and a recording pipeline can run transcription and diarization against the same audio, then NLU on the transcript.
| Interaction | Create request | What it does | Needs audio |
|---|---|---|---|
| ASR (grammar) | interaction_create_asr | Matches speech against grammars and returns structured results | Yes |
| Transcription | interaction_create_transcription | Converts speech to text without a grammar | Yes |
| Grammar parse | interaction_create_grammar_parse | Parses text against grammars | No |
| TTS | interaction_create_tts | Synthesizes speech from text or SSML | No (produces audio) |
| CPA | interaction_create_cpa | Classifies who answered a call | Yes |
| AMD | interaction_create_amd | Detects answering machine beeps, fax, SIT, busy, and screening tones | Yes |
| NLU | interaction_create_nlu | Summarizes, translates, and analyzes text | No |
| Language ID | interaction_create_language_id | Identifies the language of audio | Yes |
| Diarization | interaction_create_diarization | Identifies who spoke when | Yes |
| Normalize text | interaction_create_normalize_text | Applies text normalization to a transcript | No |
| Sentiment | interaction_create_sentiment | Scores the sentiment of text | No |
| Neuron | interaction_create_neuron | Extracts intents and slots from text | No |
Interactions in a session can run one after another or in parallel. CPA and AMD, for example, run in parallel on the same audio, as do diarization and transcription. Each interaction gets its own interaction_id, which you use in every later call about it.
Settings
Every interaction is shaped by settings, and the same setting can be defined in several places. The most specific level wins, and an interaction takes a snapshot of its settings when it is created, so later changes do not reach interactions that already exist.
| Level | Set by | Applies to |
|---|---|---|
| Cluster | Helm chart values | The whole installation |
| Deployment | The Deployment Portal or the Management API | Every session and interaction in a deployment |
| Session | session_set_settings | Interactions created in the session afterward |
| Interaction | The interaction create request | That one interaction |
The full rules, with a worked example, are in Speech Settings: Hierarchy, Inheritance and Value Types. Because an interaction takes its settings at creation, set session settings before you create the interactions that need them, and supply interaction-specific settings in the create request itself.
Audio
Audio for ASR, transcription, CPA, and AMD is pushed into the session, not into an individual interaction. You declare the audio format once with session_audio_format, before any audio is sent, and you cannot change it afterward. You then supply the audio with audio_push, either as one block or as a stream of binary chunks. Every interaction in the session that needs audio reads from this one stream, which is why a single push can feed, for example, CPA, AMD, and transcription at the same time.
TTS works the other way around. It needs no inbound audio, and its output is returned per interaction, retrieved with audio_pull. A session that only synthesizes speech can declare STANDARD_AUDIO_FORMAT_NO_AUDIO_RESOURCE, which explicitly indicates that no audio resource should be allocated.
Audio channels
An audio stream can contain any number of channels. Each interaction chooses the channel it processes with audio_consume_settings.audio_channel, a number from 0 to N, where 0 is the default.
- Mono audio has a single channel, so the default of 0 is correct.
- Stereo audio has channels 0 and 1. An interaction processes one of them. You can process the two channels with two interactions at the same time, for example the caller on one channel and the agent on the other.
- Multi-channel audio with many channels works the same way, with the channel numbered from 0 to N.
As with other interaction settings, a channel you always want can be set as a session default. For formats that carry a header, such as WAV, the channel count comes from the header. For raw (headerless) multi-channel audio, set channel_count in the audio format.
Audio formats
You select a format by name in the audio format. The platform supports these standard formats.
| Format | Description |
|---|---|
STANDARD_AUDIO_FORMAT_LINEAR16 | Uncompressed 16-bit signed little-endian samples (linear PCM), headerless |
STANDARD_AUDIO_FORMAT_ULAW | 8-bit samples using G.711 PCMU (mu-law), headerless |
STANDARD_AUDIO_FORMAT_ALAW | 8-bit samples using G.711 PCMA (a-law), headerless |
STANDARD_AUDIO_FORMAT_WAV | WAV audio. The header specifies the format, sample rate, and number of channels. |
STANDARD_AUDIO_FORMAT_FLAC | FLAC audio. The header specifies the format, sample rate, and number of channels. |
STANDARD_AUDIO_FORMAT_MP3 | MP3 audio |
STANDARD_AUDIO_FORMAT_OPUS | OPUS audio |
STANDARD_AUDIO_FORMAT_M4A | M4A audio |
STANDARD_AUDIO_FORMAT_MP4 | Audio packed into an MP4 container |
STANDARD_AUDIO_FORMAT_GSM | GSM audio |
STANDARD_AUDIO_FORMAT_NO_AUDIO_RESOURCE | Explicitly indicates that no audio resource should be allocated, so no audio is processed. Use it for TTS-only sessions. |
Not every format is supported in every case, because different operations natively handle a subset of the formats.
Sample rate
The sample rate is separate from the format. Set it with sample_rate_hertz, for example 8000 for 8 kHz. It is mandatory for raw PCM audio and optional for the other formats, and for formats that carry a header, the value in the header is used instead. The default is 8000. Most telephony audio is recorded at 8000 Hz, which is the primary supported rate.
Streaming and batch audio
Audio for ASR and transcription is consumed in one of two modes, chosen with audio_consume_settings.audio_consume_mode.
| Streaming (the default) | Batch | |
|---|---|---|
| Audio | Live audio pushed while the caller speaks | A complete, pre-existing recording |
| Where processing starts | Controlled by stream_start_location | From the beginning of the audio. You can adjust the start with start_offset_ms. |
| Where it ends | When voice activity detection (VAD) finds the end of speech, or when you finalize | When the audio has been processed, or at audio_consume_max_ms (0 means all audio) |
VAD detects when speech starts and stops, and is controlled by vad_settings.use_vad. In batch mode it can also be used, in which case the engine looks for the start of speech within the audio and stops at the end of speech. The product integration articles describe each mode in detail.
Callbacks
The session is a bidirectional stream. You send requests on it, and the platform returns everything on the same stream, so your client must listen for these messages while it sends.
| Message | What it tells you |
|---|---|
final_result | An interaction has a final result, with its status. |
partial_result | An interim result is available, if partial results are enabled. |
vad_event | Voice activity detection found something in the audio: the caller started speaking (barge-in), stopped speaking, or a timeout occurred. |
session_event | Something went wrong, such as an error. It carries a status message and, if relevant, the interaction ID. |
| Responses to your requests | Each create request returns a response with its interaction_id, and a request for results or audio returns that data. |
The barge-in event is the one to use when you play a prompt: it tells you the caller started speaking, so you can stop the prompt.
Putting it together
| Scenario | Session | Interactions |
|---|---|---|
| IVR call | One per call, with the call’s audio pushed live | TTS for each prompt, and one ASR interaction per question |
| Outbound dialer | One per call | CPA and AMD in parallel on the same audio; TTS to deliver the message |
| Recorded call analysis | One per recording, with the file pushed once | Transcription and diarization in parallel on the same audio; NLU on the transcript afterward |
| Multi-language recording | One per recording | Language identification first, then transcription in the language it found |
Where to go next
- ASR gRPC Integration, for the step-by-step call sequence.
- Transcription gRPC Integration, NLU, Language Identification and Diarization gRPC Integration, and TTS gRPC Integration, for each product.
- API Access, Authentication and Required Headers, for the identifiers every request needs.