API/Streaming

Audio Transcriptions

Streaming

This API provides streaming speech-to-text transcriptions using WebSockets.

Endpoint: wss://api.sully.ai/v1/audio/transcriptions/stream?account_id=1234567890&api_token=1234567890&sample_rate=16000&language=en

​Headers

​X-API-KEYstring

The API key to use for authentication. Required if X-API-TOKEN is not provided.

​X-API-TOKENstring

The API token to use for authentication. Required if X-API-KEY is not provided.

​X-ACCOUNT-IDstring

The account ID to use for authentication.

​Query parameters

​languagestringdefault:"en"

The language of your submitted audio. See our Supported Languages documentation for a complete list of language options.

​sample_ratestring

The sample rate of your submitted audio.

​encodingstring

Specifies the encoding format of the audio being sent.

Important: This parameter is required when transmitting raw, unstructured audio packets without headers. If the audio data is encapsulated within a container format, this parameter should be omitted.

Supported formats:

• linear16: 16-bit, little-endian PCM audio

• flac: Free Lossless Audio Codec (FLAC)

• mulaw: Mu-law encoded WAV

• amr-nb: Adaptive Multi-Rate, narrowband

• amr-wb: Adaptive Multi-Rate, wideband

• opus: Ogg Opus codec

• speex: Speex codec

• g729: G729 codec (usable with raw or containerized audio)

​account_idstring

The account ID to use for authentication. Required if X-ACCOUNT-ID is not provided.

​api_tokenstring

A temporary authentication token. Required if X-API-KEY is not provided.

​When to use

The Speech-to-Text Websockets API is designed to generate text from partial audio input. It’s well-suited for scenarios where the input audio is being streamed or generated in chunks.

​Protocol

The WebSocket API uses a bidirectional protocol that encodes all messages as JSON objects.

​Connection Status Messages

Upon successful connection, the server sends a status message:

Copy

{
  "status": "connected",
}

When the connection closes:

Copy

{
  "status": "disconnected",
}

Important: Wait for the “status”: “connected” message before sending audio data. This ensures the server is ready to process your stream.

​Streaming input audio

The client can send messages with audio input to the server. The messages can contain the following fields:

Copy

{
  "audio": "Y3VyaW91cyBtaW5kcyB0aGluayBhbGlrZSA6KQ=="
}

​audiostringrequired

A generated partial audio chunk encoded as a base64 string.

Browser MediaRecorder Notice: When using Chrome’s MediaRecorder API, the first audio chunk contains critical header information. Always send this first chunk for proper audio processing. Failing to include header information may result in transcription errors or complete failure.

​Streaming output audio

The server will always respond with a message containing the following fields:

Response messageCopy

{
  "type": "transcript",
  "audio_start": 0,
  "audio_end": 3.2,
  "duration": 3.2,
  "text": "Hello, world",
  "isFinal": false, 
  "is_final": false,
  "words": [
    {
      "word": "Hello",
      "start": 0.2,
      "end": 0.7,
      "confidence": 0.9856743,
      "punctuated_word": "Hello,"
    },
    {
      "word": "world",
      "start": 0.9,
      "end": 1.2,
      "confidence": 0.9978341,
      "punctuated_word": "world"
    }
  ],
  "timestamp": "2023-08-15T14:32:07.123Z"
}

​typestring

The type of response, will be “transcript” for transcription results.

​audio_startnumber

Start time of the audio segment in seconds.

​audio_endnumber

End time of the audio segment in seconds.

​durationnumber

Duration of the audio segment in seconds.

​textstring

The processed text sequence.

​isFinalbooleandeprecated

Indicates if the generation is complete. Deprecated: Use is_final instead.

​is_finalboolean

Indicates if the generation is complete.

​wordsarray

Array of word objects with text content and timing information:

• word: The raw word as recognized

• start: Start time of the word in seconds

• end: End time of the word in seconds

• confidence: Confidence score between 0-1 for the word recognition

• punctuated_word: The word with proper capitalization and punctuation

​timestampstring

ISO-formatted timestamp when the response was generated.

Test this API out here: https://docs.sully.ai/api-reference/audio-transcriptions/streaming