Realtime Speech to Text Using Amazon Transcribe

Part of my work is to describe and explain technically how to achieve such and such use case using AWS. I've done numerous things across the years but this has been a recurrent ask: how could we build a realtime speech to text processing pipeline using Amazon Transcribe. This article is about just this: describe how to build a simple yet extremely efficient (read: cost-effective) realtime speech to text in AWS.

At high level, what should this solution should do?

This demo uses a web application to capture the audio but the idea is the web application server can deal with any source of audio as long as it's streamed into correctly.

The realtime capture of audio is sequenced as such:

In order to implement the solution and meet the efficiency criteria the design translates into this:

The web application is served using Amazon CloudFront and an S3 Bucket. The application server is hosted on AWS Fargate compute using Amazon ECS and load balanced using Amazon ELB. The transcript processor reads of an Amazon SQS Queue and stores conversations in an Amazon DynamoDB table. Which is then fed back to the web applications connected to a specific conversation.

For the curious minds, the application, as designed above, is available here.

This is all fine and jolly but let's look on how each of the moving parts are built.

The web application is built on React using the Redux toolkit and it uses an AudioWorklet to capture the audio stream from the selected microphone. An AudioWorklet is a specialized lightweight webworker which lets developers hook onto the rendering pipeline of a browser.

Once the worklet is properly hooked, the audio that is captured is resampled to 44100KHz if necessaery and encoded into PCM format on the browser side. This ensures that the application server receives the audio in a format which is compatible with Amazon Transcribe. To communicate with the application server, the web application opens two websockets: one short running socket for audio streams which stays open at most 30 seconds and a second one which receives conversation updates every 10 seconds.

The web application's, using the audio worklet, high level design looks like this:

This component is built on top of an ExpressJS server. Web sockets are handled using the ws NodeJS module. The integration with Amazon Transcribe service is achieved using direct websockets instead of using the SDK helpers.

Once a client has connected an audio stream through websocket, the application server opens a new websocket with Amazon Transcribe and sends in the audio. When the transcription stabilizes (when Amazon Transcribe declares it has reached its peak confidence for a sequence of speech) the application server pushes the transcipt to an SQS queue for processing. The application is server runs on ECS/Fargate and is distributed using Amazon CloudFront targetting an Application Load Balancer.

It provides 3 endpoints:

A simple Lambda function connected to the SQS queue picks a speech sequence and stores the data in a new row in a DynamoDB table, the same which is used by the application server to read conversation updates.

Transcripts are stored in an SQS queue and the Lambda Function pulls messages from the queue. The partitionning on the DynamoDB Table is done on the conversation ID and the sorting key is set on the timestamp.

Of course, this is just a start. By streaming the stabilized speech to text output out of the DynamoDB, it would be possible to add sentiment detection in a hot path and intent detection in a warm path mode. The design at this point is rather demo quality than production ready.