Word Markers & Audio Statistics

Use word markers to match audio segments with words in order to make corrected transcripts or extract audio pieces with the associated text.

There are some constraints to using word markers.

• "Words" are usually words only but can also be whole phrases like "New Year's Eve" or "iPhone XS Max".
• Words can include control commands like "newline", "period", or "all caps".
• Several sets of text markers can map to the same word. For example: "$123.45" can map from "one hundred twenty three dollars and forty five cents".

There are two ways to get word markers: the New Method and the Old Method.

New Method

The new method supplies a list of word marker descriptions that is the same whether supplied in a WebSocket message or fetched via an HTTP request.  If Stable Text is used, the markers are supplied at the same time as the Stable Text, otherwise the markers are made available at the end of the dictation.

In the HTTP case, the dictation details can be fetched and there will be a wordMarkers property.

                            [{"audioStart":0.39,"audioLength":0.32999998,"textStart":0,"textLength":2,"text":"My"},
                             {"audioStart":0.71999997,"audioLength":0.51,"textStart":3,"textLength":4,"text":"text"},
                             {"audioStart":1.26,"audioLength":0.29,"textStart":8,"textLength":2,"text":"so"},
                             {"audioStart":1.55,"audioLength":0.41,"textStart":11,"textLength":3,"text":"far"},
                             {"audioStart":2.0,"audioLength":0.48,"textStart":14,"textLength":1,"text":"."}]

In the WebSocket case, the message will contain a markers field.

{
	"data" : {
		"kind" : "HYPOTHESISTEXT",
		"text" : "My text so far.",
		"substitutedText" : "My text so far.",
                "markers" : [{"audioStart":0.39,"audioLength":0.32999998,"textStart":0,"textLength":2,"text":"My"},
                             {"audioStart":0.71999997,"audioLength":0.51,"textStart":3,"textLength":4,"text":"text"},
                             {"audioStart":1.26,"audioLength":0.29,"textStart":8,"textLength":2,"text":"so"},
                             {"audioStart":1.55,"audioLength":0.41,"textStart":11,"textLength":3,"text":"far"},
                             {"audioStart":2.0,"audioLength":0.48,"textStart":14,"textLength":1,"text":"."}]
	}
}

Word Marker Object Fields and Values

• audioStart - The time, relative to the start of the dictation, in seconds, that the audio for this word starts.
• audioLength - The length of the audio for this word in seconds.
• textStart - The starting character position of this word.  The transcript for the dictation starts at 0.
• textLength - The number of characters in this word.  In general spaces are outside of words, and not part of any word marker.
• text - The text for the word.  Formatting will have been done.

The Old Method

The old method is only available for HTTP dictation requests and is detailed below.

There are two calls used to get the markers:

• A call to collect the audio statistics which contains the audio markers.
• A call to collect the text statistics which contains text markers.

These two calls are made after checking the Dictation status and before de-registering the Dictation observer.

It is important to note that the calls only return the markers and NOT the audio. The client is responsible for having or requesting the audio before deregistering the Dictation observer.

Request

The client makes a request for audio statistics.

 Single Sign-On Request

The only thing different about SSO is the “Authorization:” header.

Response

The highlighted values pertain to word and segment audio markers. These values are pairs of indices that represent the start and end byte offsets of each word or segment in the dictation audio.

Segment markers may not be valid (but word markers are).

• numWordAudioSegments—The number of words in the dictation audio.
• AudioMarkers—The byte offsets for the audio segments in the dictation.
• numWordSegments—The number of words in the dictation.
• numSegments—The number of segments in the dictation.
• WordAudioMarkers—The byte offsets of the words in the dictation. The offsets are pairs representing the beginning and end offsets for each word in the dictation audio.

The server returns the audio properties.

Request

The client makes a request for text properties.

Single Sign-On Request

The only thing different about SSO is the “Authorization:” header.

Response

The highlighted values pertain to word and segment text markers. These values are pairs of indices that represent the start and end of each word or segment in the dictation text.

For the Enhanced dictation server, segment markers may not be valid (but word markers are).

• numWordSegments—The number of words in the dictation text.
• DictationWords—The number of words in the dictation text.
• TextMarkers—The pairs of indices for the segments in the dictation.
• numWordTextSegments—The number of words in the dictation text.
• WordTextMarkers—The text indices of words in the dictation. The indices are pairs representing the beginning and end indices for each word.
• numSegments—The number of segments in the dictation.

The server returns the text properties.

From /text/stats you get string indexes into the final transcription text.  The indexes come in pairs, so there will always be an even number of entries in the list. The first value of a pair indicates the start position of a marker in the text, and the second value in the pair indicates the end of a marker value in the text.
WordTextMarkers=[0, 2, 3, 3, 5, 11, 13, 14]
 
From /audio/stats you get 16-bit 16kHz PCM byte counts.  These values also come in pairs, and indicate at what time in the audio the corresponding text was generated from.
WordAudioMarkers=[3518, 23358, 23360, 39038, 60798, 73278, 73278, 76798]
 
In order to use timestamps, not byte counts, an example method to convert the byte counts into a timestamp in millis:

WordTextMarkers and WordAudioMarkers is like forming data structure(s).  Using the above data (bolded) as an example, and the conversion method, as well as the corresponding example transcription text:   CNS: Patient is