The following is a high-level view of how we plan to stich together the speech recognition system. I'm especially interested in feedback on mime types for the various data flows. I'll post seperately on specific components as well.

Components

Feature Computation

* Data Input: audio/x-raw-int
* Metadata Input: None
* Data Outout: mpf-audio-features/rasta
* Metadata Output: None
* Parameters: None
* Source and license: ICSI Rasta library. Owned by ICSI, so any open license.
* ETA: Initial version complete. Support for more variety in features as needed.

Given audio input, generate synchronized output features based on the input. Features are used by later analytic components.

Initially, we will only support 16 bit, 16 kHz, linear PCM encoded headerless audio and rasta features. However, the component will be based on ICSI rasta library, which supports a huge variety of other options.

Speech/Non-speech Detector

* Data Input: mpf-audio-features/rasta
* Metadata Input: None
* Data Outout: mpf-audio-features/speech-nonspeech-scores
* Metadata Output: None
* Parameters: Pre-trained speech/non-speech acoustic models matching the features.
* Source and license: ICSI's fast online speech/non-speech package. Any open license.
* ETA: Very fast GMM system complete with initial models. Improved models in second installment. Increased accuracy vs. decreased speed in third installment if needed.

Given input features, estimate the likelihood that the acoustics contain speech vs. non-speech (e.g. silence, music, noise). Output frame synchronous speech and non-speech scores. This output could be used e.g. for user interface, compression, or to isolate downstream analytics only to portions of the input containing speech. Note that this component does no smoothing or temporal modeling and outputs synchronously with the input frames. Downstream components could use the speech/non-speech scores to e.g. generate a segmentation.

Speech/Non-speech Segmenter

* Data Input: mpf-audio-features/speech-nonspeech-scores
* Metadata Input: None
* Data Outout: None
* Metadata Output: mpf-audio-segments/speech-nonspeech
* Parameters: None
* Source and license: To be written by ICSI based on existing scripts.
* ETA: Simple version in second installment. Improved version with higher accuracy in third installment.

Imposes a temporal model on the speech/non-speech scores to produce segments suitable for use with speech recognition (e.g. smoothing and a minimum duration for speech).

Utterance Normalizer

* Data Input: mpf-audio-features/*
* Metadata Input: mpf-audio-segments/speech-nonspeech
* Data Outout: None
* Metadata Output: mpf-audio-features/speech-segments
* Parameters: None
* Source and license: Excerpted and ported from the ICSI quicknet package. Any open license.
* ETA: Second installment

This component takes a features and segments and performs mean and variance normalization of the segment. This has been shown to help significantly with accuracy of later analytic components by equalizing some of the effects of volume and speaker variability.

Decoder

* Data Input: None
* Metadata Input: mpf-audio-features/speech-segments
* Data Outout: None
* Metadata Output: text/x-asr-word-list
* Parameters: Pre-trained acoustic models matching the feature. Pre-trained language models. A pronunciation dictionary.
* Source and license: AMI Juicer Decoder (BSD license) ported to mpf by ICSI.
* ETA: Software port underway. "Bootstrap" models complete. Complete port by second installment with ICSI Meeting Corpus models. Better quality models in third installment.

Given a segment, compute the most likely sequence of words spoken during that segment.

Data and Metadata

mpf-audio-features (data)

Features are time synchronous data computed from the audio. For ASR work, they are typically related to the spectrum (e.g. how much energy is there at each frequency). Since features are a direct representation of the audio and are synchronized with the input stream, we consider features to be data rather than metadata. Features consist of a fixed number of floating point values per time step.

The type of feature is encoded in the caps.

mpf-audio-segments (metadata)

A segment is simply a span of time. It consists of a start time, an end time, a type, and a label. This is used, for example, to identify speech segments from non-speech segments. Segments need not be synchronized with the input stream.

ASR Word List (metadata)

This represents the most likely stream of words according to the decoder. It consists of start time, end time, the word, and a confidence score. It need not be synchronized with the input stream.

Example Dataflow

The image below demonstrates a simple dataflow for an ASR system which uses one set of features for speech/non-speech and another for decoding. This demonstrates one reason we chose to separate detection from segmentation in the speech/non-speech component. The dotted lines refer to file system resident models. The solid lines are data/metadata flow.