Vellore Institute of Technology (VIT) is a private university located in&nbsp;Tamil Nadu, India. Founded in 1984, as Vellore Engineering College, the institution offers 20 undergraduate, 34 postgraduate, four integrated and four research programs. It has campuses in Vellore, Amravati, Bhopal and Chennai.

VIT is one of the top ranked private universities in India according to NIRF, THE and QS Rankings.&nbsp;Govt. of India has recognized&nbsp;VIT, Vellore as an&nbsp;Institution of Eminence. This has allowed VIT to take independent quality initiatives and move up in world ranking.

&nbsp;

&nbsp;

VIT University

The robustness of automatic speech recognition (ASR) systems degrade due to the factors such as environmental noises, speaker variability, and channel distortion, among others. The approaches such as speech signal processing, model adaptation, hybrid techniques and integration of multiple sources are used for ASR system development. This paper focuses on building a robust ASR system by combining the complementary evidence present is the multiple modalities through which speech is expressed. Speech sounds are produced with lip radiation accompanied lip movements called Visual Speech Recognition (VSR). VSR system converts lip movement into spoken words. This system consists of lip region detection, visual speech feature extraction method and modeling techniques. Robust feature extraction from visual lip movement is a challenging task in VSR system. Hence, this paper reviews the feature extraction methods and existing databases used for VSR system. The fusion of visual lip movements with ASR system at different levels is also presented. © 2021 IEEE.

World University root node

Computer Science

School of Computer Science and Engineering

Chennai Campus

Proceedings - International Conference on Artificial Intelligence and Smart Systems, ICAIS 2021

A Survey on Visual Speech Recognition Approaches

Building voice-based Artificial Intelligence (AI) systems that can efficiently interact with humans through speech has become plausible today due to rapid strides in efficient data-driven AI techniques. Such a human–machine voice interaction in real world would often involve a noisy ambience, where humans tend to speak with additional vocal effort than in a quiet ambience, to mitigate the noise-induced suppression of vocal self-feedback. This noise induced change in the vocal effort is called Lombard speech. In order to build intelligent conversational devices that can operate in a noisy ambience, it is imperative to study the characteristics and processing of Lombard speech. Though the progress of research on Lombard speech started several decades ago, it needs to be explored further in the current scenario which is seeing an explosion of voice-driven applications. The system designed to work with normal speech spoken in a quiet ambience fails to provide the same performance in changing environmental contexts. Different contexts lead to different styles of Lombard speech and hence there arises a need for efficient ways of handling variations in speaking styles in noise. The Lombard speech is also more intelligible than normal speech of a speaker. Applications like public announcement systems with speech output interface should talk with varying degrees of vocal effort to enhance naturalness in a way that humans adapt to speak in noise, in real time. This review article is an attempt to summarize the progress of work on the possible ways of processing Lombard speech to build smart and robust human–machine interactive systems with speech input–output interface, irrespective of operating environmental contexts, for different application needs. This article is a comprehensive review of the studies on Lombard speech, highlighting the key differences observed in acoustic and perceptual analysis of Lombard speech and detailing the Lombard effect compensation methods towards improving the robustness of speech based recognition systems. © 2020, Springer Nature B.V.

Artificial Intelligence Review

Understanding Lombard speech: a review of compensation techniques towards improving speech based recognition systems

The Visual Speech Recognition (VSR) system performance is highly influenced by the selection of visual features. These features are categorized into static and dynamic features. This work proposes to exploit both lip shape (static-geometric features) as well as the temporal sequence of lip movements (dynamic-motion features) to build a combined VSR system with fusion both at feature level and model level. The digit dataset for VSR system is evaluated on the benchmark (using Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), and Zernike Moments (ZM)) systems. First, the Motion History Image (MHI) is calculated from all visemes from which wavelet and Zernike coefficients are extracted and modeled using a simple GMM L-R HMM. This proposed method shows a significant improvement in performance of 85% for MHI-DWT based features, 74% for MHI-DCT and 80% for MHI-ZM features. Geometric features are extracted using an Active Shape Model (ASM). Two types of fusion, namely feature fusion and model fusion are used. In feature level fusion, the motion features (MHI-DWT, MHI-DCT, and MHI-ZM) with geometric features (ASM) and modeled using GMM L-R HMM. The performance improves for combined features with an accuracy of 96.5% for DWT-ASM, 84% for DCT-ASM, and 93% for ZM-ASM. Model level fusion is performed using a two stream HMM model with stream weight of DWT-ASM, DCT-ASM, and ZM-ASM features. A weighted model level fusion results in further improvement, with an accuracy of 98.2% for DWT-ASM, 85% for DCT-ASM and 94.5% for ZM-ASM. The proposed work result achieves high recognition for VSR systems compared to the benchmark systems (DWT, DCT, and ZM). © 2020 The Authors. Published by Elsevier B.V.

Fulltext

Procedia Computer Science

Visual Speech Recognition using Fusion of Motion and Geometric Features

In this paper, we analyse segmented speech phonemes with Convolutional filters, after embedding them in Reconstructed Phase Space (RPS). These feature extracting Convolutional filters are trained on the embedded speech data from scratch and are also fine-tuned from networks trained with other data. Reconstruction of Phase Space portrays the dynamics of an observed system as a geometric representation. We present a study highlighting the discriminative capacity of the features extracted through Convolutional Neural Network (CNN) from the textural pattern and shape of this geometric representation. CNNs are heavily used in image-related tasks, but have not seen application on phase space portraits, possibly due to the higher dimensionality of the embedding. However, we find that the application of CNN on restricted bi-dimensional RPS, characterizes the space well than prior methods on high dimensional embeddings. We show experimental results supporting the use of RPS with CNN (RPS-CNN) for phoneme classification. The results affirm that essential signal characteristics are automatically quantified from the phase portraits of speech and can be used in place of conventional techniques involving frequency domain transformations. © 2020

Pattern Recognition Letters

Phoneme classification in reconstructed phase space with convolutional neural networks

West Nile Virus (WNV) is a disease caused by mosquitoes where human beings get infected by the mosquito’s bite. The disease is considered to be a serious threat to the society especially in the United States where it is frequently found in localities having water bodies. The traditional approach is to collect the traps of mosquitoes from a locality and check whether they are infected with virus. If there is a virus found then that locality is sprayed with pesticides. But this process is very time consuming and requires a lot of financial support. Machine learning methods can provide an efficient approach to predict the presence of virus in a locality using data related to the location and weather. This paper uses the dataset present in Kaggle which includes information related to the traps found in the locality and also about the information related to the locality’s weather. The dataset is found to be imbalanced hence Synthetic Minority Over sampling Technique (SMOTE), an upsampling method, is used to sample the dataset to balance it. Ensemble learning classifiers like random forest, gradient boosting and Extreme Gradient Boosting (XGB). The performance of ensemble classifiers is compared with the performance of the best supervised learning algorithm, SVM. Among the models, XGB gave the highest F-1 score of 92.93 by performing marginally better than random forest (92.78) and also SVM (91.16).

International Journal of Engineering and Advanced Technology Regular Issue

Prediction of West Nile Virus using Ensemble Classifiers

Forest fires have become one of the most frequently occurring disasters in recent years. The effects of forest fires have a lasting impact on the environment as it lead to deforestation and global warming, which is also one of its major cause of occurrence. Forest fires are dealt by collecting the satellite images of forest and if there is any emergency caused by the fires then the authorities are notified to mitigate its effects. By the time the authorities get to know about it, the fires would have already caused a lot of damage. Data mining and machine learning techniques can provide an efficient prevention approach where data associated with forests can be used for predicting the eventuality of forest fires. This paper uses the dataset present in the UCI machine learning repository which consists of physical factors and climatic conditions of the Montesinho park situated in Portugal. Various algorithms like Logistic regression, Support Vector Machine, Random forest, K-Nearest neighbors in addition to Bagging and Boosting predictors are used, both with and without Principal Component Analysis (PCA). Among the models in which PCA was applied, Logistic Regression gave the highest F-1 score of 68.26 and among the models where PCA was absent, Gradient boosting gave the highest score of 68.36.

International Journal of Recent Technology and Engineering 2

Predicting Forest Fires using Supervised and Ensemble Machine Learning Algorithms

Building an ASR system in adverse conditions is a challenging task. The performance of the ASR system is high in clean environments. However, the variabilities such as speaker effect, transmission effect, and the environmental conditions degrade the recognition performance of the system. One way to enhance the robustness of ASR system is to use multiple sources of information about speech. In this work, two sources of additional information on speech are used to build a multimodal ASR system. A throat microphone speech and visual lip reading which is less susceptible to noise acts as alternate sources of information. Mel-frequency cepstral features are extracted from the throat signal and modeled by HMM. Pixel-based transformation methods (DCT and DWT) are used to extract the features from the viseme of the video data and modeled by HMM. Throat and visual features are combined at the feature level. The proposed system has improved recognition accuracy compared to unimodals. The digit database for the English language is used for the study. The experiments are carried out for both unimodal systems and the combined systems. The combined feature of normal and throat microphone gives 86.5% recognition accuracy. Visual speech features with the normal microphone combination produce 84% accuracy. The proposed work (combines normal, throat, and visual features) shows 94% recognition accuracy which is better compared to unimodal and bimodoal ASR systems. © Springer Nature Singapore Pte Ltd. 2019.

International Conference on Innovative Computing and Communications Lecture Notes in Networks and Systems

Improving Recognition of Speech System Using Multimodal Approach

Epilepsy is a group of neurological disorders identifiable by infrequent but recurrent seizures. Seizure prediction is widely recognized as a significant problem in the neuroscience domain. Developing a Brain-Computer Interface (BCI) for seizure prediction can provide an alert to the patient, providing a buffer time to get the necessary emergency medication or at least be able to call for help, thus improving the quality of life of the patients. A considerable number of clinical studies presented evidence of symptoms (patterns) before seizure episodes and thus, there is large research on seizure prediction, however, there is very little existing literature that illustrates the use of structured processes in machine learning for predicting seizures. Limited training data and class imbalance (EEG segments corresponding to preictal phase, the duration just before the seizure, to about an hour prior to the episode, are usually in a tiny minority) are a few challenges that need to be addressed when employing machine learning for this task. In this paper we present a comparative study of various machine learning approaches that can be used for classification of EEG signals into preictal and interictal (Interictal is the time between seizures) using the features extracted from the intracranial EEG. Publicly available data has been used for this purpose for both human and canine subjects. After data pre-processing and extensive feature extraction, different models are trained and are effectively used to analyze the temporal dynamics of the brain (interictal and preictal) in affected subjects. We present the improved results for various classification algorithms, with AUROC values of best classification models at 0.99. © BEIESP.

International Journal of Engineering and Advanced Technology

Epileptic seizure prediction through machine learning and spatio-temporal features based time series analysis of intracranial electroencephalogram data

Speech signal distortion is unavoidable in real time applications. This distorted signal can adversely affect the performance of systems based on speech signals. Automatic speaker recognition (ASR) system performs well with clean speech signals while its performance degrades drastically with noisy speech. Enhancing the speech signal aims at improving the quality of the speech signal by reducing the noise contamination, thereby improving the performance of the ASR system. Noise could be background noise, reverberation, babble noise etc. In this paper, to improve the distorted speech signal, we propose a two stage speech enhancement algorithm where Empirical Mode Decomposition (EMD) with adaptive threshold in IMF selection is done at the first stage and then employ wavelet denoising (WD) in the second stage. The two stage denoising method is used to reduce noise in high and low frequencies. The effectiveness of the proposed algorithm is compared with a few baseline algorithms used for enhancement. © BEIESP.

International Journal of Recent Technology and Engineering

Enhancing noisy speech using WEMD

Automatic Speaker Recognition (ASR) is a process in which the person is identified or the claim made by the person is verified. In the last three or four decades lot of researches have been done in this field and it has evolved a lot over these period. But in a real world scenario, performances of these speaker recognition systems have failed in hostile conditions. Building a robust speaker recognition system is a challenging task and should address all types of distortions. In this paper, the performance of a speaker recognition system in hostile conditions is analysed and presented. Especially how the environmental noise imputes the speaker recognition system is studied using the MSR Identity toolbox. Test was conducted with clean speech signals and noisy speech signals at various SNRs. The outcome of the test clearly indicates that the accuracy of the ASRs is degraded in hostile conditions. The results may be used to come up with more robust ASR systems. © BEIESP.

International Journal of Innovative Technology and Exploring Engineering

Imputations of hostile conditions in automatic speaker recognition performance

Objectives: This paper proposes a method to improve the performance of a Visual Speech Recognition (VSR) system by combining the pixel-based and geometry-based features, so as to augment the performance of audio based Automatic Speech Recognition (ASR) systems in adverse conditions. Methods/Statistical Analysis: A video database comprising of 11000 utterances of isolated words, collected from 20 speakers, is used in this study. Pixel based features (DCT and DWT) and geometric features (Active Shape Model or ASM) are fused at two levels, one at the feature level and the other at the decision level. A simple Gaussian mixture HMM word model is built for feature level fusion, while a two stream HMM model is built for decision level fusion. Findings: The VSR system built using the combined features shows a significant improvement in performance when compared to individual VSR systems built using pixel and geometric based features. The accuracy of the individual system is 76% for geometric features, 64% for DCT and 72% for DWT pixel-based features. The performance improves for combined features with an accuracy of 80% for ASM+DCT and 84.7% for DWT+ASM. A weighted decision level fusion result in further improvement, with an accuracy of 84% for ASM+DCT and 92% for ASM+DWT. Application/Improvements: The combined VSR could be preferred over individual pixel/geometric feature based systems to augment the performance of audio based Automatic Speech Recognition (ASR) systems in adverse conditions. Further studies on improving the VSR system, which could be used in lieu of audio-based ASR systems in adverse situations, are being carried out.

Indian Journal of Science and Technology

An Improved Visual Speech Recognition of Isolated Words using Combined Pixel and Geometric Features

This paper presents a person identification system which combines recognition of facial features as well as spoken word using visual features alone. It incorporates a face recognition algorithm to identify the person, followed by spoken word recognition of 'lip-read' password. For face recognition, PCA is used for feature extraction, followed by a KNN based classification on the reduced dimensionality features. Spoken word recognition of passwords is performed using a Visual Lip reading (Visual ASR). The visual features corresponding to the spoken word is extracted using DWT, which are then recognized using a HMM based approaches. Since evidences from face recognition and visual lip reading could be complementary in nature, the scores from the two modalities are combined. Based on the combined evidences, decision making is for person identification is carried out. The performance for face identification is 90% while the accuracy for visual speech recognition is 72%. By combining these evidences, an improved accuracy 98% is achieved. © 2015 IEEE.

2015 International Conference on Computing and Network Communications (CoCoNet)

A person identification system combining recognition of face and lip-read passwords

The recognition rate of syllables in continuous speech is hampered due to the large size of the syllable vocabulary and the confusability among them. One approach to reduce the confusability and the search space is to preclassify the syllables into a small set of equivalent classes and then perform recognition within a particular equivalent class. In this study, the syllables in a language are grouped into equivalent classes based on their consonant and vowel structure. The syllables that map onto an equivalent class are called 'cohorts'. Artificial neural network models are used to preclassify the syllables into the equivalent class to which they belong. This is followed by recognition of the syllables among the smaller number of cohorts within a class by means of hidden Markov models. The preprocessing stage limits the confusable set to the cohorts within a class and reduces the search space. This hybrid approach helps improve the recognition rate over that of a plain HMM based recogniser.

Proceedings of International Conference on Intelligent Sensing and Information Processing, ICISIP 2004

Neural Network Preprocessor for Recognition of Syllables

In this paper, we present a method to combine evidence from multiple classifiers to recognize a large number of subword units of speech using small size training data sets. Grouping criteria based on phonetic description are considered, to build multiple modular networks for recognition of the large number of units. Nonlinear compression of feature vectors is carried out to obtain reduced dimensional patterns, and multiple classifiers are trained separately using the uncompressed feature vectors and compressed feature vectors. Evidence from multiple classifiers at different stages in the recognition system is combined using the sum rule. Effectiveness of the proposed method is demonstrated for recognition of isolated utterances of 145 consonant-vowel units of speech.