Aphasia is a neurological speech disorder which impairs the person's ability to understand or express and also reading and writing capability of an Individual will be affected. Aphasia is caused due to stroke, brain trauma, tumor in cerebrum, brain surgery, infection in cerebrum, some neurological diseases. It also affects the language skills of an aphasia individual due to harmness in the section of cerebrum. Individuals with Aphasia face difficulties in sentence framing, analyzing meaning of words, finding correct words to express their ideas, feelings, thoughts and also problem with pronunciation and stammering, all these difficulties greatly affect their personal and professional life. This paper focus on bridging a gap between the individual with Aphasia and the receiver. This motivated us to come up with the Speech Intelligence System which act as an interface for the communication between the individual with Aphasia and the society. Based on the severity of Aphasia, our knowledge-based system will analyze the unstructured words pronounced by the speech disorder person and transformed it to the meaningful prediction Our Speech Intelligence System will interpret speech disorder person's spoken words into text form by matching with the datasets. © 2019 IEEE.