Filter bank multicarrier (FBMC) has been developed as an alternative to overcome the drawbacks of orthogonal frequency-division multiplexing (OFDM) in certain areas, especially for improved spectral efficiency and errorless communication under doubly dispersive channel conditions. The prototype filters in FBMC systems must satisfy dual orthogonality conditions and be well localized in both time and frequency domains; hence have to be analysed in both dimensions. Ambiguity function is a two-dimensional function of time and frequency, which is the correlation between the filter coefficients and its frequency-modulated versions. In the present study, a matrix formulation of discrete ambiguity function generation is derived based on which the prototype filters for FBMC and OFDM are analysed and compared in terms of time frequency localization, dual orthogonality criterion, side-lobe levels in time and frequency, signal-to-interference ratio, constellation diagram concentration, and bit error rate plots. FBMC systems prove to be better than OFDM in all these criteria. Copyright © 2015 by the IETE.