In this paper, we investigate the phenomenon of modulation instability (MI) and its associated gain spectra in Bragg grating (BG) structure for generating the ultrashort pulses. We analyze MI for the BG structure in fibers under the influence of non-Kerr nonlinearity. In addition, we also discuss the generation of BG solitons near the photonic bandgap in the BG structure from the induced MI gain spectra. © 2005 IEEE.