A new organic nonlinear optical chalcone derivative-1-(5-bromothiophene-2-yl)-3-(3-methoxyphenyl) prop-2-en-1-one (3MO5B2SC) molecule has been grown by slow evaporation technique. Theoretically DFT method was used to study optimized structure of the molecule and compare with reported XRD data. The structure of the molecule has been confirmed by the experimental and computed FTIR and H1 NMR spectrum studies. The melting point (117 °C) and thermal stability (273 °C) of the molecule have been determined using DSC, TGA, DTA and DTG. The optical absorption spectrum revels that molecule has lower optical cut-off wavelength of 410 nm recorded by UV–Vis–NIR spectrophotometer and 480 nm calculated by DFT. The band gap calculated from experimental data and theoretical HOMO–LUMO plots is same (3.3 eV). The global chemical reactivity descriptor (GCRD) of the sample has been calculated. The nonlinear parameters such as nonlinear absorption coefficient (β ~ 10−4 cm/W), nonlinear refraction index (n2 ~ 10−8 cm2/W) and third-order nonlinear susceptibilities (χ(3) ~ 10−6 esu) and molecular hyperpolarizability (γh ~ 10−26 esu) were measured under diode-pumped continuous wave Nd:YAG laser at 532 nm wavelength. The static electric dipole moment (μ), polarizability (α), first hyperpolarizability (β), static and dynamic second hyperpolarizability (γh) have been calculated theoretically and compared with experimental values. The experimental and theoretical nonlinear parameters are many times higher than urea molecule. The mean polarizability, total polarizability, first-order hyperpolarizability and static second hyperpolarizability value of the molecule is 20, 7.5, 17 and 91 times higher than standard urea, respectively. The nonlinearity exhibited by this molecule exploited for studying its optical limiting behaviour. The structure and its corresponding properties prompt that it can be used as nonlinear optical devise applications. © 2020, Indian Association for the Cultivation of Science.