Barium borate (β-BaB2O4) nanoparticles were prepared by a hydrothermal technique using barium chloride and boric acid as starting materials. Powder x-ray diffraction confirms the pure single phase formation of β-BaB2O4. Fourier transform infrared spectroscopy spectra shows the presence of BO3 units which extend themselves to form (B3O6)3- anionic groups in β-BaB2O4. Scanning electron microscopy imaging shows the two-dimensional undulating particle surface with particle size in the range of 45-170 nm. β-BaB2O4 has strong absorption in the UV region and it possesses a wide optical transmittance window (201 to 1100 nm). The optical band gap of the β-BaB2O4 nanoparticle is estimated to be 5.9 eV. In the photoluminescence spectra the presence of self trapped excitonic state (347 nm) was identified. The third order optical nonlinearity of the sample excited in two different pumping regimes, short pulse excitation (532 nm, 5 ns) and continuous wave laser (532 nm, 50 mW) was studied. The material shows limiting behavior due to two photon absorption in the pulsed laser regime and self defocusing in the continuous wave regime. © 2015 IOP Publishing Ltd.