Controlled synthesis pure-phase Bi0.98Tb0.02Fe1−xCoxO3 (BTFCO, x = 0.01, 0.03, 0.05 and 0.07) nanoparticles with rhombohedral structure using a facile sol–gel method opens the doors to robust their magnetic and photocatalytic properties significantly. The microstructure analysis revealed improvement in the surface morphology with a decrease of average particle size from 64 to 38 nm. No mixed valance state of iron detected for any sample, which is confirmed by Mössbauer spectra. The saturation magnetization and remnant magnetization were increased dramatically from 0.802 and 0.118 emu/g for BFO to 2.261 and 0.737 emu/g, respectively by Tb, Co co-doping. The enhancement of the magnetic property is attributed to the breakdown of spin cycloidal structure, spin canting, and uncompensated spin on the surface of nanoparticles. Furthermore, the as-synthesized samples are highly effective for rhodamin B (RhB) degradation under visible light irradiation, and the possible photocatalytic mechanism discussed in detail. Bi0.98Tb0.02Fe0.93Co0.07O3 nanoparticles exhibit the best photocatalytic activity owing to their larger specific surface area, lower band gap, and reduced average particle size. Besides, 1.2% efficiency retention for the degradation of RhB dye using BTFCO7 after the 20th cycle suggests a potential candidate to address the current scenario in the environmental remediation technology. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.