In this paper, we introduce a new subclass BΣζ(m,γ,λ;φ) of bi-univalent functions defined by a new differential operator of analytic functions involving binomial series due to Frasin (Bol Soc Paran Mat (in press), 2019) in the open unit disk. We obtain coefficient bounds for the Taylor–Maclaurin coefficients | a 2 | and | a 3 | of the function f∈BΣζ(m,γ,λ;φ). Furthermore, we solve the Fekete–Szegö functional problem for functions in BΣζ(m,γ,λ;φ). The results presented in this paper improve or generalize the earlier results of Peng and Han (Acta Math Sci 34(1):228–240, 2014) and Tang et al. (J Math Inequal 10(4):1063–1092, 2016). © 2019, African Mathematical Union and Springer-Verlag GmbH Deutschland, ein Teil von Springer Nature.