We investigate the evolution of polarization of a soliton pulse with quintic self and cross phase modulation effects in birefringent nonlinear fiber. To characterize the state of polarization of a soliton pulse, we derive a system of integro-differential equations. The solitary wave solutions are obtained to discuss the different state of polarization. Moreover, after constructing suitable conserved densities, we derive the undamped Duffing oscillator equation from the evolution of the Stokes parameters. The stability analysis is carried out by both analytical and numerical methods. © 2002 Published by Elsevier Science B.V.