The dynamics of an open quantum system exhibiting the quantum Brownian motion is analyzed when the coupling between the system and its environment is nonlinear, and the system and the reservoir are initially correlated. For couplings quadratic in the environment variables, the influence functional for the system is obtained perturbatively up to second order in the coupling constant, and then the propagator is explicitly evaluated when the particle is under the influence of a harmonic potential and an additional anharmonic potential, the so-called washboard potential. As an application of the propagator, the master equation and the Wigner equation are obtained for the quantum Brownian particle moving in a harmonic potential for the generalized correlated initial condition, and then for the specific case of the simplified “thermal” initial condition. The system is shown to obey the corresponding fluctuation-dissipation theorem. © 2003 The American Physical Society.