The dynamics of rotary wing systems are complex and typically feature highly nonlinear and often unsteady aerodynamics, as well as aeroelastic influences. In ongoing efforts to reduce noise and vibration, active devices such as trailing edge flaps on the rotor blades are being studied and these devices can introduce further nonlinearities. Therefore, it is important to be able to evaluate the stability of the overall system with a proper understanding of the global nonlinear behavior. Numerical continuation and bifurcation analysis is well suited to this need, and this paper presents evidence of the technique providing a deeper insight into the stability of helicopter rotor systems than the methods typically adopted in the industry. We first investigate the aeroelastic stability of rotor blades of a medium-sized helicopter in hover and the periodically forced forward flight condition, in both trimmed and untrimmed cases. Then, bifurcation analysis is used to predict the nonlinear stability of a single degree-of-freedom trailing edge flap added to the aeroelastic system, over a range of design parameters. The approach is novel in the context of real-world aeroelastic rotor models, and the emphasis here is on the potential for revealing important multiple-attractor dynamics rather than the study of a particular system. The results presented highlight the advantages of the approach, both in terms of generating an understanding of local and more global stability, and in the efficiency in obtaining relevant results as parameters vary.