The 'Rayleigh line' μ = η2, where μ = Ωo/Ωi and η = ri/ro are respectively the rotation and radius ratios between inner (subscript 'i') and outer (subscript 'o') cylinders, is regarded as marking the limit of centrifugal instability (CI) in unstratified inviscid Taylor-Couette flow, for both axisymmetric and non-axisymmetric modes. Non-axisymmetric stratorotational instability (SRI) is known to set in for anticyclonic rotation ratios beyond that line, i.e. η2 < μ < 1 for axially stably-stratified Taylor-Couette flow, but the competition between CI and SRI in the range μ < η2 has not yet been addressed. In this paper, we establish continuous connections between the two instabilities at flnite Reynolds number Re, as previously suggested by M. Le Bars & P. Le Gal, Phys. Rev. Lett.. 99, 064502 (2007), making them indistinguishable at onset. Both instabilities are also continuously connected to the radiative instability at finite Re. These results demonstrate the complex impact viscosity has on the linear stability properties of this flow. Several other qualitative differences with inviscid theory were found, among which the instability of a non-axisymmetric mode localized at the outer cylinder without stratification, and the instability of a mode propagating against the inner cylinder rotation with stratification. The combination of viscosity and stratiflcation can also lead to a collision' between (axisymmetric) Taylor vortex branches, causing the axisymmetric oscillatory state already observed in past experiments. Perhaps more surprising is the instability of a centrifugal-like helical mode beyond the Rayleigh line, caused by the joint effects of stratification and viscosity. The threshold μ = η2 seems to remain, however, an impassable instability limit for axisymmetric modes, regardless of stratification, viscosity, and even disturbance amplitude.