Experimental studies have shown that the boundary-layer flow over a rotating cone is susceptible to cross-flow and centrifugal instability modes of spiral nature, depending on the cone sharpness. For half-angles (psi) ranging from propeller nose cones to rotating disks (psi >= 40 degrees), the instability triggers co-rotating vortices, whereas for sharp spinning missiles (psi <40 degrees), counter-rotating vortices are observed. In this paper we provide a mathematical description of the onset of co-rotating vortices for a family of cones rotating in quiescent fluid, with a view towards explaining the effect of psi on the underlying transition of dominant instability. We investigate the stability of inviscid cross-flow modes (type I) as well as modes which arise from a viscous Coriolis force balance (type II), using numerical and asymptotic methods. The influence of psi on the number and orientation of the spiral vortices is examined, with comparisons drawn between our two distinct methods as well as with previous experimental studies. Our results indicate that increasing psi has a stabilizing effect on both the type I and type II modes. Favourable agreement is obtained between the numerical and asymptotic methods presented here and existing experimental results for psi > 40 degrees. Below this half-angle we suggest that an alternative instability mechanism is at work, which is not amenable to investigation using the formulation presented here.