Long-duration observations of Neptune's brightness at two visible wavelengths provide a disk-averaged estimate of its atmospheric aerosol. Brightness variations were previously associated with the 11-year solar cycle, through solar-modulated mechanisms linked with either ultraviolet or galactic cosmic ray (GCR) effects on atmospheric particles. Here, we use a recently extended brightness data set (1972-2014), with physically realistic modelling to show, rather than alternatives, ultraviolet and GCR are likely to be modulating Neptune's atmosphere in combination. The importance of GCR is further supported by the response of Neptune's atmosphere to an intermittent 1.5-to 1.9-year periodicity, which occurred preferentially in GCR (not ultraviolet) during the mid-1980s. This periodicity was detected both at Earth, and in GCR measured by Voyager 2, then near Neptune. A similar coincident variability in Neptune's brightness suggests nucleation onto GCR ions. Both GCR and ultraviolet mechanisms may occur more rapidly than the subsequent atmospheric particle transport.