We experimentally investigate the effect of a magnetic field on photon detection in superconducting single-photon detectors (SSPDs). At low fields, the effect of a magnetic field is through the direct modification of the quasiparticle density of states of the superconductor, and magnetic field and bias current are interchangeable, as is expected for homogeneous dirty-limit superconductors. At the field where a first vortex enters the detector, the effect of the magnetic field is reduced, up until the point where the critical current of the detector starts to be determined by flux flow. From this field on, increasing the magnetic field does not alter the detection of photons anymore, whereas it does still change the rate of dark counts. This result points at an intrinsic difference in dark and photon counts, and also shows that no enhancement of the intrinsic detection efficiency of a straight SSPD wire is achievable in a magnetic field.