Linear (spectro) polarimetry is usually performed using separate photon flux measurements after spatial or temporal polarization modulation. Such classical polarimeters are limited in sensitivity and accuracy by systematic effects and noise. We describe a spectral modulation principle that is based on encoding the full linear polarization properties of light in its spectrum. Such spectral modulation is obtained with an optical train of an achromatic quarter-wave retarder, an athermal multiple-order retarder, and a polarizer. The emergent spectral modulation is sinusoidal with its amplitude scaling with the degree of linear polarization and its phase scaling with the angle of linear polarization. The large advantage of this passive setup is that all polarization information is, in principle, contained in a single spectral measurement, thereby eliminating all differential effects that potentially create spurious polarization signals. Since the polarization properties are obtained through curve fitting, the susceptibility to noise is relatively low. We provide general design options for a spectral modulator and describe the design of a prototype modulator. Currently, the setup in combination with a dedicated retrieval algorithm can be used to measure linear polarization signals with a relative accuracy of 5%.