We present an ultrabroadband, high-speed wavelength-swept source based on a self-modulated femtosecond oscillator. Photonic crystal fiber was pumped by a mode-locked Yb:CaF2 laser, resulting in a strong spectral broadening from 485 to 1800 nm. The pump laser cavity could be realigned in order to achieve total mode-locking of the longitudinal and transverse TEM00 and TEM01 electromagnetic modes. This led to spatial oscillations of the output beam, which induced modulation of the coupling efficiency to the fiber. Due to the fact that nonlinear spectral broadening was intensity dependent, this mechanism introduced wavelength sweeping at the fiber output. The sweeping rate could be adjusted between 7 and 21.5 MHz by changing the geometry of the pump cavity. By controlling the ratio of the transverse mode amplitudes, we were able to tune the sweeping bandwidth, eventually covering both the 1300 nm and 1700 nm bioimaging transparency windows. When compared with previously demonstrated wavelength-swept sources, our concept offers much broader tunability and higher speed. Moreover, it does not require an additional intensity modulator.