The manipulation of the polarization properties of light in guided media is crucial in many classical and quantum optical systems. However, the capability of current technology to finely define the state of polarization of particular wavelengths is far from the level of maturity in amplitude control. Here, we introduce a light-by-light polarization control mechanism with wavelength selectivity based on the change of the phase retardance by means of stimulated Brillouin scattering. Experiments show that any point on the Poincaré sphere can be reached from an arbitrary input state of polarization with little variation of the signal amplitude (<2.5dB). Unlike other Brillouin processing schemes, the degradation of the noise figure is small (1.5 dB for a full 2π rotation). This all-optical polarization controller can forge the development of new polarization-based techniques in optical communication, laser engineering, sensing, quantum systems, and light-based probing of chemical and biological systems.