Optoelectronic synaptic devices capable of processing multiwavelength inputs are critical for neuromorphic vision hardware, which remains an important challenge. Here, we develop a bidirectional synaptic phototransistor based on a two-dimensional ferroelectric semiconductor of α-In₂Se₃, which exhibits bidirectional potentiated and depressed synaptic weight update under optical pulse stimulation. Importantly, the bidirectional optoelectronic synaptic behavior can be extended to multiwavelengths (blue, green, and red light), which could be used for color recognition. The mechanism underlying the bidirectional synaptic characteristics is attributed to the gate-configurable barrier heights as revealed by the Kelvin probe force microscopy measurement. The α-In₂Se₃ device exhibits versatile synaptic plasticity such as paired-pulse facilitation, short- and long-term potentiation, and long-term depression. The bidirectional optoelectronic synaptic weight updates under multiwavelength inputs enable a high accuracy of 97% for mixed color pattern recognition.