This paper presents a multichannel functional continuous‐wave near‐infrared spectroscopy (fNIRS) system, which collects data under a dual‐level light intensity mode to optimize SNR for channels with multiple source‐detector separations. This system is applied to classify different cortical activation states of the prefrontal cortex (PFC). Mental arithmetic, digit span, semantic task, and rest state were selected as four mental tasks. A deep forest algorithm is employed to achieve high classification accuracy. By employing multigrained scanning to fNIRS data, this system can extract the structural features and result in higher performance. The proposed system with proper optimization can achieve 86.9% accuracy on the self‐built dataset, which is the highest result compared to the existing systems.