Unlicensed cognitive-radio (CR)-assisted Internet of Vehicles (IoV) users can access licensed providers’ radio spectrum and concurrently utilize the dedicated channel for data transmission in vehicular communication. Optimizing channel access in cognitive IoV networks can help maximize available spectrum resources. This article proposes a novel sensing and communication integrated framework, dubbed as the CR-assisted IoV network (CRAV-Net), using a cluster-based hybrid optimization approach with adaptive congestion-aware modeling for dynamic high-mobility vehicular networks in an urban city context. In CRAV-Net, intelligent hybrid learning spectrum agents are introduced, which perform spectrum sensing (SS) using a deep learning (DL) model. It dynamically learns the multilevel spatial and temporal graphical features from input spectrograms through layer-by-layer propagation. It efficiently predicts the spectrum occupancy in the primary spectrum, without a priori knowledge of the radio environment. Then, to assign the vacant channels to the secondary vehicles, a support vector machine classifier is trained based on several learning features, including the vehicle stay time, vehicle density, and network capacity, to select the optimal resource route. The proposed framework achieves an overall accuracy of 99.74% in SS using the custom data set, outperforming state of the art by 12.60% at −25-dB signal-to-noise ratio. In addition, it brings a performance gain of 0.81% in SS accuracy when evaluated on real-world signals. Furthermore, in optimal network node allocation, the proposed framework achieves a mean accuracy of 98.45%, outperforming the existing methods by 0.63% and 18.32% in terms of accuracy and allocation time, respectively.