Ambient backscatter (AB) communications and radio frequency (RF)-powered cognitive radio networks (CRNs) address the concerns of energy and spectrum scarcities from different perspectives, and the integration of them has potential benefits for throughput. Motivated by this fact, we study the RF-powered AB-assisted hybrid underlay CRN (ABHU-CRN), and optimize the long-term secondary throughput. Based on the channel states, secondary users choose to perform different actions, and two action spaces are accordingly designed. Due to dynamic and unpredictable environment states, we jointly control the time scheduling and energy management of secondary users, and propose two algorithms, i.e., adjusted-deep deterministic policy gradient (A-DDPG) and combination of A-DDPG and convex optimization (C-ADCO), for the long-term secondary throughput. A-DDPG, a deep reinforcement learning algorithm with continuous spaces, is extended from DDPG to adapt to the design of two action spaces. C-ADCO utilizes the convex optimization that can find the optimal solution to assist A-DDPG to accelerate the convergence. In simulations, the ABHU-CRN under A-DDPG and C-ADCO achieves higher throughput than the optimal throughput of AB-assisted overlay CRN and AB-assisted underlay CRN, which indicates the advantage of the hybrid transmission mode in the ABHU-CRN.