Incumbent wireless technologies for futuristic fifth generation (5G) and beyond 5G (B5G) networks, such as IEEE 802.11 ax (WiFi), are vital to provide ubiquitous ultra-reliable and low-latency communication services with massively connected devices. Amalgamating WiFi networks with 5G/B5G networks has attracted strong researcher interest over the past two decades, because over 70 percent of mobile data traffic is generated by WiFi devices. However, WiFi channel resource scarcity for 5G/B5G is becoming ever more critical. One current problem regarding channel resource allocation is channel collision handling due to increased user densities. Reinforcement learning (RL) algorithms have recently helped develop prominent behaviorist learning techniques for resource allocation in 5G/B5G networks. An agent optimizes its behavior in an RL-based algorithm based on reward and accumulated value. However, densely deployed WiFi environments are distributed and dynamic, with frequent changes. Thus, relying on individual local estimations leads to higher error variance. Therefore, this article proposes a federated RL-based channel resource allocation framework for 5G/B5G networks, and suggests collaborating learning estimates for faster learning convergence. Experimental results verify that the proposed approach optimizes WiFi performance in terms of throughput by collaborative channel access parameter selection. This study also highlights six potential applications for the proposed framework.