Direct reconstruction of positron emission tomography (PET) data using deep neural networks is a growing field of research. Initial results are promising, but often the networks are complex, memory utilization inefficient, produce relatively small 2-D image slices (e.g., 128 × 128), and low count rate reconstructions are of varying quality. This article proposes FastPET, a novel direct reconstruction convolutional neural network that is architecturally simple, memory space efficient, works for nontrivial 3-D image volumes and is capable of processing a wide spectrum of PET data including low-dose and multitracer applications. FastPET uniquely operates on a histoimage (i.e., image-space) representation of the raw data enabling it to reconstruct 3-D image volumes 67× faster than ordered subsets expectation maximization (OSEM). We detail the FastPET method trained on whole-body and low-dose whole-body data sets and explore qualitative and quantitative aspects of reconstructed images from clinical and phantom studies. Additionally, we explore the application of FastPET on a neurology data set containing multiple different tracers. The results show that not only are the reconstructions very fast, but the images are high quality and have lower noise than iterative reconstructions.