Human detection plays a crucial role in the field of computer vision. Novel methods using new techniques, such as Convolutional Neuron Networks or Recurrent Neuron Networks, can improve accuracy but occupy a large hardware footprint when implemented on FPGA. In contrast, human detection based on the HOG-SVM algorithm has much lower computational complexity, leading to a more hardware-efficient implementation on FPGA. In this paper, we proposed a novel human detection system using a HOG-SVM module in combination with Direct Memory Access (DMA) on Xilinx FPGA. The design was implemented on Xilinx FPGA Development Kit ZCU106 with direct memory access to reduce the CPU load and improve the overall system performance. The proposed system with a low hardware footprint can be utilized in vision-based IoT edge devices.