Metastasis detection of lymph nodes in Whole-slide Images (WSIs) plays a critical role in the diagnosis of breast cancer. Automatic metastasis detection is a challenging issue due to the large variance of their appearances and the size of WSIs. Recently, deep neural networks have been employed to detect cancer metastases by dividing the WSIs into small image patches. However, most existing works simply treat these patches independently and do not consider the structural information among them. In this paper, we propose a novel deep neural network, namely Spatially Structured Network (Spatio-Net) to tackle the metastasis detection problem in WSIs. By integrating the Convolutional Neural Network (CNN) with the 2D Long-Short Term Memory (2D-LSTM), our Spatio-Net is able to learn the appearances and spatial dependencies of image patches effectively. Specifically, the CNN encodes each image patch into a compact feature vector, and the 2D-LSTM layers provide the classification results (i.e., normal or tumor), considering its dependencies on other relevant image patches. Moreover, a new loss function is designed to constrain the structure of the output labels, which further improves the performance. Finally, the metastasis positions are obtained by locating the regions with high tumor probabilities in the resulting accurate probability map. The proposed method is validated on hundreds of WSIs, and the accuracy is significantly improved, in comparison with a state-of-the-art baseline that does not have the spatial dependency constraint.