Malignant melanoma is one of the rare but deadliest types of skin cancers. Clinically, the early diagnosis of this disease is based on human visual inspection with dermoscopy imaging. However, human observations are subjective and prone to errors due to huge variations within dermoscopy images. To address it, we propose a framework for automatic skin lesion recognition using cross-net based aggregation of multiple convolutional networks. The output activation maps of each network are extracted as indicator maps to select the local deep convolutional descriptors (i.e., local patterns and color) in dermoscopy images. Also, this map of a convolutional layer captures the semantic regions of the input image and localizes the target object. These selected features are aggregated into an informative feature map, which are potentially better preserved in the convolutional feature maps. Finally, we use Fisher vector (FV) to encode the selected features. Extensive experiments demonstrate the effectiveness of our proposed method. Comparing with aggregation strategy using pooling approaches, the proposed method learns more robust and discriminative representations based on the publicly available skin lesion challenge datasets from the International Symposium on Biomedical Imaging (ISBI) 2016 and 2017.