While originally designed for natural language processing tasks, the self-attention
mechanism has recently taken various computer vision areas by storm. However, the 2D …

Abstract Knowledge distillation has been applied to various tasks successfully. The current
distillation algorithm usually improves students' performance by imitating the output of the …

In deep learning, different kinds of deep networks typically need different optimizers, which
have to be chosen after multiple trials, making the training process inefficient. To relieve this …

Abstract Knowledge Distillation (KD) uses the teacher's prediction logits as soft labels to
guide the student, while self-KD does not need a real teacher to require the soft labels. This …

Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is extensively studied
as a way to boost the performance of a small model. Recently, Vision Transformer (ViT) has …

Post-training quantization (PTQ) is an effective compression method to reduce the model
size and computational cost. However, quantizing a model into a low-bit one, eg, lower than …

While large scale pre-training has achieved great achievements in bridging the gap
between vision and language, it still faces several challenges. First, the cost for pre-training …

Abstract Deep Neural Networks (DNNs) have obtained impressive performance across
tasks, however they still remain as black boxes, eg, hard to theoretically analyze. At the …

With the remarkable progress of deep neural networks in computer vision, data mixing
augmentation techniques are widely studied to alleviate problems of degraded …

Given a pre-trained in-distribution (ID) model, the inference-time out-of-distribution (OOD)
detection aims to recognize OOD data during the inference stage. However, some …