Knowledge distillation (KD) has become a well established paradigm for compressing deep
neural networks. The typical way of conducting knowledge distillation is to train the student …

Data-free knowledge distillation (DFKD) has recently been attracting increasing attention
from research communities, attributed to its capability to compress a model only using …

In the last decade, many deep learning models have been well trained and made a great
success in various fields of machine intelligence, especially for computer vision and natural …

Implicit discourse relation recognition (IDRR) remains a challenging task in discourse
analysis due to the absence of connectives. Most existing methods utilize one-hot labels as …

This paper reveals a new appeal of the recently emerged large-kernel Convolutional Neural
Networks (ConvNets): as the teacher in Knowledge Distillation (KD) for small-kernel …

Abstract Data-free Knowledge Distillation (DFKD) has attracted attention recently thanks to
its appealing capability of transferring knowledge from a teacher network to a student …

Continual pre-training has been urgent for adapting a pre-trained model to a multitude of
domains and tasks in the fast-evolving world. In practice, a continually pre-trained model is …

Though Self-supervised learning (SSL) has been widely studied as a promising technique
for representation learning, it doesn't generalize well on long-tailed datasets due to the …

Knowledge Distillation (KD) is a typical method for training a lightweight student model with
the help of a well-trained teacher model. However, most KD methods require access to …

Abstract Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some pre-
trained networks for quickly building numerous networks with different complexity and …