Data augmentation, the artificial creation of training data for machine learning by
transformations, is a widely studied research field across machine learning disciplines …

Self-supervised learning has gained popularity because of its ability to avoid the cost of
annotating large-scale datasets. It is capable of adopting self-defined pseudolabels as …

Text embeddings are useful features in many applications such as semantic search and
computing text similarity. Previous work typically trains models customized for different use …

Learning high-quality sentence representations benefits a wide range of natural language
processing tasks. Though BERT-based pre-trained language models achieve high …

Contrastive Learning has recently received interest due to its success in self-supervised
representation learning in the computer vision domain. However, the origins of Contrastive …

Transformer-based pretrained language models (T-PTLMs) have achieved great success in
almost every NLP task. The evolution of these models started with GPT and BERT. These …

State-of-the-art natural language understanding classification models follow two-stages: pre-
training a large language model on an auxiliary task, and then fine-tuning the model on a …

Pre-trained language models have proven their unique powers in capturing implicit
language features. However, most pre-training approaches focus on the word-level training …

Although BERT and its variants have reshaped the NLP landscape, it still remains unclear
how best to derive sentence embeddings from such pre-trained Transformers. In this work …

We present a self-supervised learning framework, COCO-LM, that pretrains Language
Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style …