A comprehensive survey on pretrained foundation models: A history from bert to chatgpt
Abstract Pretrained Foundation Models (PFMs) are regarded as the foundation for various
downstream tasks across different data modalities. A PFM (eg, BERT, ChatGPT, GPT-4) is …
downstream tasks across different data modalities. A PFM (eg, BERT, ChatGPT, GPT-4) is …
Pre-trained models for natural language processing: A survey
Recently, the emergence of pre-trained models (PTMs) has brought natural language
processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs …
processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs …
A primer in BERTology: What we know about how BERT works
A Rogers, O Kovaleva, A Rumshisky - Transactions of the Association …, 2021 - direct.mit.edu
Transformer-based models have pushed state of the art in many areas of NLP, but our
understanding of what is behind their success is still limited. This paper is the first survey of …
understanding of what is behind their success is still limited. This paper is the first survey of …
Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers
Pre-trained language models (eg, BERT (Devlin et al., 2018) and its variants) have achieved
remarkable success in varieties of NLP tasks. However, these models usually consist of …
remarkable success in varieties of NLP tasks. However, these models usually consist of …
Mobilebert: a compact task-agnostic bert for resource-limited devices
Natural Language Processing (NLP) has recently achieved great success by using huge pre-
trained models with hundreds of millions of parameters. However, these models suffer from …
trained models with hundreds of millions of parameters. However, these models suffer from …
GoEmotions: A dataset of fine-grained emotions
Understanding emotion expressed in language has a wide range of applications, from
building empathetic chatbots to detecting harmful online behavior. Advancement in this area …
building empathetic chatbots to detecting harmful online behavior. Advancement in this area …
Probing pretrained language models for lexical semantics
The success of large pretrained language models (LMs) such as BERT and RoBERTa has
sparked interest in probing their representations, in order to unveil what types of knowledge …
sparked interest in probing their representations, in order to unveil what types of knowledge …
Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art
A large array of pretrained models are available to the biomedical NLP (BioNLP) community.
Finding the best model for a particular task can be difficult and time-consuming. For many …
Finding the best model for a particular task can be difficult and time-consuming. For many …
Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-
attention relation distillation for task-agnostic compression of pretrained Transformers. In …
attention relation distillation for task-agnostic compression of pretrained Transformers. In …
Structured pruning of large language models
Large language models have recently achieved state of the art performance across a wide
variety of natural language tasks. Meanwhile, the size of these models and their latency …
variety of natural language tasks. Meanwhile, the size of these models and their latency …