A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

C Zhou, Q Li, C Li, J Yu, Y Liu, G Wang… - arXiv preprint arXiv …, 2023 - arxiv.org
Pretrained Foundation Models (PFMs) are regarded as the foundation for various
downstream tasks with different data modalities. A PFM (eg, BERT, ChatGPT, and GPT-4) is …

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

J Gui, T Chen, J Zhang, Q Cao, Z Sun… - IEEE Transactions on …, 2024 - ieeexplore.ieee.org
Deep supervised learning algorithms typically require a large volume of labeled data to
achieve satisfactory performance. However, the process of collecting and labeling such data …

Self-supervised learning from images with a joint-embedding predictive architecture

M Assran, Q Duval, I Misra… - Proceedings of the …, 2023 - openaccess.thecvf.com
This paper demonstrates an approach for learning highly semantic image representations
without relying on hand-crafted data-augmentations. We introduce the Image-based Joint …

A metaverse: Taxonomy, components, applications, and open challenges

SM Park, YG Kim - IEEE access, 2022 - ieeexplore.ieee.org
Unlike previous studies on the Metaverse based on Second Life, the current Metaverse is
based on the social value of Generation Z that online and offline selves are not different …

Learn from others and be yourself in heterogeneous federated learning

W Huang, M Ye, B Du - … of the IEEE/CVF Conference on …, 2022 - openaccess.thecvf.com
Federated learning has emerged as an important distributed learning paradigm, which
normally involves collaborative updating with others and local updating on private data …

Barlow twins: Self-supervised learning via redundancy reduction

J Zbontar, L Jing, I Misra, Y LeCun… - … on machine learning, 2021 - proceedings.mlr.press
Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large
computer vision benchmarks. A successful approach to SSL is to learn embeddings which …

Pre-training molecular graph representation with 3d geometry

S Liu, H Wang, W Liu, J Lasenby, H Guo… - arXiv preprint arXiv …, 2021 - arxiv.org
Molecular graph representation learning is a fundamental problem in modern drug and
material discovery. Molecular graphs are typically modeled by their 2D topological …

Self-supervised learning in remote sensing: A review

Y Wang, CM Albrecht, NAA Braham… - IEEE Geoscience and …, 2022 - ieeexplore.ieee.org
In deep learning research, self-supervised learning (SSL) has received great attention,
triggering interest within both the computer vision and remote sensing communities. While …

Vicreg: Variance-invariance-covariance regularization for self-supervised learning

A Bardes, J Ponce, Y LeCun - arXiv preprint arXiv:2105.04906, 2021 - arxiv.org
Recent self-supervised methods for image representation learning are based on maximizing
the agreement between embedding vectors from different views of the same image. A trivial …

Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods

R Balestriero, Y LeCun - Advances in Neural Information …, 2022 - proceedings.neurips.cc
Abstract Self-Supervised Learning (SSL) surmises that inputs and pairwise positive
relationships are enough to learn meaningful representations. Although SSL has recently …