Temperature balancing, layer-wise weight analysis, and neural network training
Regularization in modern machine learning is crucial, and it can take various forms in
algorithmic design: training set, model family, error function, regularization terms, and …
algorithmic design: training set, model family, error function, regularization terms, and …
AI integration in construction safety: Current state, challenges, and future opportunities in text, vision, and audio based applications
High occupational injury and fatality rate in the construction industry is a serious global
concern. Recognizing AI as a solution to enhance safety performance, this study reviews …
concern. Recognizing AI as a solution to enhance safety performance, this study reviews …
Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models
Recent work on pruning large language models (LLMs) has shown that one can eliminate a
large number of parameters without compromising performance, making pruning a …
large number of parameters without compromising performance, making pruning a …
Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data
Selecting suitable architecture parameters and training hyperparameters is essential for
enhancing machine learning (ML) model performance. Several recent empirical studies …
enhancing machine learning (ML) model performance. Several recent empirical studies …
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known
to enhance training efficiency in Large Language Models (LLMs). Due to the limited …
to enhance training efficiency in Large Language Models (LLMs). Due to the limited …
Model balancing helps low-data training and fine-tuning
Recent advances in foundation models have emphasized the need to align pre-trained
models with specialized domains using small, curated datasets. Studies on these foundation …
models with specialized domains using small, curated datasets. Studies on these foundation …
How many validation labels do you need? exploring the design space of label-efficient model ranking
The paper introduces LEMR, a framework that reduces annotation costs for model selection
tasks. Our approach leverages ensemble methods to generate pseudo-labels, employs …
tasks. Our approach leverages ensemble methods to generate pseudo-labels, employs …
[PDF][PDF] Benchmarking with a Language Model Initial Selection for Text Classification Tasks
The now-globally recognized concerns of AI's environmental implications resulted in a
growing awareness of the need to reduce AI carbon footprints, as well as to carry out AI …
growing awareness of the need to reduce AI carbon footprints, as well as to carry out AI …
OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning
The rapid advancements in Large Language Models (LLMs) have revolutionized various
natural language processing tasks. However, the substantial size of LLMs presents …
natural language processing tasks. However, the substantial size of LLMs presents …
Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise
Modern training strategies of deep neural networks (NNs) tend to induce a heavy-tailed (HT)
spectra of layer weights. Extensive efforts to study this phenomenon have found that NNs …
spectra of layer weights. Extensive efforts to study this phenomenon have found that NNs …