Holistic evaluation of language models

P Liang, R Bommasani, T Lee, D Tsipras… - arXiv preprint arXiv …, 2022 - arxiv.org
Language models (LMs) are becoming the foundation for almost all major language
technologies, but their capabilities, limitations, and risks are not well understood. We present …

Bloom: A 176b-parameter open-access multilingual language model

T Le Scao, A Fan, C Akiki, E Pavlick, S Ilić, D Hesslow… - 2023 - inria.hal.science
Large language models (LLMs) have been shown to be able to perform new tasks based on
a few demonstrations or natural language instructions. While these capabilities have led to …

Auditing large language models: a three-layered approach

J Mökander, J Schuett, HR Kirk, L Floridi - AI and Ethics, 2023 - Springer
Large language models (LLMs) represent a major advance in artificial intelligence (AI)
research. However, the widespread use of LLMs is also coupled with significant ethical and …

Sources of irreproducibility in machine learning: A review

OE Gundersen, K Coakley, C Kirkpatrick… - arXiv preprint arXiv …, 2022 - arxiv.org
Background: Many published machine learning studies are irreproducible. Issues with
methodology and not properly accounting for variation introduced by the algorithm …

Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction

R Shelby, S Rismani, K Henne, AJ Moon… - Proceedings of the …, 2023 - dl.acm.org
Understanding the landscape of potential harms from algorithmic systems enables
practitioners to better anticipate consequences of the systems they build. It also supports the …

Model evaluation for extreme risks

T Shevlane, S Farquhar, B Garfinkel, M Phuong… - arXiv preprint arXiv …, 2023 - arxiv.org
Current approaches to building general-purpose AI systems tend to produce systems with
both beneficial and harmful capabilities. Further progress in AI development could lead to …

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

S Kapoor, EM Cantrell, K Peng, TH Pham, CA Bail… - Science …, 2024 - science.org
Machine learning (ML) methods are proliferating in scientific research. However, the
adoption of these methods has been accompanied by failures of validity, reproducibility, and …

Harms from increasingly agentic algorithmic systems

A Chan, R Salganik, A Markelius, C Pang… - Proceedings of the …, 2023 - dl.acm.org
Research in Fairness, Accountability, Transparency, and Ethics (FATE) 1 has established
many sources and forms of algorithmic harm, in domains as diverse as health care, finance …

The foundation model transparency index

R Bommasani, K Klyman, S Longpre, S Kapoor… - arXiv preprint arXiv …, 2023 - arxiv.org
Foundation models have rapidly permeated society, catalyzing a wave of generative AI
applications spanning enterprise and consumer-facing contexts. While the societal impact of …

Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data

A Balagopalan, D Madras, DH Yang… - Science …, 2023 - science.org
As governments and industry turn to increased use of automated decision systems, it
becomes essential to consider how closely such systems can reproduce human judgment …