Foundational challenges in assuring alignment and safety of large language models
This work identifies 18 foundational challenges in assuring the alignment and safety of large
language models (LLMs). These challenges are organized into three different categories …
language models (LLMs). These challenges are organized into three different categories …
The instruction hierarchy: Training llms to prioritize privileged instructions
Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow
adversaries to overwrite a model's original instructions with their own malicious prompts. In …
adversaries to overwrite a model's original instructions with their own malicious prompts. In …
Are you still on track!? Catching LLM Task Drift with Activations
Large Language Models (LLMs) are routinely used in retrieval-augmented applications to
orchestrate tasks and process inputs from users and other sources. These inputs, even in a …
orchestrate tasks and process inputs from users and other sources. These inputs, even in a …
AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
AI agents aim to solve complex tasks by combining text-based reasoning with external tool
calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned …
calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned …
SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors
Evaluating aligned large language models'(LLMs) ability to recognize and reject unsafe user
requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts …
requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts …