A/B testing, also referred to as online controlled experimentation or continuous
experimentation, is a form of hypothesis testing where two variants of a piece of software are …

Finetuning large language models (LLMs) on instructions leads to vast performance
improvements on natural language tasks. We apply instruction tuning using code …

Large Language Models for Code (Code LLM) are flourishing. New and powerful models
are released on a weekly basis, demonstrating remarkable performance on the code …

In this work, we make the first attempt to evaluate LLMs in a more challenging code
generation scenario, ie class-level code generation. We first manually construct the first …

Recently, many large language models (LLMs) have been proposed, showing advanced
proficiency in code generation. Meanwhile, many efforts have been dedicated to evaluating …

The increasing popularity of large language models (LLMs) has paved the way for their
application in diverse domains. This paper proposes a benchmarking framework tailored …

Large language models (LLMs), such as OpenAI's Codex, have demonstrated their potential
to generate code from natural language descriptions across a wide range of programming …

The rapid advancements in pre-trained Large Language Models (LLMs) and Large
Multimodal Models (LMMs) have ushered in a new era of intelligent applications …

The rise of Large Language Models (LLMs) has significantly advanced many applications
on software engineering tasks, particularly in code generation. Despite the promising …

General large language models (LLMs), represented by ChatGPT, have demonstrated
significant potential in tasks such as code generation in software engineering. This has led …