Llmseceval: A dataset of natural language prompts for security evaluations

C Tony, M Mutas, NED Ferreyra… - 2023 IEEE/ACM 20th …, 2023 - ieeexplore.ieee.org
Large Language Models (LLMs) like Codex are powerful tools for performing code
completion and code generation tasks as they are trained on billions of lines of code from …

Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation

J Wang, L Cao, X Luo, Z Zhou, J Xie, A Jatowt… - arXiv preprint arXiv …, 2023 - arxiv.org
Large language models (LLMs) have brought significant advancements to code generation,
benefiting both novice and experienced developers. However, their training using …

Generate and pray: Using sallms to evaluate the security of llm generated code

ML Siddiq, J Santos - arXiv preprint arXiv:2311.00889, 2023 - arxiv.org
With the growing popularity of Large Language Models (eg GitHub Copilot, ChatGPT, etc.) in
software engineers' daily practices, it is important to ensure that the code generated by these …

On extracting specialized code abilities from large language models: A feasibility study

Z Li, C Wang, P Ma, C Liu, S Wang, D Wu… - Proceedings of the …, 2024 - dl.acm.org
Recent advances in large language models (LLMs) significantly boost their usage in
software engineering. However, training a well-performing LLM demands a substantial …

Large language models and simple, stupid bugs

K Jesse, T Ahmed, PT Devanbu… - 2023 IEEE/ACM 20th …, 2023 - ieeexplore.ieee.org
With the advent of powerful neural language models, AI-based systems to assist developers
in coding tasks are becoming widely available; Copilot is one such system. Copilot uses …

Security weaknesses of copilot generated code in github

Y Fu, P Liang, A Tahir, Z Li, M Shahin, J Yu - arXiv preprint arXiv …, 2023 - arxiv.org
Modern code generation tools use AI models, particularly Large Language Models (LLMs),
to generate functional and complete code. While such tools are becoming popular and …

Traces of memorisation in large language models for code

A Al-Kaswan, M Izadi, A Van Deursen - Proceedings of the IEEE/ACM …, 2024 - dl.acm.org
Large language models have gained significant popularity because of their ability to
generate human-like text and potential applications in various fields, such as Software …

No need to lift a finger anymore? assessing the quality of code generation by chatgpt

Z Liu, Y Tang, X Luo, Y Zhou… - IEEE Transactions on …, 2024 - ieeexplore.ieee.org
Large language models (LLMs) have demonstrated impressive capabilities across various
natural language processing (NLP) tasks, such as machine translation, question answering …

CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models

H Hajipour, K Hassler, T Holz… - … IEEE Conference on …, 2024 - ieeexplore.ieee.org
Large language models (LLMs) for automatic code generation have recently achieved
breakthroughs in several programming tasks. Their advances in competition-level …

A study on robustness and reliability of large language model code generation

L Zhong, Z Wang - arXiv preprint arXiv:2308.10335, 2023 - arxiv.org
Recently, the large language models (LLMs) have shown extraordinary ability in
understanding natural language and generating programming code. It has been a common …