Owasp LLM

title: 'LLM05: Improper Output Handling' description: OWASP LLM Top 10 reference for improper output handling vulnerabilities including injection via LLM-generated content

05 Improper Output Handling

Identifier: LLM05:2025 Category: Output Safety

Description

Improper Output Handling refers to insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems. Since LLM-generated content can be controlled by prompt input, this behavior is similar to providing users indirect access to additional functionality.

Improper Output Handling differs from Overreliance in that it deals with LLM-generated outputs before they are passed downstream whereas Overreliance focuses on broader concerns around overdependence on the accuracy and appropriateness of LLM outputs.

Successful exploitation can result in XSS and CSRF in web browsers, SSRF, privilege escalation, or remote code execution on backend systems.

The following conditions increase the impact of this vulnerability: granting the LLM privileges beyond what is intended for end users, vulnerability to indirect prompt injection attacks, third-party extensions that do not adequately validate inputs, lack of proper output encoding for different contexts, insufficient monitoring and logging of LLM outputs, and absence of rate limiting or anomaly detection for LLM usage.

Risk

• Cross-site scripting (XSS) and CSRF in web browsers from unescaped LLM-generated content.

• Server-side request forgery (SSRF) through unvalidated output used in server requests.

• Privilege escalation through LLM capabilities exceeding intended end user access.

• Remote code execution on backend systems via output passed to system shells or eval functions.

• SQL injection through unparameterized LLM-generated queries executed against databases.

• Path traversal through unsanitized file path construction from LLM output.

• Phishing attacks through unescaped LLM-generated content used in email templates.

• Insecure code generation leading to vulnerabilities when developers integrate unverified LLM code suggestions.

Vulnerability checklist

• LLM output is entered directly into system shells or functions such as exec or eval without validation.

• JavaScript or Markdown generated by the LLM is returned to users and interpreted by the browser without encoding.

• LLM-generated SQL queries are executed without proper parameterization.

• LLM output is used to construct file paths without proper sanitization.

• LLM-generated content is used in email templates without proper escaping.

• The application grants the LLM privileges beyond what is intended for end users.

• Third-party extensions do not adequately validate LLM-generated inputs.

• No proper output encoding is applied for different downstream contexts such as HTML, JavaScript, or SQL.

• Monitoring and logging of LLM outputs is insufficient to detect exploitation attempts.

• No Content Security Policy is in place to mitigate XSS from LLM-generated content.

Prevention controls

1. Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions.

2. Follow the OWASP ASVS (Application Security Verification Standard) guidelines to ensure effective input validation and sanitization.

3. Encode model output back to users to mitigate undesired code execution by JavaScript or Markdown. OWASP ASVS provides detailed guidance on output encoding.

4. Implement context-aware output encoding based on where the LLM output will be used, such as HTML encoding for web content and SQL escaping for database queries.

5. Use parameterized queries or prepared statements for all database operations involving LLM output.

6. Employ strict Content Security Policies (CSP) to mitigate the risk of XSS attacks from LLM-generated content.

7. Implement robust logging and monitoring systems to detect unusual patterns in LLM outputs that might indicate exploitation attempts.

Example attack scenarios

Scenario A — Extension shutdown via unvalidated output

An application utilizes an LLM extension to generate responses for a chatbot feature. The extension also offers administrative functions accessible to another privileged LLM. The general purpose LLM directly passes its response without proper output validation to the extension, causing the extension to shut down for maintenance.

Scenario B — Data exfiltration via summarizer

A user utilizes a website summarizer tool powered by an LLM to generate a concise summary of an article. The website includes a prompt injection instructing the LLM to capture sensitive content from either the website or from the user's conversation. The LLM encodes the sensitive data and sends it without any output validation or filtering to an attacker-controlled server.

Scenario C — SQL execution via chat

An LLM allows users to craft SQL queries for a backend database through a chat-like feature. A user requests a query to delete all database tables. If the crafted query from the LLM is not scrutinized, all database tables are deleted.

Scenario D — XSS via unsanitized content

A web app uses an LLM to generate content from user text prompts without output sanitization. An attacker submits a crafted prompt causing the LLM to return an unsanitized JavaScript payload, leading to XSS when rendered on a victim's browser.

Scenario E — Malicious email template injection

An LLM is used to generate dynamic email templates for a marketing campaign. An attacker manipulates the LLM to include malicious JavaScript within the email content. If the application does not properly sanitize the LLM output, this leads to XSS attacks on recipients who view the email in vulnerable email clients.

Scenario F — Insecure code generation

An LLM is used to generate code from natural language inputs in a software company. The approach risks exposing sensitive information, creating insecure data handling methods, or introducing vulnerabilities like SQL injection. The AI may also hallucinate non-existent software packages, potentially leading developers to download malware-infected resources.

Detection guidance

• Review code for direct use of LLM output in system commands, SQL queries, or file path construction.

• Test for XSS by examining LLM-generated content rendered in browsers for unescaped JavaScript or Markdown.

• Inspect Content Security Policies for adequacy against LLM-generated content injection.

• Monitor logging systems for unusual patterns in LLM outputs indicating exploitation attempts.

• Check all database operations involving LLM output for parameterized query usage.

• Audit third-party extensions for adequate input validation of LLM-generated content.

Remediation

• Replace all direct use of LLM output in system commands with properly validated and sanitized alternatives.

• Implement context-aware output encoding for all LLM output based on the downstream consumer.

• Replace all unparameterized queries involving LLM output with parameterized queries or prepared statements.

• Deploy strict Content Security Policies to mitigate XSS from LLM-generated content.

• Add robust logging and monitoring to detect and respond to exploitation attempts in LLM outputs.

• Conduct thorough code review and verification of all LLM-suggested packages and code before integration.

Content derived from works by the OWASP Foundation, licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). Modifications: Restructured into agent-consumable reference format with added detection and remediation guidance.