The attack vector is increasingly big and the rewards are much bigger than before: instead of just databases and root access it will be immediate access to all referenced cloud products, basically nullifying encryption and multi factor authentication
Prompt Injection
Direct Prompt Injection
LLM applications that combine carefully designed prompts with user-provided text are susceptible to prompt injection. Attackers can craft input to override system instructions, causing the model to behave unexpectedly. While this may seem harmless in simple translation apps, the risk increases for systems with greater capabilities, especially those that can act on emails or control external tools, as these could be tricked into taking unauthorized actions or leaking sensitive information.
Command Execution via Content
If an LLM assistant is given control over actions such as sending emails or executing commands, malicious instructions embedded within user-content (like an email) can prompt the assistant to perform harmful operations—such as forwarding sensitive emails to an attacker and deleting evidence—all without the legitimate user’s consent.
Search Index Poisoning
Attackers can manipulate AI-enhanced search engines that read and concatenate web content. By embedding hidden instructions (e.g., in white text on a web page), an attacker can change how a language model describes a person or product. This technique can be used both as a prank and for improper manipulation of model-generated content, creating new vulnerabilities in LLM-optimized search results.
Data Exfiltration Attacks
When LLM applications are granted access to user data or external plugins, attackers can craft prompts to exfiltrate data to third parties. This may involve generating links that send sensitive data to an attacker-controlled site or using features such as plugin-based SQL queries or markdown images to leak information through external URLs.
Indirect Prompt Injection
Indirect attacks involve embedding hidden prompt instructions in content that an LLM might process during its operation (such as invisible text on a web page). These instructions could hijack the agent’s actions and cause it to seek and leak user data to an attacker without the user’s awareness.
source https://simonwillison.net/2023/Apr/14/worst-that-can-happen/
Link to original