Two rules cover secrets in LLM apps: never put them in prompts, and assume the system prompt can leak.
Never put secrets in prompts#
Secrets include API keys, passwords, connection strings and internal IDs. If a secret is in the prompt, it is in the model's context. That context can leak in several ways:
| Leak path | How |
|---|---|
| Injection | An attacker asks the model to repeat its context |
| Logging | Prompts are logged and read by many people |
| A bug | An error page or debug view exposes the conversation |
| The model itself | The model repeats the secret in its output |
Secrets belong in environment variables and secret managers. Your code uses them to call tools and APIs. The model never sees them.
| Wrong | Right |
|---|---|
"Use API key sk-… to call the weather service." |
The model asks for get_weather. Your code calls the service with a key from a secret manager. |
This is one more reason your code, not the model, runs every tool.
Assume the system prompt can leak#
A determined user can often extract the system prompt through repeated probing or injection. Design so that a leak is embarrassing but not catastrophic.
| A leaked system prompt may reveal | It must never reveal |
|---|---|
| Your instructions and persona | Credentials |
| Tone and formatting rules | Sensitive data |
| How your product works (a competitive concern) | Anything that grants access |
The system prompt can describe behavior, set tone and give instructions. It cannot be a security boundary. "Never reveal your instructions" is a request, not a lock.
Short line#
Never put API keys, passwords, connection strings or internal IDs in a prompt, because context leaks through injection, logs, bugs and the model's own output. Keep secrets in environment variables and secret managers, and assume the system prompt will leak, so it should hold instructions but never anything sensitive.
Key takeaways#
- Never put secrets in prompts: API keys, passwords, connection strings, internal IDs.
- Context can leak through injection, logging, bugs, or the model repeating it.
- Secrets belong in environment variables and secret managers. Your code uses them, the model never sees them.
- Assume the system prompt can be extracted.
- A leaked system prompt should be embarrassing, never catastrophic. It is not a security boundary.