LLM guardrails: PII and secrets before egress

An engineer pastes a config file into a chat with a model, asking why the service will not start. The answer comes back correctly, referencing the connection string with the password intact. What the provider received was not the password. It was a placeholder, swapped in on the way out and swapped back on the way in, and the engineer’s screen never showed the difference.

That is the version worth building toward, and it is not where most rollouts start. They start with a policy that says do not paste credentials into AI tools, which no one can enforce, and then with a blocking filter people learn to route around within a month. The decisions that matter are in between: what to catch, what to do when you catch it, and who owns each of those answers.

What should guardrails actually catch?

Vendors pitch a long menu: toxicity, jailbreaks, hallucination scoring, topic filters. The two categories that produce actual incidents are short to list:

• Secrets. API keys, tokens, connection strings, private keys, arriving inside stack traces and “can you debug this” pastes. A leaked credential is a live door into your systems, sitting in a third party’s logs.
• PII. Emails, phone numbers, government IDs, health and financial records: the category with regulatory weight. One pasted customer export can put you in breach-notification territory.

Attachments count as much as typed text. A screenshot of a terminal carries the same credential, and a scanner that only reads the prompt field never sees it.

Block, mask, or log

The verdict matters more than the detection, because the verdict is what your people experience. SecondGuard, the guardrails service in SecondStack, supports three, and choosing between them per category is most of the work.

Blocking stops the request. It is the right answer for categories severe enough that the work genuinely should not proceed. For everything else it backfires, because a person whose work just stopped will find the tool that doesn’t stop it. That is how a control turns into a shadow-AI problem: the traffic is people finishing work, not people exfiltrating.

Masking keeps both properties. The sensitive value is replaced by a typed placeholder before the request leaves, and the original is restored in the response before anything is stored, so the model reasons about a placeholder while the person sees their own data. Placeholders stay stable across requests, so prompt caching keeps working and an agent that echoes one back into a tool call gets the real value.

Logging catches the category without acting on it, which is how you learn what your own traffic contains before switching anything on.

Two designs that get proposed, and why they get switched off

Almost every rollout considers one of these first. Both are reasonable on paper, and both tend to be turned off within a quarter.

1. Regex-only scanning is fast and catches formatted secrets with recognizable prefixes. It falls apart on everything else: a random 40-character token matches no pattern, while commit hashes, base64 fragments and UUIDs pile up as false positives. Users learn that the scanner cries wolf, and routing around it becomes the rational move.
2. An LLM classifier on every message inverts the problem. Detection quality is good, latency is not. A model call on every prompt of every user taxes the whole company all day for the rare message that contains something sensitive. The chat feels slow, people notice, and the pressure to switch it off follows.

Ordinary prose and a leaked API key look different to a tokenizer long before they look different to a person.

What SecondGuard runs instead

Several detectors run side by side on the same request, each cheap where it can be: pattern packs for credentials with known shapes, a statistical check for the random-looking string no pattern claims, named-entity recognition for personal data, and an LLM classifier reserved for what patterns cannot reach, images included. Findings merge and the strictest verdict wins, and repeated content is not rescanned, so agent retries cost nothing.

The classifier sees the message unmasked, which is why it belongs on a model you host or otherwise trust with that content. A guardrails product that ships your prompts to someone else’s cloud for scanning has recreated the data flow you deployed it to prevent.

One limit worth stating plainly: screening runs on the way out. What leaves your network is controlled; filtering what the model sends back is a different problem, and worth asking any vendor which of the two they mean.

Why it belongs at the gateway

Application-level guardrails cover one application. Screening the chat app your team built leaves out every other way a prompt reaches a provider, and there are always several: an IDE plugin, a scheduled script, an agent wired up last week. Nobody can sign off on a control whose scope is “whoever remembered”.

SecondGuard runs as a proxy in the request path, in front of the gateway. The ingress routes every call through it, so a request reaches a provider only after screening, whatever client sent it. If the guard proxy is down, the ingress answers with an error rather than letting traffic through.

The inventory question that follows is worth asking of any deployment, ours included: which paths from your network to a model do not pass through that control point? That list is the real scope of your control, and it is rarely empty.

Where the policy attaches, and who decides

Guards bind globally or per model, which turns out to be the useful axis: the same paste can be logged on the way to a model inside your network and masked on the way to an external provider, because the risk was never the paste, it was the egress.

Two decisions come with that, and neither belongs to the platform team. Which categories count as sensitive is a legal and data-ownership question, and most organizations already have that list in a classification policy nobody has connected to a control. Fail closed or fail open is a security question: fail closed means a scanner outage stops traffic, which is right for regulated paths; fail open lets traffic flow unscanned, which is often right for low-risk internal work. It is set per guard, so the answer can differ by category. Both are defensible; having no named owner for the choice is not.

Someone also has to answer the person who got blocked at 6 p.m., and that duty needs a name before the first block lands.

Start in log-only

Run the guards in log-only mode first, on your own traffic, for a defined window. You get your own false-positive rate instead of a vendor’s benchmark, and you can replay sample text through the real detection path before anything is enforced. Publish what is screened before the first block or mask appears: a control people were told about lands differently than one they discover.

Then set the steady-state expectation, ideally with the security sponsor in the room. Most days the guards find nothing, that quiet is what you are paying for, and a quarterly report showing two findings reads like waste to anyone who was not warned.

If you are scoping guardrails for a rollout, write to [email protected] and we will walk you through what screening looks like in practice: which categories to start in log-only, what masking does to a working session, and where each decision lands in an organization. The hard part is rarely the detector. It is deciding which categories are worth stopping a person’s work over.