Most enterprise AI governance conversations happen at the policy document level. Acceptable use policies, AI ethics principles, data handling guidelines. The infrastructure question, how those policies get technically enforced across the actual agent deployments running in production, is often still unanswered.
The Document-to-Enforcement Gap
An IT governance team at a mid-size financial services company spent several months in 2025 developing a comprehensive AI acceptable use policy. The document was thorough. It specified which data categories could be included in LLM prompts, which use cases required security review, and what disclosure requirements applied when AI was used in customer-facing workflows.
The gap was that the policy had no technical enforcement mechanism. Teams were expected to read it and comply. Whether any particular AI agent was actually following the data handling requirements was an honor-system question. The policy existed. The enforcement did not.
This is not an unusual situation. Writing a governance policy and deploying a governance infrastructure are two different projects with two different timelines. Many organizations are partway through both without completing either.
What Controls Are Actually in Place
From the IT governance perspective, the controls that tend to be working in practice fall into a few categories. Not every organization has all of them, but the pattern of what is most commonly implemented first is fairly consistent.
Model approval lists. Many IT teams have established a list of approved LLM vendors and models. Developers are expected to use only approved options. This is a policy control, not a technical one, but it is the most commonly implemented first step because it requires no infrastructure. The limitation is the same as any policy control: it only governs what people follow voluntarily.
Identity management for API keys. Organizations with mature IAM practices have started requiring that AI API keys be provisioned through the central identity management system rather than created by individual developers. This gives IT a registry of what keys exist and which teams own them. It does not tell you what those keys are being used for or at what cost.
Cloud-level budget alerts. For managed services like Azure OpenAI and Amazon Bedrock, cloud billing alerts are straightforward to configure. IT teams use these to receive notifications when spend in a particular subscription crosses a threshold. This is reactive, not preventive, but it establishes a floor of visibility for cloud-routed spend.
Routing-layer inspection. A smaller number of organizations, typically those with higher AI adoption or more stringent data handling requirements, have deployed a proxy or gateway that intercepts AI API traffic. This is where technical enforcement of data policy becomes possible. When every request passes through an inspection layer, PII detection and topic restriction can be applied uniformly, regardless of which team shipped the agent or what they configured in the agent itself.
Why Technical Enforcement Matters More Than Policy at Scale
There is a point in every organization's AI adoption curve where the volume and variety of AI agent deployments exceeds what policy-based governance can reliably cover. That point arrives earlier than most IT teams expect.
When there are two or three AI agents in the organization, a personal check from the IT team on each one is feasible. When there are twelve, a quarterly review is manageable with effort. When there are forty, policy compliance becomes a function of team culture and individual developer awareness, neither of which is auditable.
Technical enforcement does not replace policy. It makes policy enforceable at scale. A data classification rule that says "do not include customer PII in outbound prompts" is a policy. A PII detection layer at the routing level that masks customer PII before the request leaves the network is enforcement. Both are necessary. Only the second one works when you have fifty agents running without individual supervision.
The Specific Controls That Work in Practice
Based on how the problem presents in real IT environments, the controls that provide meaningful risk reduction cluster around four areas.
Centralized request routing. Requiring all AI API calls to route through a single endpoint gives IT complete request-level visibility. Every request is logged with metadata: timestamp, team, model, token count, cost, response latency. Audit log export is available for compliance review. This is the foundation that other controls layer on top of.
Data classification at the transit layer. PII detection and masking at the routing layer catches data that should not be in outbound prompts, regardless of how the agent was built. This is particularly important for customer-facing applications where support agents or document processing agents have access to identifiable customer data.
Team namespace enforcement. Every request is tagged to a team namespace. This enables cost attribution by team and creates the data structure needed for budget controls. When a team's monthly spend approaches a configured threshold, alerts can route to the team lead and to IT before the cost crosses the limit.
Model and vendor allowlisting at the routing layer. Rather than relying on developers to use only approved models, the routing layer can enforce model allowlists. Requests targeting an unapproved model are either redirected to the approved equivalent or blocked with a clear error message that tells the developer which models are available.
Getting Governance in Place Without Slowing Teams Down
One practical concern IT teams raise is that adding a governance layer to AI API calls will slow down development teams or require them to refactor their agent code. This concern is valid but addressable.
The refactoring burden for routing-layer governance is minimal: teams change the API endpoint in their configuration from the direct LLM API URL to the control plane endpoint. That is typically a one-line environment variable change. The agent code does not change. The API interface the agent calls remains the same (OpenAI-compatible interfaces are standard across most routing layers).
The governance controls themselves, data classification rules, model allowlists, budget thresholds, run at the infrastructure level and are configured by IT, not by the agent team. The team does not need to implement PII masking in their agent. IT configures it once at the platform level and it applies to all traffic.
This is the right division of responsibility. Teams own what their agents do. IT owns what happens to the traffic before it leaves the network. Those are different concerns that belong to different layers.
Where Policy and Technical Controls Should Connect
This is not a case where technical controls replace governance policy. They should connect. A well-designed governance setup has the policy document specify the requirements (which data categories require masking, which model vendors are approved, what the budget allocation process is), and the technical controls enforce those requirements without manual intervention per deployment.
The practical outcome is that when a new team deploys a new AI agent, the governance posture does not require that team to independently configure their agent to comply with the organization's data policy. It requires them to point their agent at the control plane endpoint, and the governance that IT has configured applies automatically.
That is a meaningfully different governance model from the one that relies on developers reading a policy document and doing the right thing. Both models require the right people to care. Only one of them works when you are governing 40 agents built by 12 teams over 18 months, each making their own implementation decisions.