How to Manage AI Agents Effectively

🧠 Spend management, governance, testing and safety infrastructure tactics explored. Real world examples, tools and case studies from OpenAI, AWS, Uber, Block, Atlassian, Slack and more.

🔒 The Knowledge Series breaks down emerging AI technologies with practical playbooks designed specifically for product teams. Get 100+ guides and practical tutorials covering everything from Claude Code and MCP to agentic workflows, vibe coding, and more.

---

After Amazon’s employees recently admitted that they mistakenly spent over $1.8 million on an AI feature that used the wrong model, figuring out how to manage what AI agents are doing in production is becoming increasingly important for product teams.

And in recent weeks, the market has shifted to reflect the need for companies to manage the agents they build with dedicated new products and features. LaunchDarkly launched Agent Control, Microsoft is building “agentic observability” tooling for its cloud platforms, Notion added admin controls to govern agents and Vanta revealed an open source framework called Agentic Trust Controls to set the standards in AI agent governance.

If we look back at where things stood even just six months ago, many product execs were still at the stage of identifying opportunities for AI agent involvement in both user-facing features and internal tools. But now, many leading companies have AI Agents up and running across their tech stack.

This has shifted the focus away from figuring out what agents can do to overseeing what they’re actually doing. This can be pretty daunting for product teams and execs who want to ensure agents don’t go rogue or take actions they’re not supposed to.

In this Knowledge Series, we’re going to explore how you can manage AI Agents across important areas like spend management, governance, testing and safety with real world examples and new tools / frameworks you can use from companies including OpenAI, AWS, Block, Atlassian and more.

A preview of the tools, tactics and case studies included

If you’re curious about some of the new, emerging technologies, tools and terminology in this space and you’d like to know how they might apply to your own setup, this Knowledge Series should help.

Coming up:

1. What is AI agent management?

2. The 5 core areas of AI agent management explored, including: spend management, governance and security, identity and access controls, testing and safety and observability and monitoring.

3. 20+ real world examples, case studies and tools you can use to monitor spend, implement governance and manage AI agent identities from the world’s leading companies.

---

The Knowledge Series

What is AI agent management and why does it matter to product teams?

AI Agent management is the operational layer for everything that relates to any agents a company builds. It’s everything a company does to control, observe, and govern an agent once it’s actually running against real users and real systems, as opposed to just designing or prompting the agent in the first place.

Here’s a snapshot of the 5 areas of AI Agent Management that product teams and tech leaders are currently grappling with:

Now that AI agents are working across internal and external products, the conversation is starting to shift away from ideating opportunities for agents and towards answering questions like:

• How much is this agent actually costing us per task, per user, per week?

• Does the agent get its own identity, or does it just inherit the permissions of whoever’s using it?

• Do we need sign off from legal or compliance before an agent goes live, and what does that review actually check?

The 5 core areas of AI Agent management explored

Let’s take a closer look at each of these 5 core areas in more detail.

For each area, we’ll explore some of the new, emerging concepts and terminology worth knowing about - along with real world practical examples of companies who are implementing some of these concepts.

We’ll cover:

1. Spend management

2. Governance and security

3. Identity and access controls

4. Testing and safety

5. Observability and monitoring

1. Spend management

After Uber’s CTO admitted they’d pretty much spent their entire 2026 budget and Amazon’s recent mishap, AI cost spend is top of the agenda for many product leaders right now.

And since this is all moving so quickly, one of the first problems teams run into is understanding what cost control mechanisms actually exist.

Based on what’s available at the moment, managing AI Agent spend typically falls into one of these tactics:

Gateway level budget controls

This is where you monitor and control spend at the infrastructure level; you put a gateway between your agents and the model providers and then set hard limits at that point.

For example, Databricks recently added a new feature called AI Spend Controls to its Unity AI Gateway that lets teams set budget limits and thresholds on both third party API calls and internal model usage.

Vercel has taken a similar approach with its new AI gateway, allowing users to set team and project level budget alerts.

The idea here is that the gateway is the one place all agent traffic has to pass through making it a natural place to enforce a budget limit.

Model / request routing to cut costs

Model routing is where you match the model tier to the actual difficulty of the task, ultimately saving money by not wasting spend on expensive models (something Amazon learned the hard way).