🔵 Amazon’s “Catastrophically Expensive” Mistake and the New Tools to Manage AI Token Spend
Plus: Spotify’s context-aware UI, Replit Design and how Airbnb does evals
Hi product people 👋,
This week, it emerged that Amazon mistakenly spent $1.8 million on Anthropic models for a feature that ran 860% over its allocated budget. Needless to say that managing token spend is becoming increasingly important for tech companies. We’ll take a closer look at some new tools you can use to do it.
Plus, Spotify gets new context-aware UI patterns, Replit enters the design space and the war on AI slop continues.
Key reads and resources for product teams
New from the Department of Product Substack this week:
How Netflix built a generative AI powered homepage that boosts engagement
Netflix built GenPage, an AI system that builds your entire homepage in one pass instead of separate components that know nothing about each other. The result is a page that’s 20% faster to load and measurably higher engagement. Learn more about how Netflix built it, the four core technologies powering it, and 15+ case studies from Spotify, DoorDash, and LinkedIn showing how you can apply similar AI personalization to your own product. (Department of Product)
How Airbnb involves PMs in the evals process before shipping new AI features
In a new post on the Airbnb Tech Blog, Rohit Girme and team reveal how they treat evaluation as core discipline by combining programmatic checks, AI judges, and human reviews by PMs to catch real failure modes before they reach users. Learn their five-principle framework for building trustworthy GenAI products at scale. (Airbnb)
Linear’s CPO shares the most common loop written by product teams
Linear’s Chief Product Officer explains what the most common loop created in the Linear app is - and why it misses bugs 30% of the time. (X)
Are Lean Startup principles still relevant? Stripe’s CEO Patrick Collison shares his views
In this conversion with Y Combinator, Collison suggests that while iterative building remains important, the ease of spinning up organizations with AI means founders can now pursue more ambitious, divergent ideas from day one. (Y Combinator)
9 downloadable AI skills you can use across the different layers of product design
Created by Jamie Mill, these tool-agnostic skills install once and run in any AI environment. You pick a layer, run a skill, get structured guidance. (Layers by Jamie Mill)
New product features you can use at work
Chrome now has a Gemini Spark integration. With your permission, it can access your logged in accounts and saved passwords to complete tasks on your behalf. Google calls these “errands” and for tech workers, this could include things we’d rather not do like procurement or checking up on a competitor homepage. OpenAI recently binned its Atlas browser and so it looks like we’re very much still in the age of Chrome browser supremacy.
Miro has launched ‘Talktrack 2.0’, a new update to its async video tool and rival to Loom. The updates let you edit layers, add multiple product team members into the same talk track and AI features that will build diagrams directly from the Talk track you recorded.
GitHub launched “Canvas” which lets you turn AI into interactive workspaces where you can collaborate with agents on shared visuals. Product teams can use this for things like creating swipeable cards to track issues and creating architectural diagrams:
Linear’s mobile app now supports full code reviews so your engineering teams can monitor reviews on the go.
Emerging trends that matter to product teams this week
Here are some of the emerging trends from across the industry this week based on what’s been shipped and new product market signals.
1/ AI spending control tools are accelerating after Amazon’s “catastrophically expensive” mistake
Amazon staff revealed that the company wasted $1.8 million by using Anthropic’s Claude Sonnet model to power a feature that matches author details with product listings, which equates to an 860% overspend vs its allocated budget. You could reasonably argue that $1.8 million is a drop in the ocean by Amazon’s standards but the company is well known for its cost-saving culture and one engineer told the FT that it was “difficult to figure out” how much AI features are costing.
Amazon is unlikely to be the only company to have suffered a spending blunder like this and the market is responding accordingly. This week saw the release of a bunch of new features designed to make it easier to track AI spend:
Ramp has launched token spend management - a new tool designed to help companies track and manage AI spend. It unifies cost data from all major AI vendors into a single dashboard.
Vercel shipped budget controls in its AI gateway, which can be scoped to both the team and project level.
Databricks added a spend control feature to its Unity AI Gateway which covers third party API calls and internal model usage.
OpenAI released Admin APIs for ChatGPT work which give admins a full view of vendor spend and usage limits.
2/ Authenticity is becoming a design/content signal as the war on AI slop continues
The war on AI generated slop is continuing - and product teams are shipping features that prioritise authenticity over AI content.
Following the launch of Substack’s AI detection tool a few weeks back, Snapchat has now confirmed it is changing its algorithm to favor human made content over other formats in a post entitled “Rewarding Authentic Creativity”. This week, LinkedIn followed suit with a new user reporting tool that lets users flag posts that “seem like AI slop”.
OpenAI is also filtering ChatGPT to block direct copying of named authors’ styles which is an indirect way to block users from creating content that looks and feels like another human (in this case an author).
3/ Context-aware UI patterns are spreading
Spotify shipped a running mode which changes the core UI when users are running to make it easier to use hands-free and a User Notes feature which lets people caption individual tracks with personal context. Both features are examples of designing for specific micro-contexts as products learn to become more aware of a user’s current state.
Other companies are adopting similar patterns, too. Snap has introduced a Now Playing feature that lets users share what they’re currently listening to with friends.
4/ Design to code is continuing to collapse into a single workflow
The traditional handoff process designers took for granted is collapsing and this week saw some new releases continue that trend. Replit has launched Replit Design - a new creative suite that sits inside its core product. Replit Design lets non-designers create visual work directly inside Replit’s dev platform, using what the company calls “Ambient Intelligence” to guide design decisions, reducing export/rebuild steps between design and code.
Figma moved in the same direction, albeit from the other end of the workflow, as Figma Make gained a properties panel and in-context annotation prompts to Figma Make so designers can visually edit elements and steer code generation without leaving the canvas.
📈 Product data and insights
PostHog says its shipping velocity has tripled from 1441 pull requests in January to 4725 a month in June (a 3.3x increase in six months). They’re now tracking towards 10,000 pull requests a month - and code quality is not decreasing. They also revealed that agents now create 70% of code.
ChatGPT engagement is broadening, not just growing.
The return to the office trend is far from over. Patreon is the latest company to increase its return to office numbers. This week, it confirmed that it is increasing its RTO requirement from two to three days a week and is laying off 20% of its staff.
Google’s AI Overviews now make up 43% of all searches and Google’s Chrome says it is now shifting to a twice-weekly release cycle after AI agents discovered more security bugs in the last two releases than the previous 23 releases combined:
The surge in AI powered bug reporting has forced Apple to limit the number of bug reports researchers can submit to it.
Figma says almost half (47%) of product teams report experiencing some form of AI adoption mismatch where teams and leaders aren’t adopting AI at the same speed. 20% say individual contributors are pulling ahead without organizational support, while 27% say leadership is pushing AI adoption while teams struggle to keep up:
The Briefing is your curated report on what happened in tech and AI this week - hand crafted for product teams. Paid subscribers get the full DoP Substack including: The Knowledge Series for hands-on AI tutorials and DoP Deep dive reports for in-depth analysis to learn lessons from the world’s top tech companies.