How to become “AI fluent”
🧠Tech CEOs are demanding AI fluency. Here’s some practical ways to demonstrate it. Knowledge Series #75
🔒The Knowledge Series is available for paid subscribers. Get full ongoing access to 70+ explainers and AI tutorials to grow your technical knowledge at work. New guides added every month.
Zapier’s CEO has joined the growing number of tech leaders demanding AI skills for new hires:
After this announcement, many folks rightly asked him to clarify what exactly he meant by AI fluency. And a few days later, he did just that by publishing an overview of how he intends to measure AI fluency at Zapier:
AI fluency levels are categorized into 4 core columns: unacceptable, capable, adoptive and transformative across roles including engineering, product, support, people / HR and marketing.
In this Knowledge Series, using this framework as a guide, we’re going to take a closer look at some of the practical ways product teams can demonstrate AI fluency in 2025.
These will be limited to product, engineering and customer support to make them as relevant as possible for product teams. We’ll dig deeper into some examples from each of the “capable”, “adoptive” and “transformative” columns, with real world examples from companies to bring them to life.
Coming up:
Setting the context: what is considered “unacceptable” in 2025?
5 practical ways to demonstrate AI fluency for product teams explored in more detail:
Shipping an AI-powered feature with a “human in the loop” check
Using ChatGPT / Copilot for simple coding
Choosing models based on accuracy, latency, throughput and context-window constraints
Rolling out an org-wide AI triage bot that cuts first-response times by 25%
Launching a proprietary fine-tuned LLM feature that opens up a new pricing tier
The core terminology required for AI fluency explained
What is considered “unacceptable” in 2025?
But before we dig into some of the ways you can demonstrate fluency, it’s worth taking a look at what’s now considered “unacceptable” in 2025 at least by Zapier’s CEO.
We’re fast approaching 3 years since the arrival of ChatGPT and despite that, not everyone remains convinced. The 3 core themes that emerge from the “unacceptable” column can be broadly categorised as: dismissing AI as hype, showing a lack of curiosity and remaining stubbornly dedicated to manual workflows over AI workflows.
Dismissing AI as hype was a lot easier in 2023 than in 2025. There was a time where some of us thought the AI hype cycle would implode just as the crypto did. But this never happened.
Despite this, there is still plenty of room for healthy scepticism. Some critics argue that LLMs are no more than glorified prediction machines that can never be taken seriously because of the risk that they’ll always hallucinate. And just a few days ago, Apple published a new research paper arguing that the capabilities of reasoning models are overhyped. This stance may or may not be influenced by Apple’s own struggles to play a leading role, of course.
But whatever the critics may rightly or wrongly think, given just how powerful categories like generative AI have become in just a few years, the consensus right now is that AI isn’t just a hype cycle - it is rapidly transforming industries whether we like it or not.
Likewise, stubbornly sticking to manual ways of work isn’t an option any more either. Google’s CEO recently said that AI code generation is boosting velocity by roughly 10% overall. If engineers and product teams aren’t willing to embrace AI then in a sense, they’re choosing to work at a slower pace, which won’t cut it in the post-pandemic world of mass layoffs.
5 examples of demonstrating AI fluency - a closer look at the essential skills for product teams
For this Knowledge Series, I’ve picked 5 examples of how product teams can demonstrate AI fluency, exploring these in a bit more detail with some examples and definitions to help.
Let’s start with the first example: shipping an AI-powered feature with a “human in the loop” check.
1. Shipping an AI-powered feature with a “human in the loop” check
Eventually, shipping an “AI-Powered” feature will probably sound as odd as shipping an “internet powered feature”. Some research even suggests that explicitly describing your new AI features as “AI-powered” might negatively impact how they’re perceived.
But for now, teams are doubling down on shipping AI-powered features. These features typically fall into categories including: UX enhancers, AI Assistants, productivity and time savers and new standalone products.
We regularly post updates on all of the new AI features shipped by product teams worth knowing about in the DoP Deep Dives and you can check out the latest editions of this from February and May this year here:
New AI features May 2025 - https://departmentofproduct.substack.com/p/deep-what-new-ai-features-are-product-a44
New AI features Feb 2025 - https://departmentofproduct.substack.com/p/deep-what-new-ai-features-are-product-686
What does “human in the loop” mean?
AI fluency doesn’t mean just shipping AI features, though. It also includes a requirement for features shipped specifically with “human in the loop” involvement.
Human in the loop (HITL) refers to the approach of building AI features where humans are actively involved during the development and training of the AI feature. In an AI feature development context, humans would do things like provide feedback, correct errors and ensure the AI doesn’t hallucinate.
For example, in the case where a product team is shipping a new conversational agent (let’s say a customer service bot), human evaluators would rank AI-generated responses to common queries like “How do I reset my password?” and then evaluate and correct the output accordingly.
This post from a Senior ML engineer at LinkedIn explains how companies like Airbnb use “human in the loop” alongside machine learning to make its travel categories more accurate and relevant to users.
Real world examples of using human in the loop techniques
It was recently discovered that Anthropic’s latest model blackmailed researchers when it was told it may be replaced.
In a carefully constructed fictional scenario, the model was given access to emails implying its imminent deactivation and separate information about an engineer's affair. When faced with the choice of being replaced or acting to preserve its existence, Claude Opus 4 often threatened to reveal an extra-marital affair if the replacement proceeded.
This was only found thanks to the efforts of a “red team” who are specifically tasked with identifying potential safety risks like this one. This is just one example of how human in the loop techniques are used to validate the effectiveness of AI features and models.
Here’s a snapshot of how human in the loop techniques might be used in new AI features released by other leading companies:
For product teams building AI features, demonstrating AI fluency means understanding the role that humans still actively play in the development of these features - and, increasingly, also in the features themselves. Agentic payments, for example, explicitly require humans in the loop for the feature to work.
2. Using ChatGPT / Copilot for simple coding
AI coding products are raising billions of dollars in venture capital right now and Meta predicts that within the next year, 50% of its code will be AI generated:
If you want to get up to speed with the AI coding space, here’s a deep dive and Knowledge Series on how AI coding products and agents work:
How non-engineers can use ChatGPT and Copilot to gain an understanding of web technologies
Although this AI fluency skill is primarily aimed at engineers, it’s still useful for non-engineers to understand how ChatGPT and other AI tools like Copilot work for coding.
“Vibe coding” is a lot easier when you’ve got at least a basic understanding of what exactly is being vibe coded.
Getting some hands-on experience of using tools like ChatGPT and Copilot can help. This Knowledge Series on front end technologies should help to the groundwork of understanding HTML, CSS and JS but once you’ve broadly understood the core technologies involved, you can then use ChatGPT to ask it to build you some basic HTML / CSS prototypes to see it in action: