Using AI tools to supercharge product development workflows. How PMs and builders leverage AI for productivity, coding, and decision-making from Lenny's Podcast.
“The difference between people who use Claude Code very effectively and people who use it not so effectively is are they asking for the ambitious change? And if it doesn't work the first time, asking three more times because our success rate when you just completely start over and try again is much, much higher than if you just try once.”
“It's really hard to measure productivity. So I do ask people to ask their managers, 'Would you rather give everyone on the team very expensive coding agent subscriptions or you get an extra head count?' Almost everyone, the managers will say head count. But if you ask VP level or someone who manage a lot of teams, they would say, 'Want AI assistant.' Because as managers, you are still growing, so for you having one HR head count is big. Whereas for executives, maybe you have more business metrics that you care about.”
“It writes a little to-do list for itself. It can have a little notebook, it can go and read each little thing and then write into its notebook, go down its to-do list and give you a summarized answer over multiple turns. So it's not just stuffing everything into context, which is what you'd be doing with ChatGPT chat. It's actually processing every single file that you give it.”
“One area in which we find that it's really good is for non-technical teams to be able to build little software tools for themselves. Our enterprise risk management team build a whole system for self-servicing enterprise risk, and this is compressing weeks of work into hours.”
“People spend far too much time looking for what they're hoping to see, not for what they're not looking to see. You can literally ask ChatGPT to help you find where the customer is probing at the edges of what you're trying to do, where it's wrong, where what you're saying is not what they believe.”
“If you're at a company at a scale of ours and many others in the market, you're like, this is almost like a new production function and mindset that you have to do. And there's really three components that we're working on. One is platform. The second one is the tools and the agents. And lastly is the culture.”
“When I think about agents, I think about three things. One is an increasing level of autonomy and kind of independence that you can delegate higher and higher order tasks. Second thing I think of it is complexity. It's not a one-shot, it's build me this prototype. And then the third thing I would say is asynchronous. It works when you are not working.”
“All of the conditions, all of the ingredients for a new distribution platform to emerge are essentially happening. So I think we're at an inflection point where we're going to see this emerge really fast.”
AI progress is accelerating, not plateauing -- model releases have gone from once a year to every few months, and scaling laws continue to hold across 15+ orders of magnitude.
Benjamin MannPost-training is where AI labs now differentiate - pre-training data has largely maxed out, so the competitive advantage comes from how models are fine-tuned and aligned after initial training.
Chip HuyenThe Economic Turing Test provides a concrete way to measure AGI: can an AI agent pass as human when contracted for a job for one to three months across a market basket of roles?
Benjamin MannData preparation matters more than infrastructure choices for RAG - companies see the biggest quality gains from how they chunk, annotate, and structure their data, not from which vector database they use.
Chip HuyenAI safety and product quality are convex, not at odds -- Anthropic's alignment research directly produced Claude's distinctive personality and character that users love.
Benjamin MannMeasuring AI productivity is genuinely hard - managers prefer headcount while executives prefer AI tools, revealing that incentives and metrics differ by organizational level.
Chip HuyenBuild products for where AI capabilities will be in 6-12 months, not where they are today -- features working 20% of the time now will work 100% of the time soon.
Benjamin MannHigh-performing engineers benefit most from AI coding tools - a randomized trial showed top performers gained the biggest productivity boost, likely because they already have strong problem-solving foundations.
Chip Huyen