Question 1

What are the key takeaways from Hamel Husain & Shreya Shankar's episode on Lenny's Podcast?

Accepted Answer

1. Start with error analysis by manually reviewing traces before writing any automated evals -- looking at your data is the highest-ROI activity for improving AI products. 2. Use a 'benevolent dictator' approach: appoint one trusted domain expert to lead the eval process rather than bogging down with committees, keeping the process tractable and fast. 3. Build LLM judges with binary pass/fail outputs for specific failure modes rather than Likert scales -- nobody knows what 3.2 versus 3.7 means, and simplicity drives better decisions. 4. Ground your evals in actual production data rather than hypothetical requirements -- you will always uncover failure modes and product expectations you could never have dreamed up in advance. 5. Error analysis techniques like open coding and axial coding are borrowed from social science and data science, not invented for AI -- be wary of entirely novel methodologies not grounded in established practice.

Question 2

What does Hamel Husain & Shreya Shankar say about AI evals and error analysis?

Accepted Answer

"It's really important that we don't think of evals as just tests. There's a common trap that a lot of people fall into because they jump straight to the test like, 'Let me write some tests,' and usually that's not what you want to do. You should start with some kind of data analysis to ground what you should even test, and that's a little bit different than software engineering where you have a lot more expectations of how the system is going to work." - Hamel Husain "You sample your data and just take a look, and it's surprising how much you learn when you do this. Everyone that does this immediately gets addicted to it and they say, 'This is the greatest thing that you can do when you're building an AI application.' You just learn a lot and you're like, 'Hmm, this is not how I want it to work.'" - Hamel Husain "What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell or not. For example, the hallucination about scheduling the tour, right? I can guarantee you, I would bet money on this, if I put that into ChatGPT and asked, 'Is there an error?' it would say, 'No, did a great job.'" - Shreya Shankar

Question 3

What are Hamel Husain & Shreya Shankar's views on AI evals, error analysis, LLM judges?

Accepted Answer

Shreya Shankar: "What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell or not. For example, the hallucination about scheduling the tour, right? I can guarantee you, I would bet money on this, if I put that into ChatGPT and asked, 'Is there an error?' it would say, 'No, did a great job.'" Shreya Shankar: "People always think, 'Oh, this is at least as hard as my problem of creating the original agent.' And it's not, because you're asking the judge to do one thing, evaluate one failure mode, so the scope of the problem is very small and the output of this LLM judge is pass or fail. So it is a very, very tightly scoped thing that LLM judges are very capable of doing very reliably."

Hamel Husain & Shreya Shankar

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

TRANSCRIPT

Hamel Husain & Shreya Shankar

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

Hamel Husain & Shreya Shankar

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

TRANSCRIPT

Hamel Husain & Shreya Shankar

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar