So... My AI App Has Been Lying to Users (And How I Fixed It)

Chris Raroque faces issues with accuracy in his AI-based calorie tracking app, Amy, leading to subscriber cancellations. The AI's accuracy is crucial since it pulls data from nutrition databases but often errs, especially with international products. Raroque demonstrates his approach to enhancing AI accuracy using real production data and iterative testing, known as "evals." He describes how he used Brain Trust's tools to run tests and get objective accuracy scores. Through multiple attempts to improve the AI, such as separating search and reasoning tasks, Raroque performs comparisons between different models and search providers. His trials show that although some innovations, like using Gemini 3 for reasoning, initially worsen the app's performance, others like switching search providers from Perplexity to Exa can boost the app's accuracy. Despite some failures, Exa emerges as a more effective search partner. Raroque reflects on the challenges and successes in optimizing his AI, emphasizing the need for continuous testing and adaptation to changes in data or performance over time. This ongoing process has increased the app's accuracy while maintaining cost-efficiency and speed. He further underscores the importance of setting up a robust eval system, involving judiciously choosing test cases and ensuring judgements are reliable. Finally, he encourages viewers to implement their own eval systems, sharing his tools and insights for better AI performance management.