Finetuning 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt
AI Summary
Video Summary: Scaling Production to 500 Million Agents with Method
Introduction
- Speakers: Kyle Corbett (Open Pipe) and Mustafa Ali (Method)
- Discussion on Method’s journey to scale to over 500 million agents.
Data Aggregation Challenges
- Method collects liability data from various sources, including credit bureaus, card networks (Visa, MasterCard), and banks.
- Enhances data for fintechs, banks, and lenders for debt management, loan consolidation, and personal finance.
- Early challenges included the need for specific liability data points (e.g., payoff amounts).
Inefficient Solutions
- Many companies hire offshore contractors to authenticate and gather information from banks, leading to:
- Inefficiencies
- High costs
- Errors due to manual data entry.
- The aim was to build a fast, scalable solution.
Adoption of GPT-4
- Leveraged advanced AI (GPT-4) to handle unstructured data challenges.
- Early success, but significant costs ($70,000/month) and issues with scalability arose.
- Encountered difficulties with prompt engineering and AI errors (hallucinations).
- Shifted focus to creating a robust agentic workflow capable of high-volume demands (~16 million requests/day).
Solutions and Custom Models
- Collaborated with Open Pipe to address quality, cost, and latency challenges.
- Implemented a benchmarking strategy to measure error rates, latency, and costs across models.
- Fine-tuned models to improve performance:
- Reduced error rates from 11% (GPT-4) to below 9%.
- Improved latency and decreased costs significantly.
Conclusion
- Fine-tuning empowers companies to scale efficiently in production, addressing industry-specific needs.
- Importance of patience and openness in engineering when dealing with AI agents.
- Summary emphasizes that effective AI implementation is feasible without needing extensive infrastructure (like private GPUs).