CrewAI RAG Deep Dive [Basic & Advanced Examples]

---

---

AI Summary

Summary: Building a YouTube Sponsorship Crew with RAG in Crew AI

Overview

• Objective: Automate the process of gathering information about YouTubers for potential sponsorships.
• End Result: A compiled profile of a YouTuber, including name, social links, and content topics.

Crew Components

• Scrape Agent: Fetches latest videos from a YouTube channel using fetchYouTubeURLs tool.
• Vector Database Agent: Adds YouTube videos to a vector store for retrieval using addVideoToVectorDatabase tool.
• Research Agent: Gathers initial information about the content creator using the rag tool.
• Follow-up Agent: Searches for any missing information not found in the initial search using the rag tool again.
• Fallback Agent: Uses firCrawl tool to scrape the web for any remaining missing information.

Custom Tools

• Fetch Latest Videos Tool: Retrieves the latest videos from a specified YouTube channel.
• Add Video to Vector Database Tool: Adds video transcripts to a vector store for RAG operations.

Process Flow

1. Input: YouTube channel handle.
2. Scrape YouTube: Fetch latest videos and their details.
3. Process Videos: Add video transcripts to the vector store.
4. Initial RAG Search: Retrieve initial content creator information from the vector store.
5. Follow-up RAG Search: Find missing information left out by the initial search.
6. Fallback Search: Scrape the web for any remaining missing information using firCrawl.
7. Output: Complete profile of the YouTuber.

Replay Feature

• Allows for re-running specific tasks within a crew to refine results or correct errors.
• Can be triggered manually in code or via CLI with YAML-based crew definitions.

Final Notes

• Source code available for free.
• Support and community available through School Community.
• Replay feature is part of the newer YAML-based crew definitions in Crew AI.