The Truth About Devin AI and Website Scraping

---

---

AI Summary

- Introduction  
  - Excitement about testing Devon's web scraping capabilities  
  - Task: Scrape m.v website, convert pages to Markdown, zip, and download  
  
- Video Content  
  - Request to Devon: Scrape m.v, organize Markdown files, zip for download  
  - Steps Devon will follow:  
    - Navigate and scrape main page  
    - Identify and scrape linked pages  
    - Convert pages to Markdown  
    - Organize files in a folder  
    - Zip and provide the file  
  - Execution:  
    - Devon browses category pages, converts to Markdown  
    - Folder structure organized by category  
    - Content includes author name, post date, and main content  
    - After initial pages, Devon zips the folder for download  
    - First attempt results in six pages  
    - After a deep crawl prompt, more pages are scraped  
    - Second attempt yields 183 pages, less than the total on the site  
    - A new session with explicit instructions to crawl all pages  
    - Final result: 746 pages scraped  
  
- Observations  
  - Need for specific instructions on content extraction  
  - Different instances show varied extraction results  
  - Importance of clear prompts for desired structure and content  
  - Potential for SEO optimization by extracting metadata  
  
- Conclusion  
  - Encouragement to like, share, and subscribe for future updates on SEO optimization with Devon