Real Python
  • Start Here
  • Learn Python
    Python Tutorials →
    In-depth articles and video courses     Learning Paths →
    Guided study plans for accelerated learning     Quizzes & Exercises →
    Check your learning progress     Browse Topics →
    Focus on a specific area or skill level     Community Chat →
    Learn with other Pythonistas     Office Hours →
    Live Q&A calls with Python experts     Podcast →
    Hear what’s new in the world of Python     Books →
    Round out your knowledge and learn offline     Reference →
    Concise definitions for common Python terms     Code Mentor →Beta
    Personalized code assistance & learning tools     Unlock All Content →
  • More
    Learner Stories Python Newsletter Python Job Board Meet the Team Become a Tutorial Writer Become a Video Instructor
  • Search
/
  • Join
  • Sign‑In

— FREE Email Series —

🐍 Python Tricks 💌

Python Tricks Dictionary Merge

🔒 No spam. Unsubscribe any time.

Browse Topics Guided Learning Paths
Basics Intermediate Advanced
ai algorithms api best-practices career community databases data-science data-structures data-viz devops django docker editors flask front-end gamedev gui machine-learning news numpy projects python stdlib testing tools web-dev web-scraping

Table of Contents

  • What Is Web Scraping?
    • Reasons for Automated Web Scraping
    • Challenges of Web Scraping
    • An Alternative to Web Scraping: APIs
  • Scrape the Fake Python Job Site
  • Step 1: Inspect Your Data Source
    • Explore the Website
    • Decipher the Information in URLs
    • Inspect the Site Using Developer Tools
  • Step 2: Scrape HTML Content From a Page
    • Static Websites
    • Login-Protected Websites
    • Dynamic Websites
  • Step 3: Parse HTML Code With Beautiful Soup
    • Find Elements by ID
    • Find Elements by HTML Class Name
    • Extract Text From HTML Elements
    • Find Elements by Class Name and Text Content
    • Pass a Function to a Beautiful Soup Method
    • Identify Error Conditions
    • Access Parent Elements
    • Extract Attributes From HTML Elements
  • Assemble Your Code in a Script
  • Keep Practicing
  • Conclusion
  • Frequently Asked Questions

Recommended Video Course
Web Scraping With Beautiful Soup and Python