pydumper

pydumper is a set of educational web-scraping scripts, in Go and Python, that collect pages for a search query and store what they find.

Parts

• A Go crawler that queries the Google Custom Search API and fetches the result pages concurrently with a configurable number of threads. It parses them and saves the results to an SQL database. Its settings (API key, search engine, thread count, result limits, database connection) live in a config.ini.
• A Python scraper for the same job from the command line, with a shell script that runs several instances at once.
• Benchmarks and a dictionary of search terms.

The README states that the scripts are for educational purposes only.

See also

• Systems from scratch