Today it’s estimated there are over 1 billion websites on the internet. Much of this content is optimized to be viewed by human eyes, not consumed by machines. However, creating systems to automatically parse and structure the web greatly extends its utility, and paves the way for innovative solutions and applications. The industry of web scraping has emerged to do just that. However, many websites erect obstacles to hinder web scraping. This has created a new kind of arms race between developers and anti-scraping software.

Bright Data has developed some of the most sophisticated consumer tools available to scrape public web data. Erez Naveh is an entrepreneur and former engineer at Meta. He is currently the VP of Product at Bright Data. Erez joins us in this episode to talk about Bright Data’s mission to structure the open web, and the toolkit they’ve developed to make this possible.

Full Disclosure: Bright Data is a sponsor of Software Engineering Daily

Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space, an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.

Please click here to view this show’s transcript.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Web Scraping with Erez Naveh appeared first on Software Engineering Daily.

Modern Web Scraping with Erez Naveh 57:29 Share

Data Archives - Software Engineering Daily

Shownotes Transcript

Modern Web Scraping with Erez Naveh