May 27th, 2020
I Could Have Sold TeraCrawler.io To Larry Page

I just came across a link to this Google Groups message by none other than Larry Page asking for tips on setting User-Agent string in his web crawler. (User-Agent strings are used to identify and these days spoof the identity of the crawler or browser)


It's humbling to see him ask some of the most fundamental questions in web crawling.fundamental and innocent with no sign of the potent tech giant that was to be born from this. Imagine being Joseph Millar, the person who answered this simple question.

That's why this got me daydreaming. I could have easily pointed Larry to a bunch of resources we have written at and maybe got him as a customer for TeraCrawler.io our cloud web crawling service.

Share this article:

Get our articles in your inbox

Dont miss our best tips/tricks/tutorials about Web Scraping
Only great content, we don’t share your email with third parties.
Icon