Web crawling—the process of indexing information online—has been around for decades. It has been mainly used by search engines such as Google and non-profit organizations such as Internet Archive And Joint crawling To index the contents of the open Internet and make them searchable. Until recently, the practice of web crawling was rarely seen as controversial, as websites relied on the process as a means through which people could find their content. But now crawling technology is embedded in the wonderful AI of everything, and is being used by companies like Google and Perplexity AI to ingest full articles that are fed into their summarization machines.

This week on Gadget Lab, WIRED senior writer Kate Knibbs joins the show to talk about web crawling and the controversy surrounding syndicated crawling. Then we spoke with Forbes chief content officer and editor Randall Lin about how Perplexity.AI repurposed a Forbes article and presented it as its own story, without first asking for permission or properly citing the source.

Show notes

Read Kate’s story on how to do it Publishers are chasing syndication creep Via AI training data. Read Randall’s story on how to use Preplexity.AI Copy the work To two Forbes reporters.


Randall recommends the new horse racing league National Thoroughbred Horse League. Kate recommends the book victim By Andrew Borega. Lauren recommends the show Hacks On Max.

Randall Lane can be found on social media @Randall Lane. @kateknibbsNibs. @laurengoodLauren Good. michael calore @com. snakefight. Bling Main Hotline @GadgetLab. The show is produced by Bon Ashworth (@com.booneashworth). Our theme music is by Solar switches.

How to listen

You can always listen to this week’s podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here’s how:

If you’re using an iPhone or iPad, open the app called Podcasts, or just Podcasts Click on this link. You can also download an app like Overcast or Pocket Casts, and search for Gadget Lab. If you’re on Android, you can find us right in the Google Podcasts app Tapping here. We are on Spotify also. And in case you really need it, Here’s the RSS feed.

Leave a Reply

Your email address will not be published. Required fields are marked *