How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it's different now.

[–] gressen@lemmy.world 14 points 1 year ago (6 children)

Easy - detect if you're getting accessed by a search crawler or a human. Serve a full page or just a login request.

[–] RGB3x3@lemmy.world 11 points 1 year ago (5 children)

So how can a user pretend to be a web crawler?

[–] theMightyMoonWorm@lemmy.ml 22 points 1 year ago* (last edited 1 year ago)

This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html

load more comments (4 replies)

load more comments (8 replies)