Chat

7499 readers

4 users here now

Relaxed section for discussion and debate that doesn't fit anywhere else. Whether it's advice, how your week is going, a link that's at the back of your mind, or something like that, it can likely go here.

Subcommunities on Beehaw:

AskBeehaw

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago

MODERATORS

alyaza@beehaw.org

remington@beehaw.org

I search a lot less than I thought (lemmy.zip)

submitted 3 months ago by shortwavesurfer@lemmy.zip to c/chat@beehaw.org

15 comments fedilink hide all child comments

I am a Kagi user and have been for 7 months now. I signed up for the 300 searches per month plan because I felt like that would fit me well enough and it turns out I average 87 searches per month. That's a lot lower than I thought it would be by quite a bit.

you are viewing a single comment's thread
view the rest of the comments

[–] l_b_i@yiffit.net 5 points 3 months ago (7 children)

That reddit filter will have less than an effect than it used to. Reddit blocked all crawlers except google.

[–] ardi60@reddthat.com 2 points 3 months ago (5 children)

Brave search works for Reddit

[–] l_b_i@yiffit.net 4 points 3 months ago (4 children)

reddit recently updated their robots.txt to disallow all crawlers Google paid a bunch of money to have access to crawl reddit

You'll still see old stuff, but crawlers that care about robots.txt will get no new information.

[–] ardi60@reddthat.com 1 points 3 months ago (1 children)

https://search.brave.com/search?q=site%3Areddit.com+trump&source=web&tf=pd Past day is not that old

[–] l_b_i@yiffit.net 3 points 3 months ago

A few possibilities,

brave has state in the past they use the googlebot user agent, if all reddit does is check the useragnet, it won't block brave. This does however mean brave is violating the robots.txt file.
I saw some mentions of google fall back, I don't know if they still do it, but that could be another possibility.
Brave ignores robots.txt files
Brave paid for access

No matter the reason, well behaving crawlers will no longer crawl reddit, Everything is disallowed in the robots.txt

User-agent: *
Disallow: /

load more comments (2 replies)

load more comments (3 replies)