tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agorobots.txt is a suggestionusenet.lolexternal-linkmessage-square25fedilinkarrow-up1625arrow-down13
arrow-up1622arrow-down1external-linkrobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agomessage-square25fedilink
minus-squareAlbum@lemmy.calinkfedilinkarrow-up1·5 months agoUser agent catching is rather effective. You can serve different responses based on UA. So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.
User agent catching is rather effective. You can serve different responses based on UA.
So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.