tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 4 days agorobots.txt is a suggestionusenet.lolimagemessage-square25fedilinkarrow-up1617arrow-down13
arrow-up1614arrow-down1imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 4 days agomessage-square25fedilink
minus-squareBlackPenguins@lemmy.worldlinkfedilinkarrow-up24arrow-down1·edit-24 days agoWhat else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
minus-squarelud@lemm.eelinkfedilinkarrow-up7arrow-down2·4 days agoIndexing is what “scrapers” mostly do. That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.
minus-squareAlbum@lemmy.calinkfedilinkarrow-up13·4 days agoSearch engine scrapers index. But that’s a subset of scrapers. There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data. This post is specifically about AI scrapers.
minus-squareShiggles@sh.itjust.workslinkfedilinkarrow-up5arrow-down1·4 days agoI think by shitty they mean ineffective, not poor manners.
What else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
Indexing is what “scrapers” mostly do.
That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.
Search engine scrapers index. But that’s a subset of scrapers.
There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data.
This post is specifically about AI scrapers.
I think by shitty they mean ineffective, not poor manners.