Pro@programming.dev to Technology@lemmy.worldEnglish · 4 days agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square234linkfedilinkarrow-up11.15Karrow-down111file-textcross-posted to: Technology@programming.dev
arrow-up11.14Karrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@lemmy.worldEnglish · 4 days agomessage-square234linkfedilinkfile-textcross-posted to: Technology@programming.dev
minus-squareT156@lemmy.worldlinkfedilinkEnglisharrow-up8·4 days agoThe website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
minus-squareacockworkorange@mander.xyzlinkfedilinkEnglisharrow-up1·30 minutes agoYou could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.
The website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
You could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.