• JohnEdwa@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      12
      ·
      1 year ago

      Robots.txt has been always ignored by some bots, it’s just a guideline originally meant to prevent excessive bandwidth usage by search indexing bots and is entirely voluntary.

      Archive.org bot for example has completely ignored it since 2017.