The free internet encyclopedia is the seventh-most visited website in the world, and it wants to stay that way.
    • F/15/Cali@threads.net@sh.itjust.works
      20·
      7 days ago

      I mean, yeah. AI companies are nearly universally, and objectively, piloted by fucksticks. Lemmy instances are also constantly scraped rather than just spinning up an instance and pulling the entire threadiverse.

      • baltakatei@sopuli.xyz
        11·
        7 days ago

        I have a public gitweb repository. I am constantly being hit by dumb crawlers that, left to their own devices, request every single diff of every single commit simply because links requesting such operations are presented. All of which are unnecessary if they would only do a simple git pull, then my server would happily provide the 50 MB of the entire git repo history. Instead, they download gigabytes of HTML boilerplate, probably never actually get a full commit history, and probably can’t even use what data they do scrape since they’re just randomly pulling commits in between blocks and bans.

        All of this only became an issue around a year ago. Since then, I just accept my public facing static pages are all that’s reliable anymore.

    • bamboo@lemmy.blahaj.zoneEnglish
      151·
      7 days ago

      I would imagine the torrents are a snapshot in time. I don’t think they can be updated after being created? Also, picture the average dev. Half of them are lazier and wouldn’t deal with rss or torrents, when you can just make get thousands of redundant GET get requests to use for training data.

        • F/15/Cali@threads.net@sh.itjust.works
          222·
          7 days ago

          The answer is simpler than you could ever conceive. Companies piloted by incompetent, selfish pricks are just scraping the entire internet in order to grab every niblet of data they can. Writing code to do what they’re doing in a less destructive fashion would require effort that they are entirely unwilling to put in. If that weren’t the case, the overwhelming majority of scrapers wouldn’t ignore robot.txt files. I hate AI companies so fucking much.

          • pivot_root@lemmy.world
            8·
            7 days ago

            “robots.txt files? You mean those things we use as part of the site index when scraping it?”

            — AI companies, probably

      • taiyang@lemmy.world
        53·
        7 days ago

        Close, I think the average dev can’t even imagine a product that isn’t for profit and is available to everyone as a public service. Scraping uses a lot more resources than just regularly downloading the snapshot, so it genuinely makes no sense. It’s like shoplifting from a soup kitchen.

        But… I know tech people personally. They’re really that dumb.