everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • Bahnd Rollard@lemmy.world
    link
    fedilink
    English
    arrow-up
    14
    ·
    2 days ago

    Wern’t there a few AI maze projects in the works? I wonder if running one of those for a bit will cause you to be added to an ignore list, clearly they dont respect your robots file.