I put up a vps with nginx and the logs show dodgy requests within minutes, how do you guys deal with these?

Edit: Thanks for the tips everyone!

  • Dr_Toofing@programming.devEnglish
    4·
    1 year ago

    These requests are probably made by search/indexing bots. My personal server gets a quite a lot of these, but they rarely use any bandwidth.
    The easiest choice (probably disliked by more savvy users) is to just enable cloudflare on your server. It won’t block the requests, but will stop anything malicious.
    With how advanced modern scraping techniques are there is so much you can do. I am not an expert, so take what I say with a grain of salt.

    • WasPentalive@lemmy.oneEnglish
      21·
      1 year ago

      The ligitimate web spiders (for example the crawler used by Google to map the web for search) should pay attention to robots.txt. I think though that that is only valid for web-based services.

    • Rusty@lemmy.worldEnglish
      21·
      1 year ago

      Fail2Ban is great and all, but Cloudflare provides such an amazing layer of protection with so little effort that it’s probably the best choice for most people.

      You press a few buttons and have a CDN, bot attack protection, DDOS protection, captcha for weird connections, email forwarding, static website hosting… It’s suspicious just how much stuff you get for free tbh.