I’m pretty new to selfhosting, but one thing that I know to take seriously is log collection. Since there are a lot of different type of logs (kernel log, application logs, etc) and logs come in many different formats (binary, json, strings) - it’s no easy task to collect them centrally and look through them whenever neccessarly.
I’ve looked at grafana and tried the agent briefly, but it wasn’t as easy as I thought (and it might be a too big tool for my needs). So I thought to ask the linuxlemmy community to get some inspiration.
You must log in or register to comment.
Nothing. I always use the standard configuration from the services I’ve installed. If something doesn’t work I go into those logs and look what’s wrong.
Syslog (rsyslod) is usually the standard answer for the average sysadmin, but it depends a lot on your needs. A lot of newer loggers output as pure JSON, which offer benefits to readability and more approachable search logic/filters/queries (I’m so tired of regex).
When you start venturing down the road of finding the right way to store and forward the output of logging drivers from Docker containers, as one example, rsyslod starts to feel dated.
The easy answers if you want to throw money at the problem are solutions like Splunk, Datadog, or New Relic. If you don’t want to (and most people wouldn’t), then alternatives certainly exist, but some of them are just as heavy on system resources. Greylog has relative feature parity with Splunk Enterprise, but consumes just as much compute and storage if not more, and I found it to be a much larger pain in the butt to administer and keep running.
The likeliest answer to this problem is Grafana Loki, just based on what I’ve read of its capabilities, but I haven’t had a chance to circle back and test it out. Someone here who has might be able to weigh in and speak to its strengths/weaknesses.
In terms if logging: sys journal does the job for me …
I do run prometheus + grafana for some services but that is mostly for some fancy looking graphics nothing really usefull.
I would recomend you to monitor updates of you apps so you are well informed when und what to update (i just have subsribed to all the diffrent release git rss feeds)
My logs were actually filling up my disk! I edited journald conf and limited them to 500mb . disk is only 10gb for reference
What services were you running and for how long? I ask because I rent a very cheap VPS with 10G disk and I want to prepare for this
Make sure that you have logrotate installed and working, to prevent them from piling up.
I’ve been using Loki and promtail for containers I actually care about aggregating logs for. It’s something relatively new for me but so far the experience has been positive.
Any specific guide you followed?
Systemd does all that for you. Just set a limit in journal.conf
So does syslog; and really well. Logging is just another thing Lennart tried to take over from a working system, that he and Kay half-delivered, and wandered off. See also: nfsroot, fast boots, easy init config, nfs mounts in general, and cron.
OK boomer
I’ve done everything from local syslog to various open source and proprietary log pipelines and also worked on a team dedicated to building a custom log-processing pipeline from scratch in a large environment.
One lesson I’ve learned is that no matter how nice that web UI is I still want some kind of CLI access and the equivalent of being able to cat, tail and grep logs. In large environments it won’t be actually tail-ing a logfile but I want something that can produce a live logstream for piping into Unix text (and JSON) processing tools.
Honestly? I just ignore them. Something seems to be happening to them, as they’re not growing infinitely, but no idea why.
It was a bit complicated to set this up, but it took me almost no time at all.
