I FINALLY figured out how to drive the Grafana bus. I know, I know…pretty basic stuff. In the past I always drooled over dialed out dashboards I’ve seen. Charts, graphs, readouts, dials…but I never could quite wrap my head around it all.
Well, now I have charts, graphs, readouts, logs, dials, and it feels pretty good to have finally learned something new that I’ve been whacking away at for a while. The plus side is that the whole Graphana+Promtail+Loki stack sips lightly on my resources, whereas in the past, things like the ELK stack, Opensearch, Graylog, etc, really devoured a lot of resources. I really tried with those but just didn’t like how ravenous of a RAM appetite they had.
I’m still using lnav. It’s quick and dirty and gets the job done for logs. But if I want to look at data, with nice a prettyfied interface, I go look at Grafana. So now the process will be to, dial out and graph every last little thing until I suffer from info overload, and then ease it back to just what is necessary. It’s a fucked up process, but it’s how I do.
As my lady friend is apt to say: It’s the little things…
You must log in or register to comment.
I’ve been interested in setting up a monitoring setup like this, mostly out of curiosity about what’s going on when I’m not looking. But I know what the answer is and it’s not as exciting as I’d like it to be.
But I know what the answer is and it’s not as exciting as I’d like it to be.
At least you would know, and you would learn something new as I have done. I have set up to monitor syslogs, ufw logs, assorted metrics so far. I’m going to tackle using cAdvisor and ingest the data into Prometheus and display such data on my Grafana dash.
It’s fun and educational. Try it. At the very least when you’ve worn all the new off, you can just delete the docker containers.
I just did the same thing. Grafana with Prometheus, cAdvisor, Loki, alloy. It has really stepped up my overall systems monitoring.
I have a follow up question about cadvisor, if you have a minute. Cadvisor employ Redis and Prometheus. If I understand correctly, you configure Prometheus to scrape Docker logs. All fine and good. However, my Grafana stack already employs Prometheus to do the same, just for syslogs and such, but not Docker. Can I not place the contents of the Cadvisor/Prometheus config in the existing Prometheus implementation used with Grafana, or does Cadvisor need it’s own Prometheus instance.
You should be able to just add a scrape config to your existing deployment of prometheus
Thank you for the reply. I guess I am thinking, why have multiple Prometheus instances running, eating up resources, when I could do it with just one.
Sorry I didn’t get back to you right away. But this is correct. I just have Prometheus scrape cAdvisor.
You’re fine. Thanks for the follow up.
For sure. Simplifies querying the data too, since you won’t need to deal with multiple data sources in grafana.
Awesome!
alloy
I was just commenting that I’m going to have to figure out how to migrate to Alloy now as Prom is nearing it’s eol.
TIL Promtail is going EOL. I’ve been using it forever. Guess I’m gonna have to figure out how to migrate to alloy too if that’s meant to be its successor. At home and at work lol.
I’m sorry to be the bearer of bad news. I read up on it about an hour ago. Of course, just as I finally get things running, they switch horses. le sigh…c’est la vie
Yep. Keeping up to date is a never ending battle. I try to do it often so they don’t pile up and break a ton of shit all at once. If you’re into gitops, I’m a big fan of using renovate to help automate things.
ELK stack
Lolol yes, elastic was a pig for me too
It was like running windows. The more I fed it, the more it belched and devoured more. LOL Now I see that I am going to have to figure out Grafana Alloy, as Prom is nearing it’s eol in '26. Le sigh… that’s the way it goes.
I’d love to use it but virus total always flags it as a virus and I get too scared.
Huh. I don’t know anything about that. I wonder if it’s because Grafana hooks into more sensitive points on your server and that triggers VT.
