There are plenty of options to implement a fully-fledged observability solution for Kubernetes - self-managed or fully-managed cloud solutions. Here in this blog post we will setup a open-source self-managed observability stack for Kubernetes.

We will deploy and use the following tools:

Quickstart

If you don't have a setup to try it out, you can use the following quickstart:

https://github.com/ventx/tf-k8s-eks-grafana-observability

Deployment

Edit bootstrap.hcl according to your setup, configure your Github token and OIDC client secret of your Slack app.

git clone https://github.com/ventx/tf-k8s-eks-grafana-observability.git
 export TF_VAR_github_token=123
 export TF_VAR_oidc_client_secret=123
export AWS_PROFILE=
make all

Deployment takes around ~20 minutes.

Cleanup

If you are done experimenting, simply destroy all resources with:

make destroy

Prometheus Operator

Mimir

Tempo

Loki (and Promtail)

Grafana

Links