Istio provides a lot of telemetry data for you to collect in your Kubernetes cluster mesh. Especially the metrics for the the Envoy sidecars of your workloads on the stats endpoint are often business critical.
You can optimize the bandwith usage to collect all those metrics in various ways:
- Enable HTTP gzip compression for the Envoy metric endpoints
- Drop labels / metrics in Istio completely (Istio Telemetry API MetricsOverrides)
- Reduce the Prometheus scrape interval for Istio metrics
In this post we will look at the first option as it is relatively easy to implement and does not reduce our observability capacity at all. The savings on bandwith with enabled gzip compression for these metrics is huge: >90%! And it just requires to create a new ConfigMap resource, add an annotation to your workload with which references the ConfigMap and add an Environment Variable to Istio to enable the custom bootstrap config feature - thats it :)
Reducing the scrape interval is a quick-fix solution, too - but it reduces your metrics resolution. And dropping labels / metrics which you do not need is always a good thing to do - but might involve more work and investigation to do so.
Here you see an example of the savings in bandwith:
# Uncompressed:
kubectl exec test-pod -c istio-proxy -- curl -Ss -i --compressed -w '%{size_download}' http://localhost:15090/stats/prometheus | tail -n 1
6511
# Compressed:
kubectl exec test-pod -c istio-proxy -- curl -Ss -i -w '%{size_download}' http://localhost:15090/stats/prometheus | tail -n 1
332317
## Savings in bandwith: nearly 98%
For HTTP gzip compression of the metrics endpoints to work, it needs to be supported by both ends: the client (Prometheus, which scraped the endpoint) and the server (Envoy sidecar, which provides the metrics at a specific endpoints /stats/prometheus). Prometheus supports it in their text-based metrics format: https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format. But for the Envoy sidecars we have to tell them to enable gzip compression.
As we want to enable gzip compression for ALL our envoy sidecars, we need to take a look into Envoy's bootstrap configuration. Istio needs to know how the bootstrap config looks like and where to find it. Additionally, the feature for a custom envoy bootstrap config is currently guarded with an environment variable: BOOTSTRAP_XDS_AGENT. This needs to be set to true to enable the feature.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"EnvoyFilter","metadata":{"annotations":{},"name":"stats-compression","namespace":"istio-system"},"spec":{"configPatches":[{"applyTo":"BOOTSTRAP","match":{"context":"SIDECAR_INBOUND"},"patch":{"operation":"MERGE","value":{"node":{"metadata":{"ENVOY_PROMETHEUS_PORT":16090}}}}},{"applyTo":"BOOTSTRAP","match":{"context":"SIDECAR_INBOUND"},"patch":{"operation":"MERGE","value":{"staticResources":{"listeners":[{"address":{"socketAddress":{"address":"0.0.0.0","portValue":16090}},"filterChains":[{"filters":[{"name":"envoy.filters.network.http_connection_manager","typedConfig":{"@type":"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager","httpFilters":[{"name":"envoy.filters.http.compressor","typedConfig":{"@type":"type.googleapis.com/udpa.type.v1.TypedStruct","typeUrl":"type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor","value":{"compressorLibrary":{"name":"prometheus","typedConfig":{"@type":"type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip","compressionLevel":"BEST_COMPRESSION","compressionStrategy":"DEFAULT_STRATEGY","memoryLevel":3,"windowBits":10}},"responseDirectionConfig":{"commonConfig":{"contentType":["*/*"],"minContentLength":100}}}}},{"name":"envoy.filters.http.routerx","typedConfig":{"@type":"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"}}],"routeConfig":{"virtualHosts":[{"domains":["*"],"name":"backend","routes":[{"match":{"prefix":"/stats/prometheus"},"route":{"cluster":"prometheus_stats"}}]}]},"statPrefix":"stats"}}]}]}]}}}}]}}
creationTimestamp: "2022-07-26T14:28:55Z"
generation: 2
name: stats-compression
namespace: istio-system
resourceVersion: "1224872339"
uid: bef682e0-5a09-4e8a-9f7d-05673e42d3d5
spec:
configPatches:
- applyTo: BOOTSTRAP
match:
context: SIDECAR_INBOUND
patch:
operation: MERGE
value:
node:
metadata:
ENVOY_PROMETHEUS_PORT: 16090
- applyTo: BOOTSTRAP
match:
context: SIDECAR_INBOUND
patch:
operation: MERGE
value:
staticResources:
listeners:
- address:
socketAddress:
address: 0.0.0.0
portValue: 16090
filterChains:
- filters:
- name: envoy.filters.network.http_connection_manager
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
httpFilters:
- name: envoy.filters.http.compressor
typedConfig:
'@type': type.googleapis.com/udpa.type.v1.TypedStruct
typeUrl: type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
value:
compressorLibrary:
name: prometheus
typedConfig:
'@type': type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
compressionLevel: BEST_COMPRESSION
compressionStrategy: DEFAULT_STRATEGY
memoryLevel: 3
windowBits: 10
responseDirectionConfig:
commonConfig:
contentType:
- '*/*'
minContentLength: 100
- name: envoy.filters.http.router
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
routeConfig:
virtualHosts:
- domains:
- '*'
name: backend
routes:
- match:
prefix: /stats/prometheus
route:
cluster: prometheus_stats
statPrefix: stats