[FIXED] Spring boot actuator metrics for Prometheus in Consul Connect

Issue

I have a spring boot application running in a Nomad cluster with Consul Connect enabled.

network {
  mode = "bridge"
}

service {
  name = "api"
  port = "9966"

  connect {
    sidecar_service {}
  }
}

There is no port mapping defined and the API is reachable only within consul service mesh through the proxy. Now I have prometheus running in the same cluster. How does prometheus discover the individual API instances and scrape metrics out of it. I used the below config. But without having a mapping port in the host level, it is not able to reach the individual API instance.

- job_name: 'actuator'
    metrics_path: /api/actuator/prometheus
    consul_sd_configs:
    - server: '{{ env "NOMAD_IP_prometheus_ui" }}:8500'
      services: ['api']

How to solve this problem? What is general practice to scrape metrics from a spring boot application running inside a service mesh with no host port mapping?

Solution

Finally found it. Nomad has an option to expose a particular endpoint via another sidecar proxy without mTLS authentication. The use case of this option is specifically for health check or metrics.

https://www.nomadproject.io/docs/job-specification/expose#expose-examples

The expose stanza inside connect stanza helps to achieve this.

connect {
        sidecar_service {
          proxy {
            expose {
              path {
                path             =  "/actuator/prometheus"
                protocol         =  "http"
                local_path_port  =  9966
                listener_port    =  "metrics"
              }
            }
          }
        }
      }

Answered By - Jawahar