Exporting custom metrics from JMX to Prometheus

Photo by Isaac Smith on Unsplash

Recently here at work we need to implement business metrics in one our services that would help us better understand how our customers were interacting with us and then optimize our operations.

This microservice in question has the following characteristics:

• Written in Java and using Thorntail
• Already exports JVM metrics to Prometheus using a JMX exporter

After the metrics exported in the Prometheus format, when the application is running in our cluster on Kubernetes, these metrics are collected from the pod by the cluster’s Prometheus and then made available in a datasource in Grafana.

With these definitions in mind, it was easy to export our new metrics, but first let’s undertang a little bit more about this pieces

If you want to jump and go straight to the source code, it can be found here

JMX

JMX is a Java api that was introduced in version 5, which provides a way to manage applications (locally or remotely), allowing to change or monitor the state of the application

Creating MBean class

We need create a Java interface with the operations we would like to expose, named in the following format MyCustomMBean, for example:

Registering MBean

To register our MBean,we can use an instance of MBeanServer, which can be obtained through ManagementFactory.getPlatformMBeanServer(). For example:

In my example, I used the CDI ApplicationScoped initialization event to register my Bean.

Prometheus JMX exporter

The jmx exporter project allows you to easily export our JMX metrics in the format that Prometheus can be scraping. The available documentation helps a lot in how we can be using and starting the agent responsible for making de metrics available, and if you have any doubts about how to make the configurations, the examples are very useful.

Starting application

To start our application, we need to inform the agent’s jar, port and configuration file, in our case it looks like this:

java -javaagent:./agent/jmx_prometheus.jar=9103:./agent/conf.yaml  \
 -Djava.util.logging.manager=org.jboss.logmanager.LogManager  \
 -Xbootclasspath/p:./agent/jboss-logmanager.jar \
 -jar target/custom-metrics-thorntail.jar

Check that http://localhost:9103/metrics contains your metrics in prometheus format, you can configure this url to be scraped

Full code can be found here.

I hope this has somehow been useful for you.