Observability in Quarkus 3
Observability in Quarkus
Observability on a software system can be described as the capability to allow a human to ask and answer questions. To enable developers and support engineers in understanding how their applications behave, Quarkus 3.3 includes many improvements to its main observability related extensions:
-
quarkus-opentelemetry(tracing) -
quarkus-micrometer(metrics)
OpenTelemetry
OpenTelemetry tracing is used to understand the flow of requests as they traverse through multiple services.
The quarkus-opentelemetry extension already had a major upgrade on Quarkus 3.0, where the configurations aligned with the ones used by the OpenTelemetry (OTel) community. This made available many configurations that were not previously available in Quarkus. Other improvements include:
-
The OpenTelemetry extension can be used in dev mode and now reloads properly.
-
OTel Service Provider Interface (SPI) hooks for Sampler and SpanExporter were made available along with the existing user implementations with CDI for many OTel classes:
IdGenerator, Resource attributes,SamplerandSpanProcessor. -
At the same time, the JDBC tracing activation was made simpler, just requiring the use of a property:
quarkus.datasource.jdbc.telemetry=trueOn Quarkus 3.3 many improvements were made to the quarkus-opentelemetry extension. The most impactful ones are
Removal of the OkHttp dependency
In previous versions of the Quarkus exporter used the upstream OTel libraries and leveraged the OkHttp library to send the spans to the OTel Collector. This unnecessary dependency was removed and replaced by Quarkus specific Vert.x GRPC and HTTP clients. As previously, the exporter continues to be automatically wired with CDI, that’s why the quarkus.otel.traces.exporter property defaults to cdi on Quarkus 3+.
Exporter support fot the HTTP protocol
Up until Quarkus 3.2, the OTel exporter could only use the GRPC protocol, while Quarkus now supports HTTP as well. To use the new HTTP protocol, the quarkus.otel.exporter.otlp.traces.protocol property must be set to http/protobuf. The quarkus.otel.traces.exporter.endpoint property must also be set to the OTel Collector HTTP endpoint. Here’s an example when using 4318, the default HTTP port on the OTel Collector:
quarkus.otel.exporter.otlp.traces.protocol=http/protobuf
quarkus.otel.exporter.otlp.endpoint=http://localhost:4318Exporters support TLS
Both GRPC and HTTP exporters now support Transport Layer Security (TLS) and custom certificates.
Like the other rest clients in Quarkus, the exporter’s certificate verification will also be disabled if you set:
quarkus.tls.trust-all=true|
This setting should not be used in production as it will disable any kind of SSL verification. |
Customise the propagation header
We added a way to customise the propagation header.
You can create a CDI bean implementing the TextMapPropagatorCustomizer interface.
This can be used to restrict propagation of OpenTelemetry trace headers and prevent potentially sensitive data to be sent to third party systems.
This is an example of a customizer that removes the Baggage header:
@Singleton
public static class TestTextMapPropagatorCustomizer implements TextMapPropagatorCustomizer {
@Override
public TextMapPropagator customize(Context context) {
TextMapPropagator propagator = context.propagator();
if (propagator instanceof W3CBaggagePropagator) {
return TextMapPropagator.noop();
}
return propagator;
}
}Identify the user in the spans
Valuable debugging information about the user related to each span can be added by setting:
quarkus.otel.traces.eusp.enabled=trueThe user’s ID and roles will be added to the span attributes, if available.
Hardening
Many bug fixes and small improvements were made to the extension, including:
-
Reduce the memory allocation needed to report spans.
-
Fix the
http.routespan attribute value in some cases. -
Properly report of the
service.namevalue. The value can be set in 3 different properties, from higher to lower priority:-
quarkus.otel.service.name -
quarkus.otel.resource.attributes -
quarkus.application.name
-
-
Exception’s stack traces are now reported in the span attributes.
Micrometer
Metrics in Quarkus are based on the Micrometer library. Data can be exported directly to Prometheus with the quarkus-micrometer-registry-prometheus extension. If you want to send data to an OTel Collector, you can use the quarkus-micrometer-registry-otlp Quarkiverse extension, among other options.
These were some of the recent improvements to the quarkus-micrometer extension:
Netty allocator metrics
The extension will now provide Netty allocation metrics by default. Netty uses memory allocators to pool the memory buffers for reuse. These metrics will give you a deeper understanding of the memory usage of your application. Direct and Heap memory usage will be reported.
To disable these metrics, please set:
quarkus.micrometer.binder.netty.enabled=falseCustom tags with HTTP server data
Customise emitted tags by creating a CDI bean implementing the HttpServerMetricsTagsContributor interface. This allows user code to compute arbitrary tags based on the details of HTTP requests. This is an implementation example were the Foo header value is used to set the foo tag.
@Singleton
public class HeaderTag implements HttpServerMetricsTagsContributor {
@Override
public Tags contribute(Context context) {
String headerValue = context.request().getHeader("Foo");
String value = "UNSET";
if ((headerValue != null) && !headerValue.isEmpty()) {
value = headerValue;
}
return Tags.of("foo", value);
}
}|
Only set tags with low cardinality values, meaning that the number of possible different values is low. For example, a tag with the HTTP method is a good candidate, but a tag with the HTTP full path is not. |
Use MeterRegistryCustomizer for arbitrary customizations to meter registries
By providing CDI beans that implement io.quarkus.micrometer.runtime.MeterRegistryCustomizer, the user code can change the configuration of any MeterRegistry that has been activated.
Unless an implementation is annotated with @io.quarkus.micrometer.runtime.MeterRegistryCustomizerConstraint, the customization applies to all MeterRegistry instances.
This is a method example with a customizer that sets the foo tag oo all metrics:
@Produces
@Singleton
public MeterRegistryCustomizer customizeAllRegistries() {
return new MeterRegistryCustomizer() {
@Override
public void customize(MeterRegistry registry) {
registry.config()
.meterFilter(MeterFilter.commonTags(List.of(
Tag.of("foo", "foo-value"))));
}
};
}and emitted metrics by implementing MeterRegistryCustomizer.
