OpenTelemetry in 2023
4 Years In, OpenTelemetry Delivers on its Promise for Open Observability
OpenTelemetry (OTEL) started in 2019 when the two dominant open-source projects around tracing, OpenTracing and OpenCencus, merged to form OpenTelemetry. The scope of the combined project expanded beyond tracing to cover all of observability. OTEL's mission is to help organizations deliver high-quality, ubiquitous, and portable telemetry.
Four years later, OTEL is making good on its promise. It has to date provided a stable standard for the three pillars of observability (metrics, logs, and traces), a collector that can receive, process, and export telemetry in any environment, and SDKs to instrument code in all major languages. It has also continued to expand its scope and introduced additional standards around semantic conventions and agent management.
Today, OTEL is the second most active project in the CNCF, behind only Kubernetes in popularity. Its contributors are spread across all major observability vendors and its protocol has near-universal adoption among observability providers.
In this post, we'll go over where OTEL is today, provide a summary of the various projects that fall under its domain, and talk about where it's going next.
OTEL - What's in a name?
OTEL consists mainly of two types of projects: specifications and implementations.
Specifications are the foundation of OTEL. They describe how telemetry should be captured, collected, processed, and exported. Specifications tend to be vendor-focused as they define the common standard that vendors have to implement to be compatible with OTEL.
Implementations are the specific client libraries and tooling that are used to handle telemetry data. Implementations tend to be end-user-focused as they consist of the parts that end-users use to instrument their code.
A project's stability is typically marked on a signal-by-signal basis. Each signal is a specific type of telemetry data - in OTEL, this means metrics, logs, and traces.
OTEL today consists of the following sub-projects:
OpenTelemetry Specification (specification)
OpenTelemetry SDKs (implementation)
OpenTelemetry Protocol (specification)
OpenTelemetry Collector (implementation)
Open Agent Management Protocol (specification)
OpenTelemetry Semantic Conventions (specification)
OpenTelemetry Specification (1.24.0)
The Open Telemetry specification is the foundation of OTEL - it provides the APIs, SDKs, and data models that all other OTEL standards are derived from.
A brief timeline of its development:
In terms of maturity, the OTEL spec is now stable for all signals. Because the logging signal was not deemed stable until earlier this year, many OTEL SDKs still do not support logging at this time.
OpenTelemetry SDKs (Mixed)
The OTEL SDKs provide client-side instrumentation based on the OTEL spec. For each language-specific SDK, there is a separate maturity level for each signal.
Auto Instrumentation
Depending on the programming language, some SDKs also support automatic instrumentation. This is when the SDK automatically injects signals, mainly traces, into your application without needing to manually instrument code.
Auto instrumentation doesn't work for compiled languages like go and rust. That said, you can still get automatic trace injection by going outside of the SDK and using eBPF and/or service mesh based tooling.
OpenTelemetry Protocol (OTLP - 1.0)
OTLP describes a common wire protocol for delivering observability data. There are two blessed transports in OTLP: http with protocol buffers and gRPC.
This spec is considered stable and can be implemented on any service that receives, processes, or exports OTEL data. The spec is implemented by the OpenTelemetry collector and also by agents of observability vendors like Grafana and Datadog.
OpenTelemetry Collector (0.83.0)
The OTEL collector is a vendor-agnostic agent that collects, transforms, and sends observability data.
The collector consists of the following components:
receivers: push/pull data from different sources
processors: transform/filter/enrich/derive data in flight
exporters: send data to downstream destinations
connectors: connectors are both a receiver and exporter and enable multiple pipelines to be stitched together
pipelines: a chain of receiver, zero or more processors, and exporters
extensions: components that provide additional capabilities outside of handling telemetry data (eg. basic auth, health checks, etc)
Together, the various components act as an observability pipeline, allowing you to collect telemetry data from any source, process it in flight, and send it to any destination.
The OTEL collector consists of two projects, otel-collector and otel-collector-contrib. otel-collector contains only the core components of the collector, chiefly logic strictly related to processing OLTP data.
otel-collector-contrib is a kitchen sink of every integration you can imagine, with exporters and receivers for most observability providers present. At the time of this writing, this includes 91 receivers, 48 exporters, and 24 processors.
End users are advised to use the OpenTelemetry Collector Builder to create a custom build of otel-collector-contrib with only components that they need.
In addition to otel-collector-contrib, vendors like AWS and Splunk provide their own custom distributions of OTEL.
Open Agent Management Protocol (OpAMP - Beta)
OpAMP is a network protocol for the remote management of agents. This is a recent addition to OTEL, introduced in 2022, and provides a vendor-agnostic standard for controlling fleets of agents. These agents could be instances of the otel-collector or vendor-specific agents that implement OpAMP.
With OpAMP, you can enable features like dynamic configuration deployments, agent updates, and credential management.
There is currently a work-in-progress go implementation of the OpAMP spec.
NOTE: If you're interested in managing OTEL agents via OpAMP - this is what my startup is working on.
OpenTelemetry Semantic Conventions (1.21.0)
The OTEL Semantic Conventions define a common set of attributes for observability data. They cover a wide range of areas including cloud resources, databases, exceptions, and systems.
The semantic conventions are used by the OTEL SDKs and automatically applied for SDKs that support auto instrumentation. Common semantics enables the ability to correlate between different signals and is something I'm especially excited by, the reason for which can (and probably will) be the subject of an entire blog post.
Etc
OTEL is a massive project. While we covered the flagship projects, there are additional parts of OTEL that deserve a notable mention:
OpenTelemetry Transformation Language (OTTL): a common transformation language for telemetry that is both performant and flexible - currently designed and implemented in
otel-collector-contribOTEL Demo: A micro-service-based shopping site that demonstrates a majority of features and language SDKs of OTEL
Final Thoughts
OTEL started as a merger between competing tracing specifications and evolved to become an industry standard for observability. The last four years laid the groundwork for building a common foundation across vendors and tools. The next four will reveal the fruits of that labor.
Are you having issues adopting OTEL or looking to migrate off your existing observability vendor? Checkout Nimbus - we provide a simple enterprise ready onramp into OTEL and vendor neutral observability
Thank you, this clarified some of the components relate to each other. I was struggling a bit with how the collectors operate, I was misunderstanding that they were just a way to build another Jaeger but that was a misunderstanding. They can be but they aren't. Sorry, been drinking from the OTel firehose the last few weeks and there's just like... a lot, or at least it feels like it so far.