OpenTelemetry table reference for Zerobus Ingest
This page provides reference information for the OpenTelemetry (OTLP) table schemas and data mapping used by Zerobus Ingest OTLP.
Table schema
When OTLP data arrives, Zerobus Ingest converts each record from the nested OTLP resource/scope/record hierarchy into a flat, denormalized row. Resource attributes and instrumentation scope information are embedded directly in each row, making the data immediately queryable without joins.
All attribute fields (attributes, resource.attributes, instrumentation_scope.attributes, body for logs, metadata for metrics) are stored as VARIANT columns. VARIANT is a semi-structured type in Delta Lake that stores JSON data while preserving the original types. For how each OTLP value type is stored, see Attribute values.
Each record is augmented with Databricks-specific fields:
Field | Description | Source |
|---|---|---|
| A system-generated ID for unique identification and time-ordered sorting. | Generated based on time |
| Timestamp in microseconds from the Unix epoch. | Timestamp (in microseconds) derived from |
| Date partition column, for efficient time-range filtering. | Derived from |
| Top-level column for efficient filtering by service name, as defined in the OTel semantic convention. | Extracted from |
Numeric precision and signed types
Delta types differ from OTLP types in the following ways:
- Integer metric values lose precision above 2^53.
Gauge,Sum, andExemplardata points that report their value asas_int, a 64-bit integer, are converted toDOUBLE, since Zerobus Ingest stores all metric values in a single floating-point column.DOUBLEonly represents integers exactly up to 2^53, about 9 quadrillion, so largeras_intvalues lose exact precision after conversion. This mainly matters for very large cumulative counters, such as lifetime byte or request totals. - Unsigned OTLP fields are stored as signed.
INTandBIGINTare always signed in Delta, but OTLP defines some numeric fields as unsigned (uint32,uint64,fixed64) or as very large fixed-width counters. Fields such as*_time_unix_nano,flags,dropped_attributes_count, etc., are stored as-is in a signedINTorBIGINTcolumn. A value above the signed maximum (i32::MAXforINT,i64::MAXforBIGINT) wraps to a negative number instead of erroring.
Schema mapping
Zerobus Ingest maps OTLP data to Delta table columns as described below.
Denormalization
In the OTLP protocol, telemetry data is nested like so.
ResourceSpans (or ResourceLogs, ResourceMetrics)
└── Resource (attributes, schema_url)
└── ScopeSpans (or ScopeLogs, ScopeMetrics)
└── InstrumentationScope (name, version, attributes)
└── Span (or LogRecord, Metric)
Zerobus Ingest flattens this hierarchy so that each row contains the full context:
resource: A struct containing the resource attributes (asVARIANT) anddropped_attributes_count.resource_schema_url: The schema URL from the enclosing ResourceSpans, ResourceLogs, or ResourceMetrics.instrumentation_scope: A struct containing the scope name, version, attributes (asVARIANT), anddropped_attributes_count.span_schema_url/log_schema_url/metric_schema_url: The schema URL from the enclosing ScopeSpans, ScopeLogs, or ScopeMetrics.
Metric data points
Each metric data point is stored as its own row. The metric's name, description, unit, and metadata are repeated on every row, and the data point's start_time_unix_nano and time_unix_nano are stored in the top-level columns of the same name.
metric_type is gauge, sum, histogram, exponential_histogram, or summary. The data point is stored in the struct column with that name, and the other four struct columns are NULL.
Attribute values
Attribute values and log bodies are OTLP AnyValue values, which are stored as follows:
OTLP value type | Stored as |
|---|---|
| String |
| Boolean |
| Integer |
| Double. |
| Array |
| Object |
| Lowercase hex-encoded string. For example, the bytes |
A bytes_value can't be told apart from a string_value that holds the same text. To get the original bytes, decode it with unhex, for example unhex(attributes:['payload']::string).
If the same key appears more than one time in an attribute list or a kvlist_value, the last value is stored.
ID encoding
trace_id, span_id, and parent_span_id are stored as lowercase hex-encoded strings:
trace_id: 32-character hex string (16 bytes)span_idandparent_span_id: 16-character hex string (8 bytes)
An ID is invalid if it has the wrong length or is all zeros. For a span's own trace_id and span_id, see Rejected records. Other missing and invalid IDs are handled as follows:
- A missing or invalid
parent_span_idis stored asNULL, so the span is stored as a root span. - A missing or invalid
trace_idorspan_idon a log record is stored asNULL, and the log record is ingested. - The
trace_idandspan_idof span links and exemplars are stored hex-encoded as received, without validation.
Enum encoding
kind, status.code, and severity_number are stored as their string names as defined in the OTLP specification, for example SPAN_KIND_SERVER, STATUS_CODE_OK, and SEVERITY_NUMBER_INFO. A value that the OTLP specification doesn't define is stored as its number, for example 42.
aggregation_temporality is stored without the AGGREGATION_TEMPORALITY_ prefix, as DELTA or CUMULATIVE.
Rejected records
Zerobus Ingest rejects a record that fails any of the following checks. For how rejected records are reported, see Partial success.
A span is rejected when:
trace_idorspan_idis missing or invalid.start_time_unix_nanois missing.
A metric data point is rejected when:
- Its metric has no
name, or has anaggregation_temporalityvalue that the OTLP specification doesn't define. All of the metric's data points are rejected. time_unix_nanois missing or 0.- A gauge or sum data point has no value.
- A histogram data point has
explicit_boundsbut nobucket_counts, has a number ofbucket_countsother than the number ofexplicit_boundsplus one, or hasbucket_countsthat don't add up tocount. - An exponential histogram data point has positive bucket counts, negative bucket counts, and
zero_countthat don't add up tocount. - A histogram, exponential histogram, or summary data point has a
countof 0 and asumother than 0.
Data points with the NO_RECORDED_VALUE flag set skip the value, count, sum, and bucket checks.
Log records aren't rejected for their content. For how a log record's missing or invalid IDs are stored, see ID encoding.
Next steps
- Configure OpenTelemetry (OTLP) clients to send data to Unity Catalog: Configure an OpenTelemetry SDK or Collector to send traces, logs, and metrics to Zerobus Ingest.
- Query OpenTelemetry data: Example SQL queries for exploring your telemetry data.
- Zerobus Ingest error handling: Troubleshoot common errors and error codes.
- Zerobus Ingest quotas: Review throughput and retention limits for Zerobus Ingest.