-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
fix(telemetry-processor): Clarify push-based vs pull-based #16653
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: ref/telemetry-processor-dedupe-arch
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -28,7 +28,7 @@ Configurable via weights. | |||||
|
|
||||||
| #### TelemetryBuffer | ||||||
|
|
||||||
| The telemetry buffer on the backend must follow the common [telemetry buffer requirements](/sdk/foundations/processing/telemetry-processor/#telemetry-buffer). Here are the additional requirements for the backend-specific implementation: | ||||||
| Unlike the [push-based approach](/sdk/foundations/processing/telemetry-processor/#push-based-approach), the telemetry buffer does not push data to the telemetry scheduler. Instead, the [telemetry scheduler](#telemetryscheduler) iterates through all buffers using weighted round-robin and pulls data when it determines a buffer is ready to flush. Still, the telemetry buffer **MUST** follow the common [telemetry buffer requirements](/sdk/foundations/processing/telemetry-processor/#telemetry-buffer). Here are the additional backend-specific requirements: | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| 1. The telemetry buffer **SHOULD** drop older items as the overflow policy. It **MAY** also drop newer items to preserve what's already buffered. | ||||||
|
|
||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -98,11 +98,19 @@ The telemetry buffer batches high-volume telemetry items and forwards them to th | |
|
|
||
| 1. Before adding an item to a specific buffer, the telemetry buffer **SHOULD** drop rate-limited items to avoid overhead. If doing so, it **MUST** record client reports. | ||
| 2. When the telemetry buffer overflows and it drops data, it **MUST** record client reports. | ||
| 3. The telemetry buffer **MUST** forward low-volume data, such as normal events, session replays, or user feedback, directly to the telemetry scheduler. | ||
| 4. The telemetry buffer **MUST** start a timeout of 5 seconds when the first item is added. When the timeout expires, the telemetry buffer **MUST** forward all items to the telemetry scheduler. | ||
| 5. The telemetry buffer **MUST** define a size limit of `x` items. See [Size Limit Recommendations](#size-limit-recommendations) below for more details. | ||
| 6. When the size limit is reached, the telemetry buffer **MUST** forward all items to the telemetry scheduler. The buffer **MAY** forward items in batches. | ||
| 7. The telemetry buffer **SHOULD** use separate buffers per telemetry item type (e.g., one for spans, one for logs, one for metrics). | ||
| 3. The telemetry buffer **SHOULD** use separate buffers per telemetry item type (e.g., one for spans, one for logs, one for metrics). | ||
|
|
||
| ## Push-Based Approach | ||
|
|
||
| SDKs **SHOULD** use the push-based approach, where the telemetry buffers push data to the telemetry scheduler when full or on timeout. For high-throughput scenarios, SDKs **MAY** use a [pull-based alternative](./backend-telemetry-processor/) optimized for backends, where the scheduler pulls items from the buffers. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am torn between creating a pull-based approach section on the index page or just have it only on the backend page to avoid duplication. However even if we keep it as is, let's add a section for the pull-based approach so that this comment links to the section and not the whole page of the backend processor. |
||
|
|
||
| SDKs using the push-based approach **MUST** follow these requirements: | ||
|
|
||
| 1. The telemetry buffer **MUST** forward low-volume data, such as normal events, session replays, or user feedback, directly to the telemetry scheduler. | ||
| 2. The telemetry buffer **MUST** start a timeout of 5 seconds when the first item is added. When the timeout expires, the telemetry buffer **MUST** forward all items to the telemetry scheduler. | ||
| 3. The telemetry buffer **MUST** define a size limit of `x` items. See [Size Limit Recommendations](#size-limit-recommendations) below for more details. | ||
| 4. When the size limit is reached, the telemetry buffer **MUST** forward all items to the telemetry scheduler. The buffer **MAY** forward items in batches. | ||
|
|
||
|
|
||
| ## Batch Size Limit | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a bit more clear, but feel free to disregard.