OpenTelemetry for Growing Teams: Find the Cause of Slow and Failed Requests

Imagine a customer reporting occasional long waits during checkout. The site is available, the servers look fine, and there don’t seem to be any errors. However, the customer support cannot reproduce the issue regularly. Lack of a proper means of tracing the request throughout its execution results in different theories as to what is going on.

Observability of the application allows asking better questions about the situation. Metrics will help identify a pattern; traces will demonstrate how the request traveled, and logs will describe any related events. OpenTelemetry is a vendor-neutral solution for instrumenting and collecting such information. It is not by itself the whole set of features for storing and visualizing the information.

The guide provides a methodology for designing OpenTelemetry observability in relation to meaningful business journeys in a scaling organization. This includes instrumentation, context, privacy, alerting, cost, and operations ownership considerations. All examples here are meant as illustrations and not actual Digitonix incidents. The idea is to provide meaningful data points for making decisions, not just numbers that have no connection with the process the customers need to complete.

Start with a relevant journey

Identify a particular activity – from submitting an inquiry, making an order, uploading files to customer record synchronization. Define success from the customer's point of view and identify application components necessary for that. Successful response from the server does not mean the actual work was done. Identify those business and technical events that will prove the journey resulted in the desired outcome.

Ask support and operations about which failures are hard to understand today. Mention delays, partial success, and ambiguity in state, in addition to outright failure. Pick something small enough initially and owned by someone who will make use of the insights gained. It’s more likely that instrumentation will be of continued use if it helps answer a repeating operational question rather than just being everywhere for no good reason.

Be clear on the differences between the signals

Metrics aggregate measurements over time. Traces document the flow and timing involved in handling a request or operation. Logs record events along with information needed to understand what happened. They rely on having clear semantics and context. OpenTelemetry observability primer and signal documentation will give you a start on the technical aspects of this approach.

Choose signals based on the questions you ask. There could be a trend among the failed imports in one signal but a trace can help identify a slow dependency in a certain import. The log explains why an import was validated. Gathering more of one type of signal will not automatically mean that the other signals become unnecessary. Think about how operators shift among signals while troubleshooting a certain problem.

Define identity of services consistently

Come up with consistent terms to refer to applications, services, environments, and releases. This makes it easy to tell production apart from the testing environment as well as compare the behavior of different versions of a release. Do not use a bunch of terms made up by each member of the team. It should be easy to understand the whole observability system.

Provide sufficient context for deployment without including sensitive or unnecessary information. The operator needs to know which version corresponds to the problem and who is responsible for it. When multiple pieces have some generic names, the context becomes confusing. These definitions must be regarded as part of the application contract and managed accordingly, instead of a loose collection of dashboard preferences.

Instrument useful boundaries before adding internal detail

Start from useful boundaries: incoming requests, application operations, and dependency calls. Check what is instrumented automatically by the existing libraries, and add specific context to answer questions about your application. Look through the latest integration guides corresponding to your version. Do not expect that installing one library will cover all the processes and business operations within the application.

Ensure instrumentation remains comprehensible for the developers who are maintaining the code. Clarify why a span or metric is custom and what investigation question it supports. Refrain from placing redundant instrumentation everywhere without a reason. Instrumentation in Node.js and Python development can vary, yet both require consistency in their definition and operational verification in order to yield meaningful data.

Propagate context through the real path of the request

The request can be handled via a frontend service, API, queue, and worker. Design how trace context will be passed through each boundary in use by the application. Check that the path is correct through a realistic scenario. It is not enough to have a diagram showing all the services interconnected - telemetry can fail to connect all of them even if there is just one boundary forgotten somewhere in the middle.

Exercise care regarding trust limits and the data bound to context. Do not assume that incoming context is verification of authority, nor provide private data by means of propagated fields. Adhere to the documentation for your chosen instrumentation as well as the security requirements for the application. Context is meant to tie together evidence, whereas authentication and authorization are still responsibilities of application controls guarding the real action.

Track background tasks as a workflow on their own

An API call can take a task and have a worker do it long after. Look at the age of the queue, the results of processing, and the correlation between accepted and completed work where it matters. Define the states that operators need to know about. A low error rate in HTTP calls does not mean there isn't a backlog building up in the application's queue.

Provide long-term jobs with stable references suitable for analysis without making each individual reference a metric tag. Capture meaningful progress and failure information in a relevant signal. Include retry attempts and failed tasks in the design. Support should be able to distinguish between a job waiting in a normal queue and one that has ceased to make progress, and operations should know the proper way of recovering or reconciling.

Attributes to describe the problem

Some good attribute candidates would include the operation type, a route template, or a class of controlled outcomes. Choose attributes based on the investigation questions and on the capabilities of the selected backend. Be careful not to capture every available field just because it is easy. Excessive amounts of inconsistent context may complicate searching and incur costs without providing any insight into what went wrong with the customer experience or caused delays.

Look at cardinality before using an attribute as a metric dimension. Each unique customer ID, full URL with different query strings, and each unique request ID can generate many series. Make sure you have proper controlled dimensions for aggregated metrics and document the investigation context somewhere else. Document your reasoning so that your instrumentation doesn't silently alter cost and ease-of-use of the monitoring infrastructure when you roll out a feature.

Protect sensitive data from being collected in telemetry

Describe all the possible information that could be captured by the instrumentation, including request data, headers, exception message, and database data. Decide on what information is important to collect and what filtering/redaction will occur before the information leaves the application/collector. Don’t assume that any logging level filters out personal data. Developers need to look at output based on real requests, including errors where input is copied into the error message.

Establish access and retention policies for the chosen backend in accordance with organizational needs. Telemetry can uncover operational data even without customer names. Ensure enough data to allow debugging without making unnecessary copies of company documents. Include the choice in the handover and change evaluation, so the new integration will not be collecting a particular field that was deliberately excluded from the original design.

Learn about the roles of collector and backend

The OpenTelemetry Collector is capable of receiving, processing, and exporting telemetry based on its configuration. The chosen backend will store, query, show dashboards, or alert. Take a look at the real architecture and role of each component. Installation of a collector will not result in a fully-fledged monitoring system, and connecting the backend will not necessarily mean the arrival of all the required signals to it.

Configuration, updating, and the path for each signal need to be documented. Evaluate how failures between different components are managed and what the operator can know about the telemetry pipeline itself. The halt in the data collection process does not imply that there are no new errors in the application. Cloud and DevOps services might be employed for the deployment and observation of the selected solution.

View the sampling process as the trade-off in terms of evidence

The traces might be sampled to limit the number and costs of the generated data. Make sure that you understand the approach applied by the selected instrumentation and pipeline, in particular, the places of decision making. Sampling determines the subset of requests available for analysis. The lack of traces does not mean that there were no failures.

Compare trace evidence with relevant aggregate metrics and logs. Consider whether the sampling strategy captures slow requests or important failures within the selected tools' capabilities. Test representative traffic, document limitations, and revisit the strategy when the application changes.

Building dashboards to make decisions

It is expected from a dashboard to assist the user in answering questions like "Can customers pay?" Include a relevant journey and environment in the picture and put the context of time next to your metrics. Use units correctly and define the outcome of the measurements. Do not place the charts simply because they are nice; the operator should understand the possible action in response to changing the numbers.

Consider adding a path from the summary to an example or an event trace where applicable. A spike in duration can be more valuable if there is an ability to look at the related trace or event. Go through the dashboard together with those who will use it. In case there is a necessity for an explanation of each panel by the original developer, make your presentation simpler or add a small guide of how to operate with the dashboard.

Alert on conditions that require action

Decide on alerting based on the impact and response options. Define the owner, the first step required and the escalation process. The alert on something that cannot be done about makes people learn how to neglect the whole system. Consider choosing conditions that really reflect the service problems and differentiate the urgent ones from the information that can be checked during usual working hours.

Assess alerts based on representative incidents and regular variation. What seems like an appropriate threshold at a low volume may generate noise during an expected import or promotion. Note what kind of maintenance handling you expect, and do not suppress any information without understanding the impact. The goal is to get fast and meaningful response. Having more alerts is not always better if it hides a handful of cases where intervention is needed.

Relate release changes to operational evidence

Log all the version and changes made for your ability to check the difference between the behavior prior to the release and its behavior afterwards. New delay can be associated with the change in the code, configuration, data, or third-party dependency. Correlation will help you to focus on the issue, but it will not prove cause-and-effect relationship.

Rollback and remediation must be done as part of the team's authorized release process. Observability should offer data for making that decision, not quietly cause a different significant change without a designed approach. Development services will help instrument the application, and deployment and operation needs will dictate how it's used for the release process and after that. See Software development services for related information.

Telemetry validation by means of exercise

Generate slow and failure operations in an authorized environment. Make sure that the telemetry that you expect to see appears in the appropriate metric, trace, and event with the proper context. It is necessary to make sure that there is no sensitive information in the pipeline and that it is possible to debug it. If it is possible, create a case of missing dependency or failing to export telemetry. Just the appearance of a regular request on a chart is not enough.

Have an operator examine by leveraging the tools and guide provided. Document areas where there is insufficient or unclear evidence. Make changes to instrumentation and presentation prior to the roll out. There could be evidence from AWS, Azure, or Google Cloud environments that could be combined with application evidence, although this has to be verified for the particular deployment.

Assess telemetry cost and operations ownership

Evaluate the volume, retention, export, and backend costs for the initial scope. Check the real-world use during the pilot and contrast it with assumptions. The instrumentation can make a big difference in terms of volume as traffic or attributes are changed. Ensure that the appropriate team has access to evaluate the cost in relation to value, and thus avoid stripping out the necessary investigation signal just because no one knows who pays for the bill.

Assign ownership to instrumentations, collector settings, dashboards, alerts, and back-end access. Some of these may share duties, but there should be a clear decision-making process for each. This includes maintenance and onboarding for new operators. A system that relies on the knowledge of one person privately becomes inefficient even if all technical aspects keep functioning and transmitting information as intended.

How Digitonix can contribute to observability

Prepare one incident timeline for the discussion. Make sure to note what the customer complained about, the date, the evidence, and the unanswered questions. Delete unnecessary information, such as any identifying data. This case will make it much simpler to differentiate between a problem with instrumentations, routing of alerts, dashboarding, or operation process performed by the team while investigating an issue.

Take a critical customer journey, recent events, an application architecture diagram, and the current monitoring solution. Digitonix can be contacted about instrumentation, data collection, dashboards, alerting, and the extent of operations. Require use cases showing investigations and not just telemetry installation. Determine who is responsible for privacy compliance, backend costs, maintenance, and response to alerts.

Determine what Digitonix can do and then contact them with one clear issue to describe. An ideal initial consultation would include analysis of the journey and its dependencies, missing evidence, and creating an operations playbook. Scaling up would be based on their real questions and their ability to apply the result. See technology capabilities and contact the team for related information.

Frequently Asked Questions

Is OpenTelemetry a monitoring dashboard?

OpenTelemetry offers instrumentation and collection functionalities for telemetry. Storage, querying, visualization, and alerting are dependent on the backend used as well as the overall architecture. Look into the responsibilities of each of the components in your architecture. A good implementation should also involve defined signals, processes, and people who can troubleshoot any problem that arises. Instrumentation alone cannot deliver product and process decisions automatically.

Do we really require metrics, traces, and logs for all the features?

Select the right signals based on the questions you ask and the costs involved in doing so. Begin with an important customer journey and figure out what signal will bring out the information you are looking for. Do not collect everything just because you can. Verify that the existing data can explain your failures and delays and then move forward filling any gap left behind.

Why are unique customer IDs dangerous to use in metric labels?

They will be able to generate many different kinds of series, increasing cost and complicating aggregation. They may also inadvertently share too much information. Check cardinality and privacy implications of the selected backend. Utilize controlled dimensions for aggregate measures and adequate detailed data points for analysis, with proper access and retention policies to suit organizational needs and data being collected.

If there is no trace, does it mean the request was successful?

No. Sampling, lack of instrumentation, or pipeline error could result in a request without any trace. Consider trace evidence in relation to other important indicators and ensure the soundness of your trace collection process. Know the sampling approach used and be clear on the conclusions supported by evidence and the ones that aren’t, particularly in the case of rare failures or comparison of a few requests against application behavior as a whole.

What should the observability proof of concept show?

This will aid the operator in explaining a representative slow or failing journey using the appropriate tools. It will provide useful context, cross-boundary correlations, privacy checks, alerts that can be acted upon, and a means of auditing the pipeline. Limitations and maintenance requirements need to be logged. Visualizing traffic through a dashboard is useful to get started but does not capture the investigative requirement of the business.