SignalFx Tracing Library for .NET

The SignalFx Tracing Library for .NET provides an OpenTracing-compatible tracer and automatically configured instrumentations for popular .NET libraries and frameworks. It supports .NET Core 2.0+ on Linux and Windows and .NET Framework 4.5+ on Windows.

Where applicable, context propagation uses B3 headers.

The SignalFx-Tracing Library for .NET implements the Profiling API and should only require basic configuration of your application environment.

You can link individual log entries with trace IDs and span IDs associated with corresponding events. If your application uses a supported logger, enable trace injection to automatically include trace context in your application's logs. For more information, see Inject traces in logs.

Supported libraries and frameworks

There are known .NET Core runtime issues for version 2.1.0 and 2.1.2.

Library	Versions Supported	Notes
ADO.NET	Supported .NET versions	Disable sanitization of `db.statement` with `SIGNALFX_SANITIZE_SQL_STATEMENTS=false` (`true` by default)
ASP.NET Core MVC	2.0+	`Microsoft.AspNet.Mvc.Core` NuGet and built-in packages. Include additional applicable Diagnostic Listeners with `SIGNALFX_INSTRUMENTATION_ASPNETCORE_DIAGNOSTIC_LISTENERS='Listener.One,Listener.Two'`
ASP.NET MVC on .NET Framework	`System.Web.Mvc` 4.x and 5.x
ASP.NET Web API 2 on .NET Framework	`System.Web.Http` 5.1+
Confluent.Kafka	`Confluent.Kafka` NuGet [1.4.0, 2)
Elasticsearch.Net	`Elasticsearch.Net` NuGet 5.3 - 7.x	Disable `db.statement` tagging with `SIGNALFX_INSTRUMENTATION_ELASTICSEARCH_TAG_QUERIES=false` (`true` by default, which may introduce overhead for direct streaming users).
GraphQL	`GraphQL` NuGet [2.3, 3)	Currently only instruments validation and execution functionality.
HttpClient	Supported .NET versions	by way of `System.Net.Http.HttpClientHandler` and `HttpMessageHandler` instrumentations
MongoDB	`MongoDB.Driver.Core` NuGet 2.1.0+	Disable `db.statement` tagging with `SIGNALFX_INSTRUMENTATION_MONGODB_TAG_COMMANDS=false` (`true` by default).
Npgsql	`Npqsql` NuGet 4.0+	Provided via enhanced ADO.NET instrumentation
RabbitMQ	`RabbitMQ.Client` NuGet [3.6.9, 7)
ServiceStack.Redis	`ServiceStack.Redis` NuGet 4.0+	Disable `db.statement` tagging with `SIGNALFX_INSTRUMENTATION_REDIS_TAG_COMMANDS=false` (`true` by default).
StackExchange.Redis	`StackExchange.Redis` NuGet 1.0+	Disable `db.statement` tagging with `SIGNALFX_INSTRUMENTATION_REDIS_TAG_COMMANDS=false` (`true` by default).
WCF (Server)	`System.ServiceModel` 4.x	Client requests using `WSHttpBinding` or `BasicHttpBinding` are instrumented via the `WebClient` instrumentation. There is no client-side span for `NetTcpBinding`.
WebClient	Supported .NET versions	by way of `System.Net.WebRequest` instrumentation

Configure the SignalFx Tracing Library for .NET

Configuration values

Use these environment variables to configure the tracing library:

Environment variable	Default value	Description
`SIGNALFX_ACCESS_TOKEN`		The access token for your SignalFx organization. Providing a token enables you to send traces to a SignalFx ingest endpoint.
`SIGNALFX_ADD_CLIENT_IP_TO_SERVER_SPANS`	`false`	Enable to add the client IP as a span tag when creating a server span.
`SIGNALFX_APPEND_URL_PATH_TO_NAME`	`false`	Enable to append the absolute URI path to the span name.
`SIGNALFX_ASPNET_TEMPLATE_NAMES_ENABLED`	`true`	Feature Flag: enables updated resource names on `aspnet.request`, `aspnet-mvc.request`, `aspnet-webapi.request`, and `aspnet_core.request` spans. Enables `aspnet_core_mvc.request` spans and additional features on `aspnet_core.request` spans.
`SIGNALFX_DIAGNOSTIC_SOURCE_ENABLED`	`true`	Enable to generate troubleshooting logs with the `System.Diagnostics.DiagnosticSource` class.
`SIGNALFX_DISABLED_INTEGRATIONS`		The integrations you want to disable, if any, separated by a semi-colon. These are the supported integrations: AspNetMvc, AspNetWebApi2, DbCommand, ElasticsearchNet5, ElasticsearchNet6, GraphQL, HttpMessageHandler, IDbCommand, MongoDb, NpgsqlCommand, OpenTracing, ServiceStackRedis, SqlCommand, StackExchangeRedis, Wcf, WebRequest
`SIGNALFX_DOTNET_TRACER_CONFIG_FILE`	`%WorkingDirectory%/signalfx.json`	The file path of a JSON configuration file that will be loaded.
`SIGNALFX_ENDPOINT_URL`	`http://localhost:9080/v1/trace`	The hostname and port for a SignalFx Smart Agent or OpenTelemetry Collector.
`SIGNALFX_ENV`		The value for the `environment` tag added to every span. This determines the environment in which the service is available in SignalFx µAPM.
`SIGNALFX_FILE_LOG_ENABLED`	`true`	Enable file logging. This is enabled by default.
`SIGNALFX_INSTRUMENTATION_ASPNETCORE_DIAGNOSTIC_LISTENERS`		Comma-separated list of diagnostic listeners that you subscribe to an observer.
`SIGNALFX_INSTRUMENTATION_ELASTICSEARCH_TAG_QUERIES`		Enable to tag the Elasticsearch command `PostData` as a `db.statement`.
`SIGNALFX_INSTRUMENTATION_MONGODB_TAG_COMMANDS`		Enable to tag the Mongo command `BsonDocument` as a `db.statement`.
`SIGNALFX_INSTRUMENTATION_REDIS_TAG_COMMANDS`		Enable to tag Redis commands as a `db.statement`.
`SIGNALFX_LOGS_INJECTION`	`false`	Enable to inject trace IDs, span IDs, service name and environment into logs. This requires a compatible logger or manual configuration.
`SIGNALFX_MAX_LOGFILE_SIZE`	`104857600` (10MiB)	The maximum size for tracer log files, in bytes.
`SIGNALFX_OUTBOUND_HTTP_EXCLUDED_HOSTS`		A semicolon-separated list of hosts for which HTTP outbound spans are not created.
`SIGNALFX_PROFILER_EXCLUDE_PROCESSES`		Sets the filename of executables the profiler cannot attach to. If not defined (default), the profiler will attach to any process. Supports multiple values separated with semi-colons, for example: `MyApp.exe;dotnet.exe`
`SIGNALFX_PROFILER_PROCESSES`		Sets the filename of executables the profiler can attach to. If not defined (default), the profiler will attach to any process. Supports multiple values separated with semi-colons, for example: `MyApp.exe;dotnet.exe`
`SIGNALFX_RECORDED_VALUE_MAX_LENGTH`	`1200`	The maximum length an attribute value can have. Values longer than this are truncated.
`SIGNALFX_SANITIZE_SQL_STATEMENTS`		Enable to stop sanitizing each SQL `db.statement`.
`SIGNALFX_SERVICE_NAME_PER_SPAN_ENABLED`		Enable to allow manual instrumentation to have a different service name than the one you specify with `SIGNALFX_SERVICE_NAME`. Add a tag `service.name` with the desired name to the manual instrumentation.
`SIGNALFX_SERVICE_NAME`		The name of the service.
`SIGNALFX_STDOUT_LOG_ENABLED`	`false`	Enables `stdout` logging. This is disabled by default.
`SIGNALFX_SYNC_SEND`	`false`	Enable to send spans in synchronous mode when the root span is closed. Sending spans in synchronous mode is generally recommended for only tests, but can also be useful for some special scenarios.
`SIGNALFX_TRACE_DEBUG`	`false`	Enable to activate debugging mode for the tracer.
`SIGNALFX_TRACE_DOMAIN_NEUTRAL_INSTRUMENTATION`	`false`	Sets whether to intercept method calls when the caller method is inside a domain-neutral assembly. This is recommended when instrumenting IIS applications.
`SIGNALFX_TRACE_GLOBAL_TAGS`		Comma-separated list of key-value pairs to specify global span tags. For example: `"key1:val1,key2:val2"`
`SIGNALFX_TRACE_LOG_PATH`	Linux: `/var/log/signalfx/dotnet/dotnet-profiler.log` Windows: `%ProgramData%"\SignalFx .NET Tracing\logs\dotnet-profiler.log`	The path of the profiler log file.
`SIGNALFX_TRACE_RESPONSE_HEADER_ENABLED`	`true`	If set to true enables adding `Server-Timing` header to the server HTTP responses.
`SIGNALFX_TRACING_ENABLED`	`true`	Enable to activate the tracer.
`SIGNALFX_USE_WEBSERVER_RESOURCE_AS_OPERATION_NAME`	`true`	Enable to specify the resource name as the span name. This applies only to AspNetMvc and AspNetWebApi.

Ways to configure

There are following ways to apply configuration settings (priority is from first to last):

[Environment variables)(#environment-variables)
web.config or app.config file
JSON configuration file

Environment variables

Environment variables are the main way to configure values. A setting configured via an environment variable cannot be overridden.

Web.config and app.config

For an application running on .NET Framework, web configuration file (web.config) or application configuration file (app.config) can be used to configure settings.

See example with SIGNALFX_SERVICE_NAME overload.

<configuration>
  <appSettings>
    <add key="SIGNALFX_SERVICE_NAME" value="my-service-name" />
  </appSettings>
</configuration>

Json configuration file

By default, if SIGNALFX_DOTNET_TRACER_CONFIG_FILE is unset, the application is searching for signalfx.json in the current working directory (acquired by Environment.CurrentDirectory).

See example with SIGNALFX_SERVICE_NAME overload.

{
    "SIGNALFX_SERVICE_NAME": "my-service-name"
}

Setup

Linux

After downloading the library, install the CLR Profiler and its components via your system's package manager.

Download the latest release of the library.

Install the CLR Profiler and its components with your system's package manager:

# Use dpkg:
$ dpkg -i signalfx-dotnet-tracing.deb

# Use rpm:
$ rpm -ivh signalfx-dotnet-tracing.rpm

# Install directly from the release bundle:
$ tar -xf signalfx-dotnet-tracing.tar.gz -C /

# Install directly from the release bundle for musl-using systems (Alpine Linux):
$ tar -xf signalfx-dotnet-tracing-musl.tar.gz -C /

Configure the required environment variables:

$ source /opt/signalfx-dotnet-tracing/defaults.env

Set the service name:

$ export SIGNALFX_SERVICE_NAME='MyCoreService'

Set the endpoint of a Smart Agent or OpenTelemetry Collector:

$ export SIGNALFX_ENDPOINT_URL='http://<YourSmartAgentOrCollector>:9080/v1/trace'

Optionally, enable trace injection in logs:
```
$ export SIGNALFX_LOGS_INJECTION=true
```

Optionally, create the default logging directory:

$ source /opt/signalfx-dotnet-tracing/createLogPath.sh

Run your application:
```
$ dotnet run
```

Azure Function Custom Linux Docker Image

The CLR Profiler can be added to Docker images using any of the provided packages, below an example of adding it to an Azure Function image:

# These first lines are from the example but in principle could be
# build outside of docker, but they make the example clear showing
# what should be copied to the Azure VM machine.
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS installer-env

COPY . /src/dotnet-function-app
RUN cd /src/dotnet-function-app && \
    mkdir -p /home/site/wwwroot && \
    dotnet publish *.csproj --output /home/site/wwwroot

FROM mcr.microsoft.com/azure-functions/dotnet:3.0
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

# Custom lines Adding the SignalFx Auto-Instrumentation to the image:

# First install the package. This example downloads the latest version
# alternatively download a specific version or use a local copy.
ARG TRACER_VERSION=0.1.13
ADD https://github.com/signalfx/signalfx-dotnet-tracing/releases/download/v${TRACER_VERSION}/signalfx-dotnet-tracing_${TRACER_VERSION}_amd64.deb /signalfx-package/signalfx-dotnet-tracing.deb
RUN dpkg -i /signalfx-package/signalfx-dotnet-tracing.deb
RUN rm -rf /signalfx-package

# Prepare the log directory (useful for local tests).
RUN mkdir -p /var/log/signalfx/dotnet && \
    chmod a+rwx /var/log/signalfx/dotnet

# Set the required environment variables. In the case of Azure Functions more
# can be set either here or on the application settings. 
ENV CORECLR_ENABLE_PROFILING=1 \
    CORECLR_PROFILER='{B4C89B0F-9908-4F73-9F59-0D77C5A06874}' \
    CORECLR_PROFILER_PATH=/opt/signalfx-dotnet-tracing/SignalFx.Tracing.ClrProfiler.Native.so \
    SIGNALFX_INTEGRATIONS=/opt/signalfx-dotnet-tracing/integrations.json \
    SIGNALFX_DOTNET_TRACER_HOME=/opt/signalfx-dotnet-tracing
# End of SignalFx customization.

COPY --from=installer-env ["/home/site/wwwroot", "/home/site/wwwroot"]

For more information on how to configure a custom Linux Azure Function image refer to the Azure documentation.

Windows

Warning: Pay close attention to the scope of environment variables. Ensure they are properly set prior to launching the targeted process. The following steps set the environment variables at the machine level, with the exception of the variables used for finer control of which processes will be instrumented, which are set in the current command session.

Install the CLR Profiler using an installer file (.msi file) from the latest release. Choose the installer (x64 or x86) according to the architecture of the application you're instrumenting.

Configure the required environment variables to enable the CLR Profiler:

For .NET Framework applications:

setx COR_PROFILER "{B4C89B0F-9908-4F73-9F59-0D77C5A06874}" /m

For .NET Core applications:

setx CORECLR_PROFILER "{B4C89B0F-9908-4F73-9F59-0D77C5A06874}" /m

Set the "service name" that better describes your application:
```
setx SIGNALFX_SERVICE_NAME MyServiceName /m
```
Set the endpoint of a Smart Agent or OpenTelemetry Collector that will forward the trace data:
```
setx SIGNALFX_ENDPOINT_URL http://localhost:9080/v1/trace /m
```
Optionally, enable trace injection in logs:
```
setx SIGNALFX_LOGS_INJECTION true /m
```
Optionally, if instrumenting IIS applications add the following environmet variable set to true:
```
setx SIGNALFX_TRACE_DOMAIN_NEUTRAL_INSTRUMENTATION true /m
```
Enable instrumentation for the targeted application by setting the appropriate CLR enable profiling environment variable. You can enable instrumentation at these levels:

For current command session
For a specific Windows Service
For a specific user The follow snippet describes how to enable instrumentation for the current command session according to the .NET runtime. To enable instrumentation at different levels, see this section.
- For .NET Framework applications:
```
set COR_ENABLE_PROFILING=1
```
- For .NET Core applications:
```
set CORECLR_ENABLE_PROFILING=1
```

Restart your application ensuring that all environment variables above are properly configured. If you need to check the environment variables for a process use a tool like Process Explorer.

Enable instrumentation at different levels

Enable instrumentation for a specific Windows service:

For .NET Framework applications:

reg add HKLM\SYSTEM\CurrentControlSet\Services\<ServiceName>\Environment /v COR_ENABLE_PROFILING /d 1

For .NET Core applications:

reg add HKLM\SYSTEM\CurrentControlSet\Services\<ServiceName>\Environment /v CORECLR_ENABLE_PROFILING /d 1

Enable instrumentation for a specific user:

For .NET Framework applications:

setx /s %COMPUTERNAME% /u <[domain/]user> COR_ENABLE_PROFILING 1

For .NET Core applications:

setx /s %COMPUTERNAME% /u <[domain/]user> CORECLR_ENABLE_PROFILING 1

Configure custom instrumentation

You can build upon the provided tracing functionality by modifying and adding to automatically generated traces. The SignalFx Tracing library for .NET provides and registers an OpenTracing-compatible global tracer you can use.

OpenTracing versions 0.11.0+ are supported and the provided tracer offers a complete implementation of the OpenTracing API.

The auto-instrumentation provides a base you can build on by adding your own custom instrumentation. By using both instrumentation approaches, you'll be able to present a more detailed representation of the logic and functionality of your application, clients, and framework.

Add the OpenTracing dependency to your project:

<PackageReference Include="OpenTracing" Version="0.12.0" />

Obtain the OpenTracing.Util.GlobalTracer instance and create spans that automatically become child spans of any existing spans in the same context:

using OpenTracing;
using OpenTracing.Util;

namespace MyProject
{
    public class MyClass
    {
        public static async void MyMethod()
        {
            // Obtain the automatically registered OpenTracing.Util.GlobalTracer instance
            var tracer = GlobalTracer.Instance;

            // Create an active span that will be automatically parented by any existing span in this context
            using (IScope scope = tracer.BuildSpan("MyTracedFunctionality").StartActive(finishSpanOnDispose: true))
            {
                var span = scope.Span;
                span.SetTag("MyImportantTag", "MyImportantValue");
                span.Log("My Important Log Statement");

                var ret = await MyAppFunctionality();

                span.SetTag("FunctionalityReturned", ret.ToString());
            }
        }
    }
}

When using manual instrumentation it is possible to set different service names under the same process. This is disabled by default but can be enabled by configuring the environment variable SIGNALFX_SERVICE_NAME_PER_SPAN_ENABLED to true and following the OpenTracing semantic conventions by setting the tag service.name to the desired service name.

For more examples and information on how to do manual instrumentation see:

https://github.com/signalfx/tracing-examples/tree/master/dotnet-manual-instrumentation
https://github.com/signalfx/tracing-examples/tree/master/signalfx-tracing/signalfx-dotnet-tracing

Troubleshooting

Check if you are not hitting one of the issues listed below.

IIS applications not instrumenting expected services

Set the environment variable SIGNALFX_TRACE_DOMAIN_NEUTRAL_INSTRUMENTATION to true - without it the CLR profiler can't instrument many libraries/frameworks under IIS.

Linux instrumentation not working

The proper binary needs to be selected when deploying to Linux, eg.: the default Microsoft .NET images are based on Debian and should use the deb package, see the Linux setup section.

If you are not sure what is the Linux distribution being used try the following commands:

$ lsb_release -a
$ cat /etc/*release
$ cat /etc/issue*
$ cat /proc/version

High CPU usage

The default installation of auto-instrumentation enables tracing all .NET processes on the box. In the typical scenarios (dedicated VMs or containers), this is not a problem. Use the environment variables SIGNALFX_PROFILER_EXCLUDE_PROCESSES and SIGNALFX_PROFILER_PROCESSES to include/exclude applications from the tracing auto-instrumentation. These are ";" delimited lists that control the inclusion/exclusion of processes.

Custom instrumentation not being captured

If the code accessing GlobalTracer.Instance executes before any auto-instrumentation is injected into the process the call to GlobalTracer.Instance will return the OpenTracing No-Operation tracer. In this case it is necessary to force the injection of the SignalFx tracer by running a method like the one below before accessing GlobalTracer.Instance.

        static void InitTracer()
        {
            try
            {
                Assembly tracingAssembly = Assembly.Load(new AssemblyName("SignalFx.Tracing, Culture=neutral, PublicKeyToken=def86d061d0d2eeb"));
                Type tracerType = tracingAssembly.GetType("SignalFx.Tracing.Tracer");

                PropertyInfo tracerInstanceProperty = tracerType.GetProperty("Instance", BindingFlags.Public | BindingFlags.Static);
                object tracerInstance = tracerInstanceProperty.GetValue(null);
            }
            catch (Exception ex)
            {
                // TODO: Replace Console.WriteLine with proper log of the application.
                Console.WriteLine("Unable to load SignalFx.Tracing.Tracer library. Exception: {0}", ex);
            }
        }

Investigating other issues

If none of the suggestions above solves your issue, detailed logs are necessary. Follow the steps below to get the detailed logs from SignalFx Tracing for .NET:

Set the environment variable SIGNALFX_TRACE_DEBUG to true before the instrumented process starts. By default, the library writes the log files under the below predefined locations. If needed, change the default location by updating the environment variable SIGNALFX_TRACE_LOG_PATH to an appropriate path. On Linux, the default log location is /var/log/signalfx/dotnet/ On Windows, the default log location is %ProgramData%\SignalFx .NET Tracing\logs\ Compress the whole folder to capture the multiple log files and send the compressed folder to us. After obtaining the logs, remember to remove the environment variable SIGNALFX_TRACE_DEBUG to avoid unnecessary overhead.

Contributing

See docs/README.md and docs/CONTRIBUTING.md.

About

The SignalFx-Tracing Library for .NET is a fork of the .NET Tracer for Datadog APM that has been modified to provide Zipkin v2 JSON formatting, a complete OpenTracing API implementation, B3 propagation, and properly annotated trace data for handling by SignalFx Microservices APM.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

SignalFx Tracing Library for .NET

Supported libraries and frameworks

Configure the SignalFx Tracing Library for .NET

Configuration values

Ways to configure

Environment variables

Web.config and app.config

Json configuration file

Setup

Linux

Azure Function Custom Linux Docker Image

Windows

Enable instrumentation at different levels

Configure custom instrumentation

Troubleshooting

IIS applications not instrumenting expected services

Linux instrumentation not working

High CPU usage

Custom instrumentation not being captured

Investigating other issues

Contributing

About

Files

README.md

Latest commit

History

README.md

File metadata and controls

SignalFx Tracing Library for .NET

Supported libraries and frameworks

Configure the SignalFx Tracing Library for .NET

Configuration values

Ways to configure

Environment variables

Web.config and app.config

Json configuration file

Setup

Linux

Azure Function Custom Linux Docker Image

Windows

Enable instrumentation at different levels

Configure custom instrumentation

Troubleshooting

IIS applications not instrumenting expected services

Linux instrumentation not working

High CPU usage

Custom instrumentation not being captured

Investigating other issues

Contributing

About