Trigger trace based on collection rules

[tbajwa2017]

I am trying to configure dotnet monitor to run trace based on collection rules.

According to the docs, I have setup my dotnet container as follows using env basically putting the dotnet monitor in listen mode:

env:

• name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Key
  value: "aspnetapp"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Value
  value: "dotnet"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__MatchType
  value: "Exact"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Type
  value: "EventCounter"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__ProviderName
  value: "System.Runtime"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__CounterName
  value: "cpu-usage"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__GreaterThan
  value: "5"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__SlidingWindowDuration
  value: "00:00:01"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Type
  value: "CollectTrace"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Profile
  value: "Cpu"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Duration
  value: "00:01:00"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Egress
  value: "TmpDir"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCount
  value: "2"
• name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCountSlidingWindowDuration
  value: "1:00:00"
• name: DotnetMonitor_DiagnosticPort__ConnectionMode
  value: "Listen"
• name: DotnetMonitor_DiagnosticPort__EndpointName
  value: "/diag/port.sock"**

and I have configured my dotnet app with the following inside the container image:

export DOTNET_DiagnosticPorts="/diag/port.sock,suspend"

I setup the CPU threshold very low to trigger the trace, but I get the following in the logs for the app container:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.

DOTNET_DiagnosticPorts="/tmp/port.sock,suspend"

DOTNET_DefaultDiagnosticPortSuspend=0

Just wondering, what am I doing wrong here?

Thanks.

[jander-msft]

and I have configured my dotnet app with the following inside the container image:

export DOTNET_DiagnosticPorts="/diag/port.sock,suspend"

You should probably be using ENV to add environment variables to your container image, otherwise I think they are only available at image build time. However, in general, I would not recommend setting this environment variable as part of your container image; it should be set by your deployment mechanism as an environment variable on the container.

I get the following in the logs for the app container:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.

DOTNET_DiagnosticPorts="/tmp/port.sock,suspend"

DOTNET_DefaultDiagnosticPortSuspend=0

This tells me that some other aspect of your deployment is setting the DOTNET_DiagnosticPorts environment variable on the container, which is running contrary to how you've configured dotnet monitor to establish its server socket. You'll need to figure out what is setting the value to /tmp/port.sock,suspend and correct it.

[tbajwa2017]

Thanks. I will check the deployment and see what is setting DOTNET_DiagnosticPorts environment variable on the container and will use ENV rather than export.

[tbajwa2017]

I made the necessary changes and my deployment manifest looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
name: dotnet-hello-world
spec:
replicas: 1
selector:
matchLabels:
app: dotnet-hello-world
template:
metadata:
labels:
app: dotnet-hello-world
spec:
volumes:
- name: diagnostics
emptyDir: {}
containers:
- name: server
image: mcr.microsoft.com/dotnet/samples:aspnetapp
ports:
- containerPort: 80
env:
- name: DOTNET_DiagnosticPorts
value: "/diag/port.sock,suspend"
volumeMounts:
- mountPath: /tmp
name: diagnostics
- name: sidecar
image: mcr.microsoft.com/dotnet/monitor:6
env:
- name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Key
value: "ProcessName"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Value
value: "aspnetapp"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__MatchType
value: "Contains"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Type
value: "EventCounter"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__ProviderName
value: "System.Runtime"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__CounterName
value: "cpu-usage"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__GreaterThan
value: "0"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__SlidingWindowDuration
value: "00:00:1"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Type
value: "CollectTrace"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Profile
value: "Cpu"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Duration
value: "00:01:00"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Egress
value: "TmpDir"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCount
value: "2"
- name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCountSlidingWindowDuration
value: "1:00:00"
- name: DotnetMonitor_DiagnosticPort__ConnectionMode
value: "Listen"
- name: DotnetMonitor_DiagnosticPort__EndpointName
value: "/diag/port.sock"
- name: DotnetMonitor_DiagnosticPort__MaxConnections
value: "10"
ports:
- containerPort: 52323
args: ["--no-auth", "--urls", "http://*:52323"]
volumeMounts:
- name: diagnostics
mountPath: /tmp

Now when I deploy it in the kubernetes cluster, I get the following in the sample dotnet app log:

The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts=""
DOTNET_DefaultDiagnosticPortSuspend=0

and I get the following errors in the dotnet monitor container logs:

at Microsoft.Diagnostics.NETCore.Client.IpcServerTransport.Create(String address, Int32 maxConnections, Boolean enableTcpIpProtocol, IIpcServerTransportCallbackInternal transportCallback)
at Microsoft.Diagnostics.NETCore.Client.ReversedDiagnosticsServer.ListenAsync(Int32 maxConnections, CancellationToken token)
at Microsoft.Diagnostics.NETCore.Client.ReversedDiagnosticsServer.DisposeAsync()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at Microsoft.Diagnostics.NETCore.Client.ReversedDiagnosticsServer.Start(Int32 maxConnections)
at Microsoft.Diagnostics.Tools.Monitor.ServerEndpointInfoSource.ExecuteAsync(CancellationToken stoppingToken)
at Microsoft.Diagnostics.Tools.Monitor.ServerEndpointInfoSource.ExecuteAsync(CancellationToken stoppingToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Diagnostics.Tools.Monitor.Commands.CollectCommandHandler.Invoke(CancellationToken token, String[] urls, String[] metricUrls, Boolean metrics, String diagnosticPort, Boolean noAuth, Boolean tempApiKey, Boolean noHttpEgress, FileInfo configurationFilePath)
at Microsoft.Diagnostics.Tools.Monitor.Commands.CollectCommandHandler.Invoke(CancellationToken token, String[] urls, String[] metricUrls, Boolean metrics, String diagnosticPort, Boolean noAuth, Boolean tempApiKey, Boolean noHttpEgress, FileInfo configurationFilePath)
at Microsoft.Diagnostics.Tools.Monitor.Program.<>c.<b__1_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass21_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass18_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c.<b__17_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c.<b__4_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass7_0.<b__0>d.MoveNext()

Looks like the environment variable DOTNET_DiagnosticPorts is not getting set.

[jander-msft]

You need to declare an emptydir volume and mount in both containers at the /diag directory. Please read Kubernetes example for how to do this.

[tbajwa2017]

Thanks!. I will do that and will deploy again.

[tbajwa2017]

I deployed. This time I am not getting any errors in app container. In the collection rules I setup very small threshold of CPU usage greater than 0. Here are my collection rules:

spec:
  restartPolicy: Always
  containers:
  - name: app
    image: mcr.microsoft.com/dotnet/samples:aspnetapp
    imagePullPolicy: Always
    env:
    - name: ASPNETCORE_URLS
      value: http://+:80
    - name: DOTNET_DiagnosticPorts
      value: /diag/dotnet-monitor.sock,suspend
    volumeMounts:
    - mountPath: /diag
      name: diagvol
    resources:
      limits:
        cpu: 250m
        memory: 512Mi
  - name: monitor
    image: mcr.microsoft.com/dotnet/monitor:6
    # DO NOT use the --no-auth argument for deployments in production; this argument is used for demonstration
    # purposes only in this example. Please continue reading after this example for further details.
    args: [ "--no-auth" ]
    imagePullPolicy: Always
    env:
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Key
      value: "ProcessName"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__Value
      value: "aspnetapp"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Filters__0__MatchType
      value: "Contains"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Type
      value: "EventCounter"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__ProviderName
      value: "System.Runtime"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__CounterName
      value: "cpu-usage"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__GreaterThan
      value: "0"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Trigger__Settings__SlidingWindowDuration
      value: "00:00:01"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Type
      value: "CollectTrace"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Profile
      value: "Cpu"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Duration
      value: "00:01:00"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Egress
      value: "diagvol"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCount
      value: "2"
    - name: DotnetMonitor_CollectionRules__HighCpuRule__Limits__ActionCountSlidingWindowDuration
      value: "1:00:00"
    - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
      value: Listen
    - name: DOTNETMONITOR_DiagnosticPort__EndpointName
      value: /diag/dotnet-monitor.sock
    - name: DOTNETMONITOR_Storage__DumpTempFolder
      value: /diag/dumps
    # ALWAYS use the HTTPS form of the URL for deployments in production; the removal of HTTPS is done for
    # demonstration purposes only in this example. Please continue reading after this example for further details.
    - name: DOTNETMONITOR_Urls
      value: http://localhost:52323
    volumeMounts:
    - mountPath: /diag
      name: diagvol
    resources:
      requests:
        cpu: 50m
        memory: 32Mi
      limits:
        cpu: 250m
        memory: 256Mi
  volumes:
  - name: diagvol
    emptyDir: {}

~
~

[tbajwa2017]

I get the following error in the dotnet monitor logs:

{"Timestamp":"2022-12-18T23:12:02.0394299Z","EventId":30,"LogLevel":"Error","Category":"Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService","Message":"Collection rule \u0027HighCpuRule\u0027 failed.","Exception":"Microsoft.Extensions.Options.OptionsValidationException: Egress provider \u0027diagvol\u0027 does not exist. at Microsoft.Extensions.Options.OptionsFactory\u00601.Create(String name) at Microsoft.Extensions.Options.OptionsMonitor\u00601.\u003C\u003Ec__DisplayClass10_0.\u003CGet\u003Eb__0() at System.Lazy\u00601.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy\u00601.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) at System.Lazy\u00601.CreateValue() at System.Lazy\u00601.get_Value() at Microsoft.Extensions.Options.OptionsCache\u00601.GetOrAdd(String name, Func\u00601 createOptions) at Microsoft.Diagnostics.Tools.Monitor.DynamicNamedOptionsCache\u00601.GetOrAdd(String name, Func\u00601 createOptions) at Microsoft.Extensions.Options.OptionsMonitor\u00601.Get(String name) at Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleContainer.RunRuleAsync(String ruleName, TaskCompletionSource\u00601 startedSource, CancellationToken token)","State":{"Message":"Collection rule \u0027HighCpuRule\u0027 failed.","ruleName":"HighCpuRule","{OriginalFormat}":"Collection rule \u0027{ruleName}\u0027 failed."},"Scopes":[{"Message":"TargetProcessId:1 TargetRuntimeInstanceCookie:e9ceabb581e04f09a1b4c65b3dabeb1b CollectionRuleName:HighCpuRule","TargetProcessId":"1","TargetRuntimeInstanceCookie":"e9ceabb581e04f09a1b4c65b3dabeb1b","CollectionRuleName":"HighCpuRule"}]}

[tbajwa2017]

The error is related to Egress provider:

Message":"Collection rule \u0027HighCpuRule\u0027 failed.","Exception":"Microsoft.Extensions.Options.OptionsValidationException: Egress provider \u0027diagvol\u0027 does not exist

I have tried the Egress provide "TmpDir" and I get the same error that says the Egress provider does not exist.

[jander-msft]

The tool doesn't know what TmpDir is since you haven't configured an egress provider with that name: https://github.com/dotnet/dotnet-monitor/blob/main/documentation/egress.md

Here's the settings for the FileSystem one (as implied by your usage of TmpDir): https://github.com/dotnet/dotnet-monitor/blob/main/documentation/configuration.md#filesystem-egress-provider

[tbajwa2017]

I added the following in the env section for the dotnet monitor container for the Filesystem egress provider.

• name: DOTNETMONITOR_Storage__DumpTempFolder
  value: /diag/dumps

Now I am getting the error below which says "An egress provider must be specified if there is no default egress provider".
Can you please advise what egress provider should I declare in the env:?

{"Timestamp":"2022-12-19T21:41:33.2337393Z","EventId":30,"LogLevel":"Error","Category":"Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService","Message":"Collection rule \u0027HighCpuRule\u0027 failed.","Exception":"Microsoft.Extensions.Options.OptionsValidationException:
An egress provider must be specified if there is no default egress provider. at Microsoft.Extensions.Options.OptionsFactory\u00601.Create(String name) at Microsoft.Extensions.Options.OptionsMonitor\u00601.\u003C\u003Ec__DisplayClass10_0.\u003CGet\u003Eb__0() at System.Lazy\u00601.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy\u00601.ExecutionAndPublication(La

[jander-msft]

DOTNETMONITOR_Storage__DumpTempFolder is not an egress provider; that is the location where dumps are written by the target runtime and read by dotnet monitor, which can then send it back to you via the HTTP response or through an egress provider.

Please read the documentation that I linked to in the last reply. An example of how to configure an egress provider is:

- name: DotnetMonitor_Egress__FileSystem__TmpDir__directoryPath
  value: /tmp

This creates a file system egress provider named "TmpDir" that writes artifacts to the /tmp directory of the dotnet monitor container.

Then you can use egressProvider=TmpDir in query part of your HTTP calls or DotnetMonitor_CollectionRules__HighCpuRule__Actions__0__Settings__Egress=TmpDir in your collection rule.

[tbajwa2017]

Thanks Justin. I was able to get the trace file based on the collection rules after I configured the egress provider that points to the /tmp directory. Have a great Christmas and happy new year.