Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ObjectDisposedException on shutdown #293

Open
Tratcher opened this issue Jan 30, 2021 · 4 comments
Open

ObjectDisposedException on shutdown #293

Tratcher opened this issue Jan 30, 2021 · 4 comments
Labels
Milestone

Comments

@Tratcher
Copy link
Member

An app running in IIS with OWIN integration gets ObjectDisposedExceptions on shutdown rather than gracefully draining requests.

Exception Details: System.ObjectDisposedException, Cannot access a disposed object.
Object name: '[Redacted].AuthenticationHandler'.,
    at System.Net.Http.DelegatingHandler.CheckDisposed()
   at System.Net.Http.DelegatingHandler.SetOperationStarted()
   at System.Net.Http.DelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at [Redacted].AuthenticationHandler.<SendAsync>d__12.MoveNext()

Here's where dispose is called:

   at [Redacted].AuthenticationHandler.Dispose(Boolean disposing)
   at System.Net.Http.DelegatingHandler.Dispose(Boolean disposing)
   at System.Web.Http.HttpServer.Dispose(Boolean disposing)
   at System.Net.Http.HttpMessageInvoker.Dispose(Boolean disposing)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.OnAppDisposing()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException)
   at Microsoft.Owin.Host.SystemWeb.ShutdownDetector.Cancel()
   at Microsoft.Owin.Host.SystemWeb.ShutdownDetector.Stop(Boolean immediate)
   at System.Web.Hosting.HostingEnvironment.StopRegisteredObjects(Boolean immediate)
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()'

The root event here is HostingEnvironment.StopListening. "Occurs when the worker process or application pool associated with this host has stopped listening for new requests and will eventually shut down." The app is shutting down, but there may still be requests in flight that should be completed if possible.

Microsoft.Owin wired into the StopListening event so that SignalR could gracefully shut down long polling and WebSocket requests. WebAPI however wired up it's pipeline disposal to this event. This causes the pipeline to be disposed while request are in flight.

AppDisposing = builder.GetOnAppDisposingProperty()

IDictionary<string, object> properties = builder.Properties;
if (properties == null)
{
return CancellationToken.None;
}
object value;
if (!properties.TryGetValue("host.OnAppDisposing", out value))

https://github.com/aspnet/AspNetWebStack/blame/ba26cfbfbf958d548e4c0a96e853250f13450dc6/src/System.Web.Http.Owin/HttpMessageHandlerAdapter.cs#L87

There is no alternate event in IIS that fires when requests have drained.

Workaround: Remove the CancellationToken from the IAppBuilder so WebAPI cannot register with it.

appBuilder.Properties.Remove("host.OnAppDisposing");

Proposed fix: WebApi should not register for disposal with this event.
https://github.com/aspnet/AspNetWebStack/blame/ba26cfbfbf958d548e4c0a96e853250f13450dc6/src/System.Web.Http.Owin/HttpMessageHandlerAdapter.cs#L87
(When should it be disposed then?)

@mkArtakMSFT mkArtakMSFT added this to the Backlog milestone Nov 10, 2021
@mkArtakMSFT
Copy link
Member

@Tratcher wouldn't the propose fix be a breaking change? What will be the downsides of it?

@Tratcher
Copy link
Member Author

The handlers are disposed too soon, causing unnecessary failures. We'd have to figure out a better signal to dispose them before answering questions about breaking changes.

@Bykiev
Copy link

Bykiev commented Apr 19, 2022

Is any chance it will be ever fixed?

@k290
Copy link

k290 commented Sep 3, 2024

Hit this bug in our app in prod too during a deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants