$metadata endpoint fails with "An asynchronous operation is already in progress."

[vcaraulean]

Assemblies affected

<PackageReference Include="Microsoft.AspNetCore.OData" Version="9.0.0" />

Describe the bug

Calling $metadata endpoint fails with System.InvalidOperationException An asynchronous operation is already in progress error. Some data is returned (streamed), but the response is incomplete, the content is not consistent, varies from call to call.

Stack traces are sometimes different. Some specimens:

System.InvalidOperationException
An asynchronous operation is already in progress.
   at System.Xml.XmlAsyncCheckWriter.CheckAsync()
   at System.Xml.XmlAsyncCheckWriter.Flush()
   at Microsoft.OData.ODataMetadataOutputContext.Dispose(Boolean disposing)
   at Microsoft.OData.ODataOutputContext.Dispose()
   at Microsoft.OData.ODataMessageWriter.Dispose(Boolean disposing)
   at Microsoft.OData.ODataMessageWriter.Dispose()
   at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest request, IHeaderDictionary requestHeaders, IODataSerializerProvider serializerProvider)

Another is slightly different, which is not surprising as there is probably a race condition somewhere...

System.InvalidOperationException
An asynchronous operation is already in progress.
   at System.Xml.XmlAsyncCheckWriter.CheckAsync()
   at System.Xml.XmlAsyncCheckWriter.WriteEndElementAsync()
   at Microsoft.OData.Edm.Csdl.Serialization.EdmModelCsdlSchemaXmlWriter.WriteEndElementAsync()
   at Microsoft.OData.Edm.Csdl.Serialization.EdmModelCsdlSerializationVisitor.VisitEdmSchemaAsync(EdmSchema element, IEnumerable`1 mappings)
   at Microsoft.OData.Edm.Csdl.CsdlXmlWriter.WriteSchemasAsync()
   at Microsoft.OData.Edm.Csdl.CsdlXmlWriter.WriteODataCsdlAsync()
   at Microsoft.OData.Edm.Csdl.CsdlXmlWriter.WriteCsdlAsync()
   at Microsoft.OData.Edm.Csdl.CsdlWriter.TryWriteCsdlAsync(IEdmModel model, XmlWriter writer, CsdlTarget target, CsdlXmlWriterSettings writerSettings)
   at Microsoft.OData.ODataMetadataOutputContext.WriteMetadataDocumentImplementationAsync()
   at Microsoft.OData.ODataMetadataOutputContext.WriteMetadataDocumentAsync()
   at Microsoft.OData.ODataMessageWriter.WriteToOutputAsync(ODataPayloadKind payloadKind, Func`2 writeAsyncAction)
   at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataMetadataSerializer.WriteObjectAsync(Object graph, Type type, ODataMessageWriter messageWriter, ODataSerializerContext writeContext)
   at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest request, IHeaderDictionary requestHeaders, IODataSerializerProvider serializerProvider)

Reproduce steps

Happens on our large-ish solution. I was unable to reproduce the issue with a more reduced reproduction solution.

Data Model

We have 2 large EDM models:

• first has over 150 entities. $metadata returns the schema without issues
• second is ~80. $metadata fails with the error above

I tried to try to understand why only second fails, but been unable to do so. Cannot pinpoint the failure to a specific entity or set.

Expected behavior

$metadata endpoint allows retrieval of OData metadata.

Additional context

The failure has another indirection layer - environment:

• running the solution/project locally (from Rider): $metadata succeeds
• running from integration test (using TestServer): $metadata fails
• running in a deployed environment (corporate k8s cluster): $metadata fails

And success/failures are consistent for each environment.

[julealgon]

• second is ~80. $metadata fails with the error above

Do you think you could create a repro project out of that scenario? I think that would help the team immensely in debugging this.