Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type when writting to NLog

[fsimonovski]

NEST/Elasticsearch.Net version: 7.13.2

Elasticsearch version: Elastic cluster is v7.10

.NET runtime version: 8.0

Operating system version: Windows

Description of the problem including expected versus actual behavior:
When using NLog together with elasticsearch, I am getting Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type errors. One thing to note is that they are not immediate, one or 2 calls will go through and after that, it starts failing.

Steps to reproduce:
1.
2.
3.

Expected behavior
Being able to log on elasticsearch, this was not an issue on net6.0 framework with the same config file.

Provide ConnectionSettings (if relevant):

<target xsi:type="ElasticSearch" index="${configsetting:item=Nlog.Log_Index}-${date:format=yyyy.MM.dd}" uri="${configsetting:item=Nlog.Log_Url}" layout ="API:SpecificName |${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}"documentType="logevent"includeAllProperties="true">

This is the nlog.config file, I am reading configsetting items from appsettings.json, which is as follows:
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"NLog": {
"Log_Url": "{someurl}",
"Log_MinLevel": "Info",
"Log_Index": "{someindex}"
},

"AllowedHosts": "*"
}

These are installed package versions:

Provide DebugInformation (if relevant):
2024-04-16 17:32:46.3235 Error ElasticSearch: Error while sending log messages Exception: Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type: Microsoft.AspNetCore.Routing.RouteEndpoint.
The type defines the following properties:
'Order' of type System.Int32
'RoutePattern' of type Microsoft.AspNetCore.Routing.Patterns.RoutePattern
'DisplayName' of type System.String
'Metadata' of type Microsoft.AspNetCore.Http.EndpointMetadataCollection
'RequestDelegate' of type Microsoft.AspNetCore.Http.RequestDelegate
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: Microsoft.AspNetCore.Routing.RouteEndpoint.
The type defines the following properties:
'Order' of type System.Int32
'RoutePattern' of type Microsoft.AspNetCore.Routing.Patterns.RoutePattern
'DisplayName' of type System.String
'Metadata' of type Microsoft.AspNetCore.Http.EndpointMetadataCollection
'RequestDelegate' of type Microsoft.AspNetCore.Http.RequestDelegate
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: Microsoft.AspNetCore.Http.EndpointMetadataCollection.
The type defines the following properties:
'Item' of type System.Object
'Count' of type System.Int32
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: System.Runtime.CompilerServices.NullableContextAttribute.
The type defines the following properties:
'TypeId' of type System.Object
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: System.RuntimeType.
The type defines the following properties:
'IsCollectible' of type System.Boolean
'DeclaringMethod' of type System.Reflection.MethodBase
'FullName' of type System.String
'AssemblyQualifiedName' of type System.String
'Namespace' of type System.String
'GUID' of type System.Guid
'IsEnum' of type System.Boolean
'IsConstructedGenericType' of type System.Boolean
'IsGenericType' of type System.Boolean
'IsGenericTypeDefinition' of type System.Boolean
'GenericParameterAttributes' of type System.Reflection.GenericParameterAttributes
'IsSZArray' of type System.Boolean
'GenericParameterPosition' of type System.Int32
'ContainsGenericParameters' of type System.Boolean
'StructLayoutAttribute' of type System.Runtime.InteropServices.StructLayoutAttribute
'IsFunctionPointer' of type System.Boolean
'IsUnmanagedFunctionPointer' of type System.Boolean
'Name' of type System.String
'DeclaringType' of type System.Type
'Assembly' of type System.Reflection.Assembly
'BaseType' of type System.Type
'IsByRefLike' of type System.Boolean
'IsGenericParameter' of type System.Boolean
'IsTypeDefinition' of type System.Boolean
'IsSecurityCritical' of type System.Boolean
'IsSecuritySafeCritical' of type System.Boolean
'IsSecurityTransparent' of type System.Boolean
'MemberType' of type System.Reflection.MemberTypes
'MetadataToken' of type System.Int32
'Module' of type System.Reflection.Module
'ReflectedType' of type System.Type
'TypeHandle' of type System.RuntimeTypeHandle
'UnderlyingSystemType' of type System.Type
'GenericTypeParameters' of type System.Type[]
'DeclaredConstructors' of type System.Collections.Generic.IEnumerable1[System.Reflection.ConstructorInfo] 'DeclaredEvents' of type System.Collections.Generic.IEnumerable1[System.Reflection.EventInfo]
'DeclaredFields' of type System.Collections.Generic.IEnumerable1[System.Reflection.FieldInfo] 'DeclaredMembers' of type System.Collections.Generic.IEnumerable1[System.Reflection.MemberInfo]
'DeclaredMethods' of type System.Collections.Generic.IEnumerable1[System.Reflection.MethodInfo] 'DeclaredNestedTypes' of type System.Collections.Generic.IEnumerable1[System.Reflection.TypeInfo]
'DeclaredProperties' of type System.Collections.Generic.IEnumerable1[System.Reflection.PropertyInfo] 'ImplementedInterfaces' of type System.Collections.Generic.IEnumerable1[System.Type]
'IsInterface' of type System.Boolean
'IsNested' of type System.Boolean
'IsArray' of type System.Boolean
'IsByRef' of type System.Boolean
'IsPointer' of type System.Boolean
'IsGenericTypeParameter' of type System.Boolean
'IsGenericMethodParameter' of type System.Boolean
'IsVariableBoundArray' of type System.Boolean
'HasElementType' of type System.Boolean
'GenericTypeArguments' of type System.Type[]
'Attributes' of type System.Reflection.TypeAttributes
'IsAbstract' of type System.Boolean
'IsImport' of type System.Boolean
'IsSealed' of type System.Boolean
'IsSpecialName' of type System.Boolean
'IsClass' of type System.Boolean
'IsNestedAssembly' of type System.Boolean
'IsNestedFamANDAssem' of type System.Boolean
'IsNestedFamily' of type System.Boolean
'IsNestedFamORAssem' of type System.Boolean
'IsNestedPrivate' of type System.Boolean
'IsNestedPublic' of type System.Boolean
'IsNotPublic' of type System.Boolean
'IsPublic' of type System.Boolean
'IsAutoLayout' of type System.Boolean
'IsExplicitLayout' of type System.Boolean
'IsLayoutSequential' of type System.Boolean
'IsAnsiClass' of type System.Boolean
'IsAutoClass' of type System.Boolean
'IsUnicodeClass' of type System.Boolean
'IsCOMObject' of type System.Boolean
'IsContextful' of type System.Boolean
'IsMarshalByRef' of type System.Boolean
'IsPrimitive' of type System.Boolean
'IsValueType' of type System.Boolean
'IsSignatureType' of type System.Boolean
'TypeInitializer' of type System.Reflection.ConstructorInfo
'IsSerializable' of type System.Boolean
'IsVisible' of type System.Boolean
'CustomAttributes' of type System.Collections.Generic.IEnumerable1[System.Reflection.CustomAttributeData] ---> System.InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true. at System.RuntimeType.get_DeclaringMethod() at Serialize(Byte[][], Object[], JsonWriter&, RuntimeType, IJsonFormatterResolver) at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Serialize(Byte[][], Object[], JsonWriter&, NullableContextAttribute, IJsonFormatterResolver) at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase4.Serialize(JsonWriter& writer, TCollection value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) --- End of inner exception stack trace --- at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
at Serialize(Byte[][], Object[], JsonWriter&, RouteEndpoint, IJsonFormatterResolver)
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) --- End of inner exception stack trace --- at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.DictionaryFormatterBase5.Serialize(JsonWriter& writer, TDictionary value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.LowLevelRequestResponseSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) at Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) at Elasticsearch.Net.PostData1.Write(Stream writableStream, IConnectionConfigurationValues settings)
at Elasticsearch.Net.HttpConnection.SetContent(HttpRequestMessage message, RequestData requestData)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.Bulk[TResponse](PostData body, BulkRequestParameters requestParameters)
at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)
2024-04-16 17:32:46.4161 Trace AsyncWrapper_ElasticSearchTarget(Name=elasticsearch): Writing 1 events (Timer)
2024-04-16 17:36:38.0247 Debug Message Template Auto Format enabled
2024-04-16 17:36:38.0579 Info Loading assembly name: NLog.Targets.ElasticSearch
2024-04-16 17:36:38.0782 Info NLog.Targets.ElasticSearch, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3eb62cd298662c6a. File version: 1.0.0.0. Product version: 1.0.0.

[flobernd]

It seems like you are using a 3rd party package NLog.Targets.ElasticSearch. If you can give me a code which is able to reproduce the problem without using that package, I'm happy to investigate further.

Otherwise, please first ask the author of NLog.Targets.ElasticSearch to check the bug 🙂

[fsimonovski]

Hi,

This is a duplicate of this issue on Nlog.targets markmcdowell/NLog.Targets.ElasticSearch#182

For this issue, when setting NLog to log to file target only, I have no problems. Only when trying to log onto elastic cluster v7.10.2, on NET8.0.
Here is a link to a sample repo, which has the same problem, when we take a look at the internallog file which is generated after running the webapi project.
It only needs a running elastic cluster to see the problem
https://github.com/fsimonovski/NLog-ElasticSearch-Net8
Thanks for the help.

[flobernd]

Hi @fsimonovski,

it seems like a workaround was already suggested to you here.

NEST 7.13.2 is very old and I unfortunately can't spend any time digging into that. You could try to upgrade to NEST 7.17.x or even better to the new 8.x client. Please as well note that NEST is deprecated and will reach end of life at the end of the year.

The problem seems to be the changed type Microsoft.AspNetCore.Routing.RouteEndpoint which is not related to NEST. Probably this type got changed as well in the NET8 packages of AspNetCore and causes the issues. My advice is to never serialize any external complex types over which you don't have control. Creating a custom DTO object and just assigning the primitive properties to it in which you are interested to log, is most of the time a way more robust solution.