Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Newtonsoft.Json package and migrate to System.Text.Json #2125

Open
wants to merge 26 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d8c606f
feat: rm newtonsoft
EngRajabi Jul 15, 2024
635439b
feat: add system text json config
EngRajabi Jul 19, 2024
4d14ebb
fix: AcceptanceTest ConsulServiceDiscovery
MohammadAminPourmoradian Jul 19, 2024
646b81c
fix: fix indent
EngRajabi Jul 19, 2024
fe5b2fd
feat: change un use
EngRajabi Jul 19, 2024
d5c23c0
feat: add json benchmark
EngRajabi Jul 19, 2024
c7a641c
feat: add json serialize result json
EngRajabi Jul 19, 2024
61247cc
fix
EngRajabi Jul 19, 2024
0048df8
update doc
EngRajabi Jul 19, 2024
c30672f
Revert file MessageInvokerPoolTests.cs
raman-m Jul 22, 2024
6215382
Revert file ocelot.json
raman-m Jul 22, 2024
dfbc87f
Revert file Steps.cs
raman-m Jul 22, 2024
12f1b21
Apply changes in file Steps.cs
raman-m Jul 22, 2024
8667047
Revert file MessageInvokerPool.cs
raman-m Jul 22, 2024
ac74d64
Code review by @raman-m
raman-m Jul 22, 2024
c6897c9
fix: rename and format JsonSerializerOptions class
MohammadAminPourmoradian Jul 28, 2024
147271c
fix: remove unused JsonDocument object
MohammadAminPourmoradian Jul 28, 2024
b65a6d6
add
EngRajabi Oct 3, 2024
0fda6af
fix: rm JsonPath.Net
EngRajabi Oct 3, 2024
4863b2e
fix: add more test
EngRajabi Oct 4, 2024
dea5e37
Fix errors and warnings
raman-m Nov 2, 2024
885ee5b
Fix Ocelot.Benchmarks build
raman-m Nov 2, 2024
bd2da05
fix: fix something
EngRajabi Dec 18, 2024
6b67250
Fix build errors, warnings, messages after rebasing
raman-m Dec 20, 2024
f59ca5e
fix: rm newstonsoft
EngRajabi Dec 21, 2024
2a213d2
fix: rename JsonSerializerOptionsFactory to OcelotSerializerOptions
EngRajabi Dec 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 8 additions & 69 deletions docs/features/dependencyinjection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,14 @@ Current `implementation <https://github.com/search?q=repo%3AThreeMammals%2FOcelo
.AddApplicationPart(assembly)
.AddControllersAsServices()
.AddAuthorization()
.AddNewtonsoftJson();
.AddJsonOptions(op =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hesitant to rewrite the documentation until all the code has been verified and approved.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not believe we have the authority to change the default settings of the library. Providing recommendations in the documentation to the community should have reasonable goals!
I do not think all Ocelot solutions on the Internet should adhere to these JSON options.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change does not change the library's behavior, so changing this method does not cause any problems.

{
op.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.IgnoreCycles;
op.JsonSerializerOptions.NumberHandling = JsonNumberHandling.AllowReadingFromString;
op.JsonSerializerOptions.WriteIndented = false;
op.JsonSerializerOptions.PropertyNameCaseInsensitive = true;
op.JsonSerializerOptions.Encoder = JavaScriptEncoder.Create(UnicodeRanges.All);
});
}

The method cannot be overridden. It is not virtual, and there is no way to override current behavior by inheritance.
Expand All @@ -154,74 +161,6 @@ The next section shows you an example of designing custom Ocelot pipeline by cus

.. _di-custom-builder:

Custom Builder
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that this sample might contain outdated code, yet it's essential to convert it into something usable. The sample illustrates how to override the default services in the DI feature. We can demonstrate how to revert to the old NewtonSoft services using new package versions. However, the rewriting should be deferred until all C# code is approved. Ultimately, updating the documentation should be the final step in the PR process.

--------------

**Goal**: Replace ``Newtonsoft.Json`` services with ``System.Text.Json`` services.

Problem
^^^^^^^

The main `AddOcelot`_ method adds
`Newtonsoft JSON <https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.dependencyinjection.newtonsoftjsonmvccorebuilderextensions.addnewtonsoftjson>`_ services
by the ``AddNewtonsoftJson`` extension method in default builder (`AddDefaultAspNetServices`_ method).
The ``AddNewtonsoftJson`` method calling was introduced in old .NET and Ocelot releases which was necessary when Microsoft did not launch the ``System.Text.Json`` library,
but now it affects normal use, so we have an intention to solve the problem.

Modern `JSON services <https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.dependencyinjection.mvccoremvccorebuilderextensions.addjsonoptions>`_
out of `the box <https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.dependencyinjection.mvccoremvccorebuilderextensions>`_
will help to configure JSON settings by the ``JsonSerializerOptions`` property for JSON formatters during (de)serialization.

Solution
^^^^^^^^

We have the following methods in `ServiceCollectionExtensions`_ class:

.. code-block:: csharp

IOcelotBuilder AddOcelotUsingBuilder(this IServiceCollection services, Func<IMvcCoreBuilder, Assembly, IMvcCoreBuilder> customBuilder);
IOcelotBuilder AddOcelotUsingBuilder(this IServiceCollection services, IConfiguration configuration, Func<IMvcCoreBuilder, Assembly, IMvcCoreBuilder> customBuilder);

These methods with custom builder allow you to use your any desired JSON library for (de)serialization.
But we are going to create custom ``MvcCoreBuilder`` with support of JSON services, such as ``System.Text.Json``.
To do that we need to call ``AddJsonOptions`` extension of the ``MvcCoreMvcCoreBuilderExtensions`` class
(NuGet package: `Microsoft.AspNetCore.Mvc.Core <https://www.nuget.org/packages/Microsoft.AspNetCore.Mvc.Core/>`_) in **Startup.cs**:

.. code-block:: csharp

using Microsoft.Extensions.DependencyInjection;
using Ocelot.DependencyInjection;
using System.Reflection;

public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services
.AddLogging()
.AddMiddlewareAnalysis()
.AddWebEncoders()
// Add your custom builder
.AddOcelotUsingBuilder(MyCustomBuilder);
}

private static IMvcCoreBuilder MyCustomBuilder(IMvcCoreBuilder builder, Assembly assembly)
{
return builder
.AddApplicationPart(assembly)
.AddControllersAsServices()
.AddAuthorization()

// Replace AddNewtonsoftJson() by AddJsonOptions()
.AddJsonOptions(options =>
{
options.JsonSerializerOptions.WriteIndented = true; // use System.Text.Json
});
}
}

The sample code provides settings to render JSON as indented text rather than compressed plain JSON text without spaces.
This is just one common use case, and you can add additional services to the builder.

------------------------------------------------------------------

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
using Microsoft.Extensions.Options;
using Newtonsoft.Json;
using Ocelot.Cache;
using Ocelot.Configuration;
using Ocelot.Configuration.File;
using Ocelot.Configuration.Repository;
using Ocelot.Infrastructure;
using Ocelot.Logging;
using Ocelot.Provider.Consul.Interfaces;
using Ocelot.Responses;
using System.Text;
using System.Text.Json;

namespace Ocelot.Provider.Consul;

Expand Down Expand Up @@ -53,14 +54,14 @@ public async Task<Response<FileConfiguration>> Get()

var bytes = queryResult.Response.Value;
var json = Encoding.UTF8.GetString(bytes);
var consulConfig = JsonConvert.DeserializeObject<FileConfiguration>(json);
var consulConfig = JsonSerializer.Deserialize<FileConfiguration>(json, OcelotSerializerOptions.Web);

return new OkResponse<FileConfiguration>(consulConfig);
}

public async Task<Response> Set(FileConfiguration ocelotConfiguration)
{
var json = JsonConvert.SerializeObject(ocelotConfiguration, Formatting.Indented);
var json = JsonSerializer.Serialize(ocelotConfiguration, OcelotSerializerOptions.WebWriteIndented);
var bytes = Encoding.UTF8.GetBytes(json);
var kvPair = new KVPair(_configurationKey)
{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
using Microsoft.AspNetCore.Hosting;
using Newtonsoft.Json;
using Ocelot.Configuration.ChangeTracking;
using Ocelot.Configuration.File;
using Ocelot.DependencyInjection;
using Ocelot.Infrastructure;
using Ocelot.Responses;
using System.Text.Json;
using FileSys = System.IO.File;

namespace Ocelot.Configuration.Repository;
Expand Down Expand Up @@ -49,14 +50,14 @@ public Task<Response<FileConfiguration>> Get()
jsonConfiguration = FileSys.ReadAllText(_environmentFile.FullName);
}

var fileConfiguration = JsonConvert.DeserializeObject<FileConfiguration>(jsonConfiguration);
var fileConfiguration = JsonSerializer.Deserialize<FileConfiguration>(jsonConfiguration, OcelotSerializerOptions.Web);

return Task.FromResult<Response<FileConfiguration>>(new OkResponse<FileConfiguration>(fileConfiguration));
}

public Task<Response> Set(FileConfiguration fileConfiguration)
{
var jsonConfiguration = JsonConvert.SerializeObject(fileConfiguration, Formatting.Indented);
var jsonConfiguration = JsonSerializer.Serialize(fileConfiguration, OcelotSerializerOptions.WebWriteIndented);

lock (_lock)
{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
using Microsoft.Extensions.Hosting;
using Newtonsoft.Json;
using Ocelot.Configuration.Creator;
using Ocelot.Configuration.File;
using Ocelot.Infrastructure;
using Ocelot.Logging;
using System.Text.Json;

namespace Ocelot.Configuration.Repository;

Expand Down Expand Up @@ -68,7 +69,7 @@ private async Task Poll()

if (fileConfig.IsError)
{
_logger.LogWarning(() =>$"error geting file config, errors are {string.Join(',', fileConfig.Errors.Select(x => x.Message))}");
_logger.LogWarning(() => $"error geting file config, errors are {string.Join(',', fileConfig.Errors.Select(x => x.Message))}");
return;
}

Expand All @@ -95,7 +96,7 @@ private async Task Poll()
/// <returns>hash of the config.</returns>
private static string ToJson(FileConfiguration config)
{
var currentHash = JsonConvert.SerializeObject(config);
var currentHash = JsonSerializer.Serialize(config, OcelotSerializerOptions.Web);
return currentHash;
}

Expand Down
21 changes: 11 additions & 10 deletions src/Ocelot/DependencyInjection/ConfigurationBuilderExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Configuration.Memory;
using Newtonsoft.Json;
using Ocelot.Configuration.File;
using Ocelot.Infrastructure;
using System.Text;
using System.Text.Json;

namespace Ocelot.DependencyInjection;

Expand Down Expand Up @@ -91,8 +92,8 @@ public static IConfigurationBuilder AddOcelot(this IConfigurationBuilder builder
private static IConfigurationBuilder ApplyMergeOcelotJsonOption(IConfigurationBuilder builder, MergeOcelotJson mergeTo, string json,
string primaryConfigFile, bool? optional, bool? reloadOnChange)
{
return mergeTo == MergeOcelotJson.ToMemory ?
builder.AddJsonStream(new MemoryStream(Encoding.UTF8.GetBytes(json))) :
return mergeTo == MergeOcelotJson.ToMemory ?
builder.AddJsonStream(new MemoryStream(Encoding.UTF8.GetBytes(json))) :
AddOcelotJsonFile(builder, json, primaryConfigFile, optional, reloadOnChange);
}

Expand Down Expand Up @@ -133,8 +134,8 @@ private static string GetMergedOcelotJson(string folder, IWebHostEnvironment env
}

var lines = File.ReadAllText(file.FullName);
var config = JsonConvert.DeserializeObject<FileConfiguration>(lines);
if (file.Name.Equals(globalFileInfo.Name, StringComparison.OrdinalIgnoreCase) &&
var config = JsonSerializer.Deserialize<FileConfiguration>(lines, OcelotSerializerOptions.Web);
if (file.Name.Equals(globalFileInfo.Name, StringComparison.OrdinalIgnoreCase) &&
file.FullName.Equals(globalFileInfo.FullName, StringComparison.OrdinalIgnoreCase))
{
fileConfiguration.GlobalConfiguration = config.GlobalConfiguration;
Expand All @@ -144,8 +145,8 @@ private static string GetMergedOcelotJson(string folder, IWebHostEnvironment env
fileConfiguration.Routes.AddRange(config.Routes);
}

return JsonConvert.SerializeObject(fileConfiguration, Formatting.Indented);
}
return JsonSerializer.Serialize(fileConfiguration, OcelotSerializerOptions.WebWriteIndented);
}

/// <summary>
/// Adds Ocelot configuration by ready configuration object and writes JSON to the primary configuration file.<br/>
Expand All @@ -158,11 +159,11 @@ private static string GetMergedOcelotJson(string folder, IWebHostEnvironment env
/// <param name="optional">The 2nd argument of the AddJsonFile.</param>
/// <param name="reloadOnChange">The 3rd argument of the AddJsonFile.</param>
/// <returns>An <see cref="IConfigurationBuilder"/> object.</returns>
public static IConfigurationBuilder AddOcelot(this IConfigurationBuilder builder, FileConfiguration fileConfiguration,
public static IConfigurationBuilder AddOcelot(this IConfigurationBuilder builder, FileConfiguration fileConfiguration,
string primaryConfigFile = null, bool? optional = null, bool? reloadOnChange = null) // optional injections
{
var json = JsonConvert.SerializeObject(fileConfiguration, Formatting.Indented);
return AddOcelotJsonFile(builder, json, primaryConfigFile, optional, reloadOnChange);
var json = JsonSerializer.Serialize(fileConfiguration, OcelotSerializerOptions.WebWriteIndented);
return AddOcelotJsonFile(builder, json, primaryConfigFile, optional, reloadOnChange);
}

/// <summary>
Expand Down
20 changes: 14 additions & 6 deletions src/Ocelot/DependencyInjection/OcelotBuilder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@
using Ocelot.ServiceDiscovery.Providers;
using Ocelot.WebSockets;
using System.Reflection;
using System.Text.Encodings.Web;
using System.Text.Json.Serialization;
using System.Text.Unicode;

namespace Ocelot.DependencyInjection;

Expand Down Expand Up @@ -108,7 +111,7 @@ public OcelotBuilder(IServiceCollection services, IConfiguration configurationRo
Services.TryAddSingleton<IHttpHandlerOptionsCreator, HttpHandlerOptionsCreator>();
Services.TryAddSingleton<IDownstreamAddressesCreator, DownstreamAddressesCreator>();
Services.TryAddSingleton<IDelegatingHandlerHandlerFactory, DelegatingHandlerHandlerFactory>();

Services.TryAddSingleton<IOcelotConfigurationChangeTokenSource, OcelotConfigurationChangeTokenSource>();
Services.TryAddSingleton<IOptionsMonitor<IInternalConfiguration>, OcelotConfigurationMonitor>();

Expand Down Expand Up @@ -152,7 +155,6 @@ public OcelotBuilder(IServiceCollection services, IConfiguration configurationRo
/// <summary>
/// Adds default ASP.NET services which are the minimal part of the gateway core.
/// <para>
/// Finally the builder adds Newtonsoft.Json services via the <see cref="NewtonsoftJsonMvcCoreBuilderExtensions.AddNewtonsoftJson(IMvcCoreBuilder)"/> extension-method.<br/>
/// To remove these services, use custom builder in the <see cref="ServiceCollectionExtensions.AddOcelotUsingBuilder(IServiceCollection, Func{IMvcCoreBuilder, Assembly, IMvcCoreBuilder})"/> extension-method.
/// </para>
/// </summary>
Expand All @@ -163,11 +165,10 @@ public OcelotBuilder(IServiceCollection services, IConfiguration configurationRo
/// - <see cref="AnalysisServiceCollectionExtensions.AddMiddlewareAnalysis(IServiceCollection)"/><br/>
/// - <see cref="EncoderServiceCollectionExtensions.AddWebEncoders(IServiceCollection)"/>.
/// <para>
/// Warning! The following <see cref="IMvcCoreBuilder"/> extensions being called:<br/>
/// Warning! The following <see cref="IMvcCoreBuilder"/> extensions being called<br/>
/// - <see cref="MvcCoreMvcCoreBuilderExtensions.AddApplicationPart(IMvcCoreBuilder, Assembly)"/><br/>
/// - <see cref="MvcCoreMvcCoreBuilderExtensions.AddControllersAsServices(IMvcCoreBuilder)"/><br/>
/// - <see cref="MvcCoreMvcCoreBuilderExtensions.AddAuthorization(IMvcCoreBuilder)"/><br/>
/// - <see cref="NewtonsoftJsonMvcCoreBuilderExtensions.AddNewtonsoftJson(IMvcCoreBuilder)"/>, removable.
/// - <see cref="MvcCoreMvcCoreBuilderExtensions.AddAuthorization(IMvcCoreBuilder)"/>.
/// </para>
/// </remarks>
/// <param name="builder">The default builder being returned by <see cref="MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection)"/> extension-method.</param>
Expand All @@ -184,7 +185,14 @@ protected IMvcCoreBuilder AddDefaultAspNetServices(IMvcCoreBuilder builder, Asse
.AddApplicationPart(assembly)
.AddControllersAsServices()
.AddAuthorization()
.AddNewtonsoftJson();
.AddJsonOptions(op =>
{
op.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.IgnoreCycles;
op.JsonSerializerOptions.NumberHandling = JsonNumberHandling.AllowReadingFromString;
op.JsonSerializerOptions.WriteIndented = false;
op.JsonSerializerOptions.PropertyNameCaseInsensitive = true;
op.JsonSerializerOptions.Encoder = JavaScriptEncoder.Create(UnicodeRanges.All);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain the final goal of these changes to me please?
What problem do these settings solve?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This setting prevents characters from being escaped when creating JSON. It causes problems in Arabic, Persian, etc. languages.

});
}

public IOcelotBuilder AddSingletonDefinedAggregator<T>()
Expand Down
Loading