Valid UTF-8 character in feature description-field crashes Unleash Client

[mamort]

Describe the bug
Valid UTF-8 characters in feature description (and likely other fields) crashes Unleash Client

To Reproduce
Create a features.json file (with UTF-8 encoding) with the valid UTF-8 character "ø" in the description-field:

{
  "version": 2,
  "features": [
    {
      "name": "shouldUseFeature",
      "type": "release",
      "description": "ø",
      "enabled": true,
      "stale": false,
      "impressionData": false,
      "project": "Test",
      "strategies": [],
      "variants": []
    }
  ]
}

Unleash test with features.json file (using Moq):

[Fact]
public async Task CreateUnleashClient_WithFeatureWithUtf8Character_ShouldSuccessfullyCreateUnleashClient()
{
	var responseMessage = await CreateMockResponseMessageAsync("features.json");
	var httpClient = CreateHttpClient(responseMessage);
	
	var settings = new UnleashSettings();
	var mockClientFactory = new Mock<IHttpClientFactory>();
	mockClientFactory.Setup(f => f.Create(It.IsAny<Uri>())).Returns(() => httpClient.Object);
	settings.HttpClientFactory = mockClientFactory.Object;
	
	var unleash = new UnleashClientFactory();
	await unleash.CreateClientAsync(settings, synchronousInitialization: true);
}

Helper functions:

private static Mock<HttpClient> CreateHttpClient(HttpResponseMessage responseMessage)
{
	var httpMessageHandler = new Mock<HttpMessageHandler>();
	httpMessageHandler.Protected()
		.Setup<Task<HttpResponseMessage>>("SendAsync",
			ItExpr.IsAny<HttpRequestMessage>(),
			ItExpr.IsAny<CancellationToken>())
		.ReturnsAsync(responseMessage);

	var httpClient = new Mock<HttpClient>(httpMessageHandler.Object)
	{
		CallBase = true,
		Object =
		{
			BaseAddress = new Uri("https://uri.for.test")
		}
	};

	return httpClient;
}
private static async Task<HttpResponseMessage> CreateMockResponseMessageAsync(string resourceName)
{
	await using var jsonStream = Assembly.GetExecutingAssembly()
		.GetManifestResourceStream(
			$"Responses.{resourceName}");

	var response = await new StreamReader(jsonStream!).ReadToEndAsync();
	var stringContent = new StringContent(response);

	var responseMessage = new HttpResponseMessage
	{
		StatusCode = HttpStatusCode.OK,
		Content = stringContent
	};

	responseMessage.Headers.ETag = new EntityTagHeaderValue("\"33a64df551425fcc55e4d42a148795d9f25f89d4\"");

	return responseMessage;
}

When running this test we get:

Unleash.UnleashException: Exception while updating toggle collection

Unleash.UnleashException
Exception while updating toggle collection
   at Unleash.Scheduling.FetchFeatureTogglesTask.ExecuteAsync(CancellationToken cancellationToken)
   at Unleash.ClientFactory.UnleashClientFactory.CreateClientAsync(UnleashSettings settings, Boolean synchronousInitialization, IStrategy[] strategies)
(...)

Yggdrasil.YggdrasilEngineException
Error: Detected a non UTF-8 string in the input, this is a serious issue and you should report this as a bug.
   at Yggdrasil.FFIReader.CheckResponse(IntPtr ptr)
   at Yggdrasil.YggdrasilEngine.TakeState(String json)
   at Unleash.Scheduling.FetchFeatureTogglesTask.ExecuteAsync(CancellationToken cancellationToken)

Removing the UTF-8 character "ø" makes the test run successfully. However, the test also runs successfully when replacing the character "ø" with the unicode escape sequence \u00f8 which represents the same character. I am unable to figure out why it behaves this way, but it might be a clue to why it fails.

Also worth mentioning that Unleash Client started to fail for us when we upgraded from version 4.1.15 to 5.0.3 because we had a real feature with UTF-8 characters like the one in the above test.

Expected behavior
Expected Unleash client to be created/work without any problems.

Desktop (please complete the following information):

• OS: Windows
• Unleash version: 5.0.3

[daveleek]

Thank you @mamort for bringing this to our attention. I was able to reproduce this on windows quite simply by adding a "description": "ø", to test data of an existing test, and even by adding an ø into the feature name. But oddly enough not on MacOS. We're looking into it and will get back to you!

[sighphyre]

Yup. Works on Ubuntu as well. Also confirmed it breaks on Windows