-
Notifications
You must be signed in to change notification settings - Fork 846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
schema_registry_encode
truncation of input json before encode
#2525
Comments
schema_registry_encode
missing keys causing errorschema_registry_encode
truncation of input json before encode
Hey @hendoxc, I had a quick look at this, but I can't come up with a way to reproduce it. Are you able to trigger it consistently and is it possible to come up with a setup that I can run myself? |
Hey @mihaitodor I've looked into this more, and the underlying issue was that my schema had a field that was not-nullable, but the message was null. I've run a local example and stepped through the benthos code, and down into the another example is if a avro schema has a I think the issue can be closed, however given that however again, this issue can be closed :) |
Thank you for digging into it @hendoxc! Indeed, goavro has some bugs / limitations (252 and 253 are the ones I'm aware of) and, since LinkedIn doesn't seem very motivated to invest in it further and maybe work towards a V2, I'm not sure if anyone is willing to pick up the slack, at least not when it comes to AVRO. One big issue is that serialising AVRO to JSON without preserving the message schema is lossy in some ways when it comes to logical types / unions and each implementation makes different tradeoffs... I did see https://github.com/hamba/avro pop up a while back, but I think it was missing certain features like marshalling logical types in a way that is compatible with whatever Kafka Connect is doing. It did get plenty of updates since then, so maybe now one can fully replicate the Kafka Connect behaviour using this alternative library, but I don't really have a use case for AVRO right now, so it's hard to justify diving into that can of worms... One way to do it would be to add an advanced flag to |
I previously posted an issue with
schema_registry_encode
that was resolved by fixing a race conditionI'm getting similar issues with, but an extra key is now truncated:
would result in an error message like
The text was updated successfully, but these errors were encountered: