Avro 1.12.0 breaks jackson-dataformat-avro (also: requires JDK 11)

[rdifrango]

We were attempting to upgrade our project to Avro 1.12.0 with jackson-dataformat-avro version 2.17.2

The failure I get is as follows:

Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.avro.Schema$Parser org.apache.avro.Schema$Parser.setValidate(boolean)'
	at com.fasterxml.jackson.dataformat.avro.AvroMapper.schemaFrom(AvroMapper.java:240)
	at com.fmr.difrango.ztester.DemoJacksonAvroFailure.main(DemoJacksonAvroFailure.java:23)

To demonstrate this, I created the following example program that uses issue19.avsc as the schema to load.

package com.difrango.ztester;

import com.fasterxml.jackson.dataformat.avro.AvroMapper;
import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

public class DemoJacksonAvroFailure {
    private static final Logger LOGGER = LoggerFactory.getLogger(DemoJacksonAvroFailure.class);

    public static void main(String[] args) {
        var expectedStream =
                DemoJacksonAvroFailure.class.getClassLoader().getResourceAsStream("issue19.avsc");

        var mapper = new AvroMapper();
        mapper.registerModule(new Jdk8Module());
        mapper.registerModule(new JavaTimeModule());

        try {
            var expected = mapper.schemaFrom(expectedStream);
        } catch (IOException e) {
            LOGGER.error("Error Handling the Schema", e);
        }
    }
}

[cowtowncoder]

Yes, see #167 -- you cannot upgrade Jackson 2.17 and below to Avro library past 1.8.2.

But upcoming 2.18.0 should allow upgrade.

[rdifrango]

Thanks @cowtowncoder. It's interesting that we've been successfully using AVRO 1.11.2 so it must be some edge cases that we aren't currently hitting.

[cowtowncoder]

@rdifrango Ah. I forgot that the issue wrt upgrade were mostly in unit tests, not on binary compatibility between module and codec library.

But I hope 2.18.0-rc1 -- being released today! -- will work with 1.12.x and above.

[cowtowncoder]

2.18.0-rc1 now released: @rdifrango not sure if it's easy to do, but it'd be great to see if upgrade would work with 2.18, for eventual 2.18.0 release

[rdifrango]

2.18.0-rc1 now released: @rdifrango not sure if it's easy to do, but it'd be great to see if upgrade would work with 2.18, for eventual 2.18.0 release

I just tested it and I am seeing the same issue.

I just looked at AvroMapper and noticed that here it still has new Schema.Parser().setValidate(true) and it's the .setValidate(true) that was removed in 1.12.0

[cowtowncoder]

@rdifrango rats. Ok. That may also explain why upgrade was to 1.11.3 not 1.12.0.

[rdifrango]

@rdifrango rats. Ok. That may also explain why upgrade was to 1.11.3 not 1.12.0.

Quite possibly...and I'll grant that the move to 1.12.0 is not backwards compatible.

[cowtowncoder]

It seems odd for Avro library to nominally follow SemVer but... seemingly not?
(although as a library maintainer I understand how difficult backwards-compatibility is).

But would be good to know why the method was removed & if it had been at least deprecated earlier. Plus whether there might be replacement to use, from older versions too.

[rdifrango]

It seems odd for Avro library to nominally follow SemVer but... seemingly not? (although as a library maintainer I understand how difficult backwards-compatibility is).

But would be good to know why the method was removed & if it had been at least deprecated earlier. Plus whether there might be replacement to use, from older versions too.

I'd agree @cowtowncoder especially since I'm having a conversation with on this PR with the commons-validator team and they are being very strict.

I did look over on the avro side, but couldn't find any specific mention of why this was changed and it was unclear to me if it's now simply the default behavior of the parser or not.

Should we ask over there as well?

[cowtowncoder]

@rdifrango I guess damage is done, API compatibility broken b/w versions. So not sure how much value there would be.
(not sure how commons-validator is related -- is that a dependency they added/removed?)

[rdifrango]

@rdifrango I guess damage is done, API compatibility broken b/w versions. So not sure how much value there would be. (not sure how commons-validator is related -- is that a dependency they added/removed?)

agreed the damage is done, the commons-validator was more an indicator that other apache projects seem to hold to strict guidelines around binary and API compatibility which clearly Avro has not.

[cowtowncoder]

One other reason why we probably cannot upgrade: looks like Avro 1.12.0 is built with JDK 11, and Jackson 2.18 still has Java 8 as the baseline.

[rdifrango]

One other reason why we probably cannot upgrade: looks like Avro 1.12.0 is built with JDK 11, and Jackson 2.18 still has Java 8 as the baseline.

That definitely makes this tricky, any thoughts on a path forward?

My concern is that if some sort of CVE is discovered in the AVRO project there currently isn't a path forward.

[cowtowncoder]

@rdifrango No idea short-term. Things are tricky as it's multi-module Maven project so would need to at least build all with JDK 11; but then try to target JDK 8 (if even possible) for non-Avro backends.

But on very short term I think baseline code needs to remain 1.11.x.

Ideally we would let users override with newer version tho.

And I think it'd actually be possible to use matrix build for CI testing to try version override, if we get module itself run against newer library version -- but that's not solved yet; how to avoid/change parser.setValidate() call.

[cowtowncoder]

One quick note: Jackson 3.0 (from master branch) will have JDK 17 (at least) as baseline, so dependency could be increased there without problems. While that won't be released until maybe 2025/march (first Release Candidate), it'd be useful upgrade, if anyone could create a PR.

[cowtowncoder]

Ok, so Avro lib backwards-compatibility is really... not. There isn't any. 1.11.3 -> 1.11.4 breaks tests, as per #539. Patches can be as breaking as minor version or so.