Should we drop Python 2.7 support?

[pquentin]

When this project was conceived in mid-2017, it was obvious that we needed to support Python 2.7, because:

• Python 2.7 was going to be a thing until April 2020.
• As of November 2019, urllib3 still gets slightly more than 50% of Python 2 downloads, does not plan to drop 2.7 anytime soon, and Red Hat is going to support Python 2.7 until 2024.
• This was designed as urllib3 v2, so it was going to support the same versions as urllib3.

However, it's now clear that this project won't be urllib3 v2, and we're going to offer a completely different API. We won't offer a stable version soon, and nobody should be starting new Python 2.7 projects.

What is Python 2.7 support costing us? Surprisingly, not much, because it's already here, so it's not a lot of effort to keep the support. Here are possible benefits of dropping Python 2.7:

• reduce the number of builds we have to worry about in CI
• use nonlocal instead of the context hack in connection.py, and remove other Python 2 compat code, especially Python < 2.7.9 SSL stuff
• stop excluding async backends from coverage reports
• use the latest versions of test dependencies such as Tornado and pytest who no longer support Python 2.7
• possibly stop using code generation for runtime code and use run_secretly_sync_async_fn instead for sync support
• possibly use type annotations without hacks

My personal conclusion: I think it makes sense to remove Python 2.7 support (and App Engine!) at the same time we stop merging changes directly from urllib3, which should happen real soon now (TM).

[sethmlarson]

We should drop 2.7 and 3.5 IMO. 3.5 usage is dwindling even now.

[njsmith]

So I think the main reason to consider supporting py2 is not in your list, and that's to ease adoption for projects that will be stuck supporting py2 indefinitely. I'm not convinced this is a good reason either, but let's at least think about it.

I'm thinking of projects like pip, botocore, docker-py, etc. These are major, extremely widely used projects that currently use urllib3. They would like to have async support. But they're not even considering dropping py2 support yet, because, you know, enterprise. So it might be significantly easier for them to adopt hip if they can use it on py2 as well, and maybe that's the difference between them adding async support in our lifetimes, or not.

If the bargain was that botocore will get native trio/asyncio support if and only if we keep supporting py2 in hip, then I would say let's hold our nose and support py2.

Also, if the bargain was that hip supporting py2 -> more folks switch away from urllib3 -> we get to stop supporting legacy urllib3 sooner, that might also be worth it.

I'm not sure that's actually how it will work though :-). Some reasons it might be wrong:

• these projects are conservative and we haven't been moving super quickly, so they aren't even going to consider switching to hip for at least another year or two, at which point the landscape might look very different. And there are definitely going to be stragglers using urllib3 for the rest of time, no matter what we do.

• http clients all have pretty similar interfaces. Maybe it's just as easy for these projects to write a bit of code to switch between using legacy urllib3/requests when running on py2, and hip on py3. They might even prefer this, to reduce the risks of switching all their users to a new http client at once.

• even if py2 support does turn out to be important, we know it's definitely not important right now while we're in the fast-moving development stage aimed at early adopters. So maybe we shouldn't be paying the carrying costs of py2 now, when we don't even know whether it will turn out to be important. Instead maybe we should wait until we are ready for widespread adoption, and then if py2 turns out to be a significant barrier then we can port to py2 at that point.

[pquentin]

Thanks! I think we can continue as we've done before, and keep Python 2.7 unless it costs us too much. For example, it might make things to hard when testing on multiple backends.

[sethmlarson]

Here's a thought from me: can we drop Python <2.7.9? I'm actually considering doing this for upstream urllib3 as well because it's both insecure (because it depends on pyOpenSSL which is a pile of hacks and doesn't work on macOS), very EOL, and it means we can depend on a proper SSLContext being available so we don't have to have all the work-arounds and test three separate TLS backends (it'd be only two instead 🎉).

It's a good amount of work to make hip and all of it's dependencies continue to support Python 2.7 but I'd be willing to do it as long as we don't have to support ancient Python versions.

ref: psf/requests#5267

[pquentin]

Yes, sure, let's do that! I'd be happy to work on a patch for urllib3 first and then merge it from there.

[sethmlarson]

Would require coordination with Requests / Pip / Boto before we could ship it, especially if we remove inject_pyopenssl_X() or turn it into a no-op. Cuz in a way we're basically making a versioning decision for them which we've been nice enough never to do in the past.

[agronholm]

I already noticed that having a confest.py use Python 3 syntax results in a collection failure on 2.7. I'm therefore leaning towards dropping support entirely.

[pquentin]

@agronholm is it because of the nonlocal keyword? We can use a ctx dictionary instead or put the file in a place where pytest won't collect it in Python 2.

I don't know if we should keep Python 2 or not as there are good arguments either way, but I think the decision should be taken more globally

[agronholm]

is it because of the nonlocal keyword? We can use a ctx dictionary instead or put the file in a place where pytest won't collect it in Python 2.

That was the immediate issue, yes. I know we can work around these problems, but the question is: why should we? They WILL be causing bigger problems down the road, and this is time I would personally rather spend writing new features instead.