Notes on "high-level" HTTP support #124
Comments
|
A few notes from an outsider 😉:
Oh, and while I'm at it, I'd like to thank you and the other people working on this, this project must meet at lot of difficult resistance, and just be really difficult in general, I really hope you succeed eventually! |
Whoa, thank you! I had no idea anyone was even paying attention to this repo. Getting feedback is very exciting :-) (Btw, if you want to get more involved and stop being an outsider, we can make that happen.)
I totally get the feeling of what you're saying. But I have trouble mapping it back to specific details in the requests code. This is one of my major motivations for reading through it in the first place – to figure out what requests actually does, and hopefully figure out which parts are obvious and uncontroversial, and which parts are too smart for their own good. So I'm hoping you could elaborate a bit more? Is it the automatic redirect handling that gives you problems? Do you hate having the option to call
Thanks! |
|
More unstructured brain dump: So one natural-ish design would be to say that the low-level API exposes a single operation: class AbstractAsyncHTTPTransport(ABC):
async def request(self, verb: bytes, headers: List[Tuple[bytes, bytes]], body: AsyncIterator[bytes]):
...
return (status_code, status_message, headers, response_body)And (And of course we'd have a sync version of this interface too.) That API is missing some options/configuration, especially for TLS and proxy settings. I guess users expect to be able to set these per-request, which creates some interesting challenges for pooling. (If a connection was originally created with cert verification disabled, then you'd better not pull that connection out of the pool later when cert verification is turned on!) Maybe we should make them always set per-request, to simplify things. Let the higher-level code worry about proxy configuration policy. Our default implementation of Then the higher-level API has a
redirects have to be somewhat tightly integrated with the user-friendly APIs, because the redirect code has to know how to rewind file objects, and to be able to report the history of redirects back in the final There's also handling of responses with code 401 Gimme Some Authentication Please. I think maybe this can be kind-of like a redirect? requests has some code for this but I don't understand the details. Need to look into this more. Exotic cases that we might want to make possible, or at least not accidentally rule out?Upgrade/CONNECT: these are interesting because you either get a regular response back (if the upgrade/CONNECT is denied), or else you get a new bidirectional byte stream. There's also an annoying complication HTTP/1.1 has both upgrade+CONNECT, but in HTTP/2 they dropped upgrade and instead extended CONNECT (see RFC 8441). So if you want to abstract over HTTP/1.1 versus HTTP/2, then you have to translate between these two approaches or... something. This has some interesting use cases: websockets are the most prominent (and it would be neat to be able to multiplex websocket+regular HTTP to the same server over a single HTTP/2 connection! That's actually supported by the RFCs), but there are also some cool things you can do if you can negotiate a new protocol when talking to an HTTP server. E.g. the docker API uses this for It wouldn't be too wacky to support websockets in the same library, at the Maybe the thing to do here is to have a second method on the Bidirectional communication over vanilla HTTP/2 (without CONNECT): grpc is a bit weird: it uses the HTTP/2 wire format but with slightly different semantics than real standard-compliant HTTP/2. In particular:
And actually, HTTP/1.1 has a similar situation: there's nothing in the wire protocol that prevents doing bidi communication like this, but it's not really part of the HTTP semantics and isn't well-supported by middleboxes. And in our current HTTP/1.1 code we actually abort sending the request as soon as we start seeing a response, because that's the most correct thing to do for regular HTTP semantics. So... do we need a way to enable bidi incremental request/response bodies? Do we need it in general, or just for HTTP/2? What would the semantics and interface even look like? I'm really not sure about any of this. Trailers: Also needed by grpc, as well as some real HTTP use cases. These are an optional block of headers that can arrive after reading the response body. So I guess the interface for response bodies isn't quite a LifecyclesHow do we manage the lifetime of session objects and individual response objects? traditionally libraries like requests/urllib3 have allowed users to use context managers, but this is optional – if you don't, then the GC will eventually clean things up. This also matches how Python handles most resources, like files. Async and new HTTP protocols make this more complicated. Let's talk about session lifetimes first. HTTP/2 and HTTP/3 assume that there's some kind of continuously-running background-task to handle stuff like PINGs. Tom Christie has argued there might be some way to finesse this – encode/httpx#258 (comment) – so I guess we should double-check – but I think we'll need a background task anyway to avoid nasty issues with cancellation, so the most we could hope for is to make tasks that live as long as individual response objects. Anyway, the reason all this matters is that we believe in structured-concurrency so we think that background tasks require an The simplest way to handle this would be to simply make it mandatory to use Things to think about:
The actual resource release for a session/connection pool is pretty trivial... basically just close all the sockets and you're done. So async doesn't make that part any more complicated – the GC works as well for resource cleanup as it ever does. Then there are responses. If we scope background tasks to the session level, then we don't need background tasks for individual responses. So in that regard, no [Note to self: check how aiohttp handles this. I think they at least STRONGLY RECOMMEND Also related: Should we have a This has an interesting interaction with resource management: if you do it lazily in general, then you have to decide what policy to use when someone closes the response without accessing the body. Do you close the connection, forcing it to be re-opened? Or do you try to read the response body, so the connection can be returned to the pool? This is also closely related to my questions in my first post, about how to handle the body of redirect responses – do we read it, discard it, what? (Well, if you have a Oh hmm we also need to avoid the situation where the server can cause Another question: Should closing a session immediately close all responses, even ones that are currently "alive"? It makes a lot of sense for HTTP/2, since there responses are relatively thin wrappers around some shared state that's most easily-managed at the session level. And probably we want HTTP/2 and HTTP/1.1 to act the same, to avoid surprising bugs. So maybe tentatively, yes, we should track "checked out" connections and close them when the session is closed. |
|
Another thing to think about: "retry handling", which is similar to "redirect handling" in some ways, but different. Retry = automatically responding to a failed request by re-issuing it, versus redirects which are explicitly initiated by the server. You can't do retry on all failure modes, but there are some where it makes sense – in particular early failures like DNS resolution or a timeout on connection establishment, where you know you never started submitting the request to the server. It's safe, you know these kinds of problems are often transient, just do it. There are also server-initiated retries, involving the Do you want to handle retries at the highlevel Currently when using requests, I think requests handles redirects while urllib3 handles retries, so it's like the |
|
I should also look through urllib3 to think about what stuff is happening there and whether it makes sense as the place to do it. For example, I know urllib3 handles automatic decompression of gzipped responses – is that something you would want at the low level or the high level? |
|
wacky idea for dealing with draining connections: put the connect back into the pool undrained, and then if we pull it out again, try to drain it then (with some limit on how much to read). The idea is that this moves the potentially-blocking draining into another operation that also blocks waiting for the same server, so if the user is using timeouts properly then it should Just Work. This is probably Too Clever By Half, but I figured I'd capture the thought... |
Huh, apparently aiohttp actually implemented its own cookie jar code! It still uses some of the code from |
|
I figured out what's going on with the 401 handling in requests' auth code. If you're using digest authentication, then you have to do two requests: the first one gets denied and in the denial it includes a nonce; then the nonce is used to generate the actual authentication token; and then we repeat the request with our new token. I guess this is one reason that digest auth has always been unpopular... if you want to upload a large file with authentication, then you have to upload it twice! It seems like there's a fundamental similarity between redirects, the On another topic: does it even make sense to keep a split between "high-level" and "low-level" APIs? I'm looking at that list of value-added features that requests gives, and it's like... well, you probably want redirect handling almost always, and if not there will be way to disable it. Auth doesn't do anything unless you ask for it. And people probably do want cookies in pretty much all situations, or if not then we'd want a way to disable it anyway. And if you don't want to use helpers like So it sounds like the remaining reason people rely on this split is because they're using requests's "mounting" capability as an extension point so they can interpose and mess with things. I looked around to find real-examples of how people use this:
Actually in general requests-toolbelt is a treasure trove of info on exotic extensions people have asked for. Some more interesting extensions:
We can group these into three categories:
|
I think the concept is sound in some sense – the rules for keeping auth are basically "if and only if you keep the same destination schema+host+port". But there's a problem: the user can also set their own manual Auth headers (and in fact this is super common, since the normal HTTP auth mechanisms are so limited). And on redirect, you want to strip those too. So for simple kinds of auth, maybe you want to set up the headers just once, before entering the redirect handling loop. And then inside the redirect loop, as a separate matter, you want to strip the headers, because that way you treat them uniformly regardless of how exactly the user set them. For auth that requires a 401 back-and-forth, like Digest and I think some of the other fancy ones like Kerberos or NTLM, it's a little trickier. I guess those need to go "inside" the redirect loop, because if you get redirected from A -> B, and B sends a 401, and you have credentials for B, then you should use those credentials to retry the request to B, not go back to A and send it credentials that it might not even know what to do with. But, of course, the redirect loop needs to cancel this entirely if it crosses between sites. So actually I'm not so sure... maybe you want to handle all "structured" auth at this stage, and strip user headers as a separate thing. (It's important to think carefully about these cases... requests has had 4 CVEs over its life, and every single one involved bad interactions between redirects+auth.) Another bit of fun: proxies can also require authentication. For |
|
I realized Go's The overall structure is actually eerily similar to the straw-man There's even a dispatch mechanism similar to requests's "mounting". Except, it's not on the Their Their multipart parser has a mandatory (!) There's a trace hook mechanism, to register callbacks that get called at various points in the request lifecycle (https://golang.org/pkg/net/http/httptrace/). I think this must be the inspiration for aiohttp's similar feature? You can do They support sending trailers on requests, via a somewhat awkward API (you fill in a Redirects: they have an interesting linked-list kind of representation – each In their redirect chain, all the previous responses don't have a body attached – it's automatically closed. Normally this would kill the connection, which is inefficient since you might want to reuse the connection for the redirect. In order to avoid this, they have some special-case code that runs on redirects to drain and discard up to // Close the previous response's body. But
// read at least some of the body so if it's
// small the underlying TCP connection will be
// re-used. No need to check for errors: if it
// fails, the Transport won't reuse it anyway.
const maxBodySlurpSize = 2 << 10
if resp.ContentLength == -1 || resp.ContentLength <= maxBodySlurpSize {
io.CopyN(ioutil.Discard, resp.Body, maxBodySlurpSize)
}
resp.Body.Close()There's lots of interesting undocumented tidbits like this in the source. Here's another example: // If the caller specified a custom Host header and the
// redirect location is relative, preserve the Host header
// through the redirect. See issue #22233.You can register a callback that gets consulted before following each redirect, and it gets the option to stop following the chain at any point. The callback gets a reference to the next Unrecognized 1xx responses are mostly ignored, except that they trigger a trace event. The exceptions are 101 (which indicates a successful Upgrade, as noted above), and 100 Continue. For 100 Continue, they have a setting on the There's a mechanism for setting a header to The details of how they handle headers on redirects are subtle and interesting to compare to what the Python-land libs do:
Proxy configuration: the default To find out how people take advantage of the
I'd say that overall, these are more compelling use cases than GAE. Test fakes and debugging tools are especially compelling, maybe enough to justify their own dedicated API. For several of the rest, I wonder if it would be better to wrap something around the |
|
Something that might be a useful pointer here is that I'd be super keen to have see a dispatch class implementation based on your work here, or discover if there's any blockers that make that infeasible for any reason. Concretely: Implementing a dispatch class means subclassing the |
|
Some critique of Go's net/http: https://github.com/gorilla/http And fasthttp is mostly focused on speed, not ergonomics, but from their README: |
|
IMO it's preferable to not have a "PreparedRequest" API, and instead supporting all requests via I'm also a fan of only exposing a string interface for URLs on However that presents some unique challenges to support all of HTTP without PreparedRequests. A solution I quickly thought of would be a flag in |
|
Yeah, |
I've heard repeatedly that
requestsis a "high-level" HTTP API andurllib3is a "low-level" HTTP API, but I wasn't super clear on what exactly that meant in practice. So I readrequeststoday in hopes of learning more about what secret sauce it's adding. These are some notes/thoughts I wrote down while doing that.Things that requests does:
Convenience functions:
get,post, etc., plus all the helpers for encoding and decoding bodies. Including:the thing where you can pass in a bunch of different formats of request body and requests will try to DWIM
ability to read the response body as bytes, text, json, streaming bytes, streaming text, lines, ...
charset autodetection when reading text/json
adding
http://on the front of URLs that are missing a schemechoosing between
Content-LengthandTransfer-Encoding: chunkedframingmeasures time from starting the request until response headers are received (or something like that)
lots of fiddling with URLs and URL encodings. Seems to normalize URL encoding even. Uses
urlparse, which is highly dubious. Not sure what this is actually about.a "hooks" system that lets you register callbacks on particular events: https://2.python-requests.org//en/master/user/advanced/#event-hooks
Responseobject before it's returned to the userautomatic redirect handling. this has a lot of interesting complexities
there's some redirect defaults in
requests/sessions.pyforget,options,head... but not the other methods for some reason? I'm not sure what this actually does. (they're also inrequests/api.py, but AFAICT that's totally redundant)requests accepts file objects as request bodies, and streams them. This requires seeking back to the beginning on redirects
switching proxy configuration when redirected
saves the list of response objects from intermediate requests. I think the body is unconditionally read and stashed in memory on the response object! this is plausibly correct, since I guess redirect bodies are always small, and you want to consume the body so the connection can be released back to the pool? though there's some kind of DoS opportunity here if the server gives back a large body – normally a client can protect themselves against large response bodies by choosing to use the streaming body reading APIs, but in this case requests unconditionally fetches and stores the whole body and there's no way to stop it. Not the scariest thing, but probably should be considered a bug.
there's a lower-level "incremental" API: if you set
allow_redirects=False, you get back the response object for the first request in the chain, and that object has some extra state on it that tracks where it is in the request chain, which you can access withrequests.Response.next. However, I'm not 100% sure how you use this... seems like it would be more useful fornext()to go ahead and perform the next request and return the response? maybe if the API were that easy, that could be what gets recommended to everyone who wants to see the detailed history of the redirect chain, and avoid some of the awkwardness about intermediate bodies mentioned in the previous bullet point?the redirect code uses a ton of mutable state and is entangled with a lot of the other features here, which makes it hard to understand and reason about
authorization
digest auth handling is really complex: thread-local variables! relies on the hook system to integrate into redirect handling! reimplements the file seek on redirect handling (I have no idea why)! some silly code for generating nonces that combines
os.urandomand ad hoc entropy sources!if the user passed
https://username:password@host/URLs, requests removes them and converts them into aHTTPBasicAuthsetupcookies: you can pass in a cookiejar of cookies to use on requests, and it will automatically update it across a
Sessionmounting different adapters
setting up TLS trust via certifi
proxy autoconfiguration from environment (reading
.netrc,$http_proxy,$NO_PROXY, etc.)forces all HTTP verbs to be uppercase -- interesting choice. the HTTP RFCs say that method names are case-sensitive, so in principle
GETandgetare both valid HTTP verbs, and they're different from each other. In practice, I've never heard of anyone using anything except all-uppercase, and apparently the requests devs never have either.pickling!
Dealing with cookies is really messy! AFAICT there isn't a modern WHATWG-style standard yet, so no-one entirely knows how they're supposed to work. In Python there's the venerable
http.cookies, whose API is super gnarly to work with, and I'm skeptical that its semantics are really suitable for the modern world, given that it comes from the same era ashttp.client. There's thiscookiespackage on PyPI, which hasn't had a release since 2014, but the README has a bunch of important-sounding critiques ofhttp.client. And... that's about it! Eek.I've been thinking about architectures like: a
package.Sessionwith a highlevel API (similar to requests), andpackage.lowlevel.ConnectionPoolwith a lowlevel API (similar to what urllib3 has now, but simplified), where the low-level object is "mounted" onto the highlevel object, so we can use the mounting feature as an API for users to inject quirky configuration stuff. This is on the same lines as httpx's middleware system. httpx wants to use this for redirects/auth/etc. Looking at the notes above, I'm not sure they're that easily splittable? But need to think about it more. What parts would people reasonable need to be able to mix and match?Note: I literally just mean I've been thinking about these kinds of architectures, like as a design puzzle. I'm not saying that we should adopt an architecture like that for this project. But I am pondering what we should do.
The text was updated successfully, but these errors were encountered: