Scala-Native Support (1000USD Bounty)

[lihaoyi]

STTP Support Scala-Native via Curl. Requests-Scala should be able to support it via the same approach. We just need it to be wired into the Requests-Scala codebase, possibly refactoring out the actual making of the request into src-jvm/ and src-native/ platform-specific code.

Any platform-specific functionality that doesn't work on Scala Native should be called out, explained clearly, and their associated tests moved into platform-specific folders as well

To incentivize contribution, I'm putting a 1000USD bounty on resolving this ticket. This is payable via bank transfer, and at my discretion in case of ambiguity.

[domaspoliakas]

I'll give this one a go 🙂

[domaspoliakas]

Just wanted to mention that I am still working on this and have not abandoned it! I have PRed HttpCookie to scala native now scala-native/scala-native#3927 and I'll be looking at how to solve the issue of SSLContext next.

[domaspoliakas]

A wee little update - the approach that sttp has taken unfortunately does not seem viable here; sttp simply does not allow any streaming with the curl backend which is how they got away with only using the libcurl easy API, but given that this lib is based on InputStream at its core - my understanding is that this is not quite possible. That is - curl's easy API does not allow piecemeal consumption of the data, and therefore we'd have to buffer the whole response into memory if we were to use the same approach. However curl's multi API I think has all the pieces that are needed here. There's http4s-curl as prior art which does exactly that, so I'm trying to see if I can adapt a similar approach for this repo.

[lihaoyi]

@domaspoliakas sounds good!

[lqhuang]

I have subscribed to this issue for a long time due to I also want to give a try in the beginning. But I'm a little worry about my junior skills in Scala Naive. Excited that @domaspoliakas is taking over the ticket.

I still hope I could give a hand, or we could cooperate to solve some problems?

Like above mentioned, we're currently trying to build Scala native API on top of libcurl just like https-curl did. I'm thinking could we extract the core libcurl as an individual binding lib, so both https-curl and requests are able to create their user interface on it?

Someone has tried to create a binding (https://github.com/fsat/scala-native-libcurl-bindings) years ago and obviously abandoned already.

What's your opinion?

Regards

[domaspoliakas]

I personally do like the idea (and indeed you could add sttp to the list of libs that could benefit from such a thing here), but one downside is that it would add a dependency. This has a couple of drawbacks. The first is that requests-scala is a very minimalist library and practically does not add any transitive dependencies, which this would change a little. The second is that it would be a dependency outside of this organisation's control. Whether these are blockers or not is for @lihaoyi to decide.

If you do get the green light - take a look at https://github.com/domaspoliakas/requests-scala/blob/scala-native-cross-build/requests/src-native/requests/curl.scala . I've already covered a chunk of the API, at least the things that I needed so far, which you could use as a starting point. I used sttp as a reference for that, so you'll notice that the style of the bindings is in many ways reminiscent of their implementaiton. Another thing to mention is that I of course treated this as an internal detail of requests-scala so it may need some cleanup in order to be published as a library of its own.

[lihaoyi]

I dont mind a third party dep as along as it is well maintained and stable. Most important thing is that the end to end workflow works; we can refactor the implementation however we like later

[sideeffffect]

You could potentially use curl bindings automatically generated by sn-bindgen
https://github.com/indoorvivants/sn-bindgen-examples by @keynmol

[keynmol]

Note that sn-bindgen produces Scala3-only bindings.
My recommendation for this would be for libraries to ship their own handcrafted bindings as long as the API surface is small.

Alternatively, lots of binding code can be copypasted from https://github.com/indoorvivants/sn-bindgen-examples/blob/main/example-curl/src/main/scala/generated/curl.scala#L3113 and adapted to fit Scala 2. It should be a lot easier than handcrafting the entire API

[domaspoliakas]

The scala-3 requirement was the main reason for hand-crafting the bindings in this case, although I have used the generated bindings as a guide for some of the trickier parts of the API. The API is actually fairly small in the end, and I don't think I'll be adding anything to the bindings at this point.

I'll take this opportunity to mention that parts of the test suite work already, so there is light at the end of the tunnel. There's still a bit of work before all of it is ok (the looming threat is still SSLContext and however I'll need to solve that), and then I'll need to clean it all up, but the end is definitely getting nearer :)

[lihaoyi]

Sounds great @domaspoliakas !

[lihaoyi]

@domaspoliakas any updates here? If not I'll be putting it up in the next set of bounties

[domaspoliakas]

June was unfortunately a tough month to find time for this and I apologise for disappearing as I have. I have been working on this again as of last weekend however. Simple functionality is working at the moment, but there are still quite a few things left:

• SSL related stuff
• Concurrent transfers (single-threaded)
• Multithreading
• Redirect handling
• Cookie handling
• A bug in multipart uploads

That last one has eaten quite a lot of my time so far with unfortunately not much to show for it. Generally uploads seem to work (some hand testing and some of the test suite corroborate this), but so far I am yet to find the cause or the right combination of flags to make the multipart test pass.

I am happy to continue working on it now that I have time again, but if you would rather I passed the torch on to someone else - that is totally understandable.

---

EDIT: got the multipart bug fixed now, so there's that

[lihaoyi]

Got it! If you're still involved then go ahead and keep working on it, just wanted to make sure it isn't dropped

[lihaoyi]

FYI for anyone reading, this bounty is still open in case anyone else wants to take a crack at this

[domaspoliakas]

Sorry for disappearing once again. I'll officially step aside for this one. For those of you interested in picking this up - I think my WIP branch here could be a useful resource, with the most interesting bits being the curl bindings here and the actual implementation here. A lot of the basic tests (e.g. these) pass on this branch, but there's an issue somewhere that is causing this test to fail that I never managed to track down. Due to that and also some other missing pieces I think I would personally restart with a fresh implementation rather than try to fix that one, but I think some of the trickier bits (e.g. headerfunction and friends) work well and could be very useful for any future implementors. Lastly - I have spent many hours reading curl docs and its code to try and get this to work, and would be more than happy to answer any questions. Feel free to tag me here, or find me on discord (wisetree on discord) and fire away. There's a lot to curl, and I'd love to help springboard any new attempts at this.

Finally - thank you for the opportunity to work on this one. It has proven to be too much for me, but I hope to see someone else succeed.

[lihaoyi]

No worries @domaspoliakas ! Thanks for taking a crack at it. Hopefully others will be able to build upon your work and move the ball forward

[lilac]

I think leveraging the http4s-amber (src) for support of Scala Native is better than relying on curl in the long term.

The http4s-ember currently only supports Scala Native 0.4.3 version, so it lags 1 version behind current one (0.5.x). We have to migrate it to latest version, along with its dependencies.

• fs2
• cats-effect

This route is more challenging, but once it's done, it also benefits the whole Scala Native community. What's more, both http client and server should work. What do you think?

[lihaoyi]

Bumping the bounty on this from 500 -> 1000USD

[lihaoyi]

IMO the best way to approach this would to implement the Java standard library HttpClient, after which requests-scala should just work out of the box without changes

[lqhuang]

I'm trying to implement this feature by following @domaspoliakas's attempts, which are introducing curl as HTTP client (my fork). @lihaoyi, thank you for providing more details to help.

This is also a message to clarify my interesting of bounty, but I'll probably be slow, like 1 or 2 months. Anyone others cloud also give their own PR :).