Streaming/interactive reads

[jjttjj]

When I started using convex from clojure, it occurred to me that it would be nice to be able to easily turn my active clojure repl into a convex repl. I thought it might be easy to plug in convex to clojure.main/repl, since it already has its own versions of read and eval. However convex's read currently doesn't support reading a single form at a time, which makes this not possible. This limitation is referenced in a comment here.

It turns out antlr isn't great for supporting this kind of streaming read. I've found a good explanation of the problems in this stackoverflow answer, in both the answers and all the comments.

That said, I have hacked together something that seems to be a working solution to this problem in this fork. Things mostly seem to work. There are a couple tests that fail and some might need to be re-examined and the rest should be fixable. There may be other new issues I've introduced though and it probably needs some more days worth of testing/trying out. There are some breaking changes but I think I could mostly avoid them, if desired, by renaming the interactive reader function to read1 instead of changing the current read for example.

Now I can just do something like this from clojure to get a convex repl:

(require '[clojure.main :as main]
         '[convex.cell :as cell]
         '[convex.cvm :as cvm])
(import convex.core.lang.reader.AntlrReader)

(let [ctx (cvm/ctx)]
  (cvm/juice-set ctx Long/MAX_VALUE)
  (main/repl
    :read (fn [request-prompt request-exit]
            (let [x (or ({:line-start request-prompt :stream-end request-exit}
                         (main/skip-whitespace *in*))
                        (AntlrReader/read *in*))]
              (if (= x (cell/* :quit))
                request-exit
                x)))
    :eval (fn [x] (->> x (cvm/eval ctx) cvm/result))
    :print println))

I mainly wanted to log this as an issue to get a sense if this is worth continuing to explore and get some general feedback. My instinct is that the changes required here, and the fact that they're kind of hacky to do in antlr, might not be justified for what it actually adds to convex. But I'm happy to continuing exploring this a bit and eventually turn it into a PR if that's desired.

[mikera]

Hi @jjttjj this looks awesome! I'd definitely be happy with a PR that adds this kind of support.

Your fork looks like a fairly simple and clean implementation so I think this is the way to go. Some quick thoughts:

• We need to differentiate between cases where we want to read a single form (and fail if this doesn't consume the entire input) and cases where we want to read a single form (but don't care about the rest of the input). From an API perspective that seems like separate functions are warranted. I probably like read (one cell, consume all input), readOne (one cell, leave remaining input on Reader) and readAll (where you want a sequence of cells).
• Need clear docs and definitions throughout, since this will be depended upon by external libraries / tools
• Need good tests including failure / edge cases

[helins]

Thanks, very useful (and I like the simple idea of using main/repl)!
Intuitively, I thought Antlr would have support but the post you link makes sense.

The Reader is primarily targeted for tooling. Hence, while it is not crucial to the very core of Convex (since not used for anything on-chain), it is still pretty important for about anything else. For instance, when working on the Convex Shell, I've learned to live without streaming but often wished I had it.

Can you spot any obvious shortcomings with your current solution, besides that couple tests failing?