basic: merge eval_ast into EVAL

[asarhaddon]

I intend to successively push and test small commits until the macroexpand function can be removed.

[kanaka]

@asarhaddon Sounds like a good plan. I've been quite impressed by your rapid progress so far. But I wasn't sure if you were going to tackle basic. The combination of early BASIC's lack of features along with garbage collection makes it pretty daunting TBH.

[asarhaddon]

The last commit fails on step8 for both variants, with the same message: "file not found".
I fail to relate this with the changes from step7 to step8, which almost only affect macro expansion.
Any clue ?

[kanaka]

Let me try this locally and see if I can find a clue to what's happening.

[kanaka]

@asarhaddon Somewhere along the way the filename string (for the command line argument file to load ) that is passed to DO_READ_FILE gets corrupted:

$ docker pull ghcr.io/kanaka/mal-test-basic:20200211_055016-g8a19f603

# add debug line to DO_READ_FILE in impls/basic/core.in.bas

$ make DOCKERIZE=1 repl^basic^step7
docker run -it --rm -u 1000 -v /home/joelm/personal/programming/mal/:/mal -w /mal/impls/basic -e basic_MODE=cbm ghcr.io/kanaka/mal-test-basic:20200211_055016-g8a19f603 make basic_MODE=cbm step7_quote.bas
make: 'step7_quote.bas' is up to date.
REPL implementation basic, step file: impls/basic/step7_quote.bas
Running: docker run -e STEP=step7_quote -e MAL_IMPL=js -it --rm -u 1000 -v /home/joelm/personal/programming/mal/:/mal -w /mal/impls/basic -e basic_MODE=cbm ghcr.io/kanaka/mal-test-basic:20200211_055016-g8a19f603 ../basic/run 
DO_READ_FILE: .args.mal
user>

$ make DOCKERIZE=1 repl^basic^step8
docker run -it --rm -u 1000 -v /home/joelm/personal/programming/mal/:/mal -w /mal/impls/basic -e basic_MODE=cbm ghcr.io/kanaka/mal-test-basic:20200211_055016-g8a19f603 make basic_MODE=cbm step8_macros.bas
make: 'step8_macros.bas' is up to date.
REPL implementation basic, step file: impls/basic/step8_macros.bas
Running: docker run -e STEP=step8_macros -e MAL_IMPL=js -it --rm -u 1000 -v /home/joelm/personal/programming/mal/:/mal -w /mal/impls/basic -e basic_MODE=cbm ghcr.io/kanaka/mal-test-basic:20200211_055016-g8a19f603 ../basic/run 
DO_READ_FILE: f

?FILE NOT FOUND  ERROR IN 246

[asarhaddon]

Thanks. I was forgetting that MAL itself read files. I intend to backport the separate evaluation of the function and the arguments from step8 downto step2. They are only required by macros, but it is good to debug them step by step. This one will probably explode far before step6.

[kanaka]

Thanks. I was forgetting that MAL itself read files. I intend to backport the separate evaluation of the function and the arguments from step8 downto step2. They are only required by macros, but it is good to debug them step by step. This one will probably explode far before step6.

That makes a lot of sense, especially for basic (and probably wasm too) where the memory is managed. Getting similar code paths in place as early as possible is good so that problems with reference counting and memory management can be caught earlier where it's simpler to debug.

[asarhaddon]

I am giving up for now. Here are the remaining blockers.

• aggressively reduce diff between steps passes CI with cbm but breaks the starting of step0 with qbasic.
• separate evaluation with cbm fails during step4 tests (?BAD SUBSCRIPT ERROR IN 138, that is when HASHMAP_GET compares the searched and current keys), but the same test in a local REPL succeeds. So maybe a gc issue.

I have updated the Dockerfile in order to try some local tests. The qb64 part seems completely out of date. This is not a prerequisite for this merge request.

Ironically, merge eval_ast and macroexpand into EVAL seems OK apart from these issues.

[kanaka]

@asarhaddon I did a little searching and found that FreeBASIC (freebasic.net) has qbasic support (fbc -lang qb). So I updated the Dockerfile and switched from qb64 (which has always been a bit weird under Linux because it kind of assumes a UI) to freebasic for the qbasic mode. There were a few other minor changes I needed to make around file I/O and the TIMER function but now both modes are passing up through aggressively reduce diff between steps. So I pushed those to upstream and rebased this PR. This PR now just has the f/args separation and the eval_ast/macroexpand commits.

I might be able to look at why those are failing in the next couple of weeks but won't be able to touch it again for a few days at least.

If you do the following at the REPL you can trigger errors that are pretty clearly a GC/reference counting issue:

$ make DOCKERIZE=1 basic_MODE=cbm repl^basic^step4
...
user> ( (fn* (f x) (f x)) (fn* (a) (+ 1 a)) 7)
8
user> ( (fn* (f x) (f x)) (fn* (a) (+ 1 a)) 7)
8
user> ( (fn* (f x) (f x)) (fn* (a) (+ 1 a)) 7)
Error: '+' not found
user> ( (fn* (f x) (f x)) (fn* (a) (+ 1 a)) 7)
Error: '+' not found
user> ( (fn* (f x) (f x)) (fn* (a) (+ 1 a)) 7)
[HANG]

The above starts happening with the f/args separation (and it's the same with cbm or qbasic).