README / documentation update (#14)

Update README.md with more verbosity and a description of current project status, and some instructions.

README.md

@@ -1,96 +1,140 @@
-# 'moor'; lambdaMOO in Rust
+# 'moor'; lambdaMOO all over again.
 
-(note: name is provisional and awful)
+This is a rewrite / reimplementation of the LambdaMOO. It's done in Rust, and is intended to be a modernized version
+of the original LambdaMOO server, with the goal of being able to run existing LambdaMOO worlds, but also to provide a
+more modern foundation for future development.
 
-## Intent
-Because I don't have enough incomplete projects ...
+(note: name is provisional and awful, suggestions accepted)
 
-And in the general theme that all things get to be rewritten in Rust, because that is the way of things in the 2020s.
+## Background on LambdaMOO
 
-And scratching an itch of many years to build a more modern LambdaMOO inspired thing, without actually building a more
-modern LambdaMOO inspired thing and instead just building LambdaMOO itself
+LambdaMOO is/was a shared virtual social world system similar to a (or kind of) MUD, but (usually) with a social (not RPG/game)
+focus.
 
-### Well, only partially a jest...
+Chances are if you landed here you already know what this is, but I have a blurb here
+[here](doc/lambda-background.md) briefly summarizing what it is and why it's interesting.
+
+## Project goals / status
 
 The intent here is to start out at least fully compatible with LambdaMOO 1.8.x series and to be able to read and
-execute existing cores. 
+execute existing cores, and the 1.0 feature release is targeting this rather ambitious but also rather restricted goal.
+(Primarily to maintain focus so I don't get distracted by the shiny things I've wanted to do for the last 30 years.)
 
-### But then...
+### LambdaMOO is 30+ years old, why remain compatible?
+
+* Because it's easy to go into the weeds creating new things, and never finishing. By having a concrete goal, and something
+  to compare and test against, I may actually get somewhere.
+* Because the *actual* useful and hard parts of those old MOO-type systems was the "user-space" type pieces (like
+  LambdaCore/JHCore etc) and by making a new system run those old cores, there's more win.
+* Because LambdaMOO itself is actually a very *complicated system with a lot of moving parts*. There's a compiler,  
+  an object database, a virtual machine, a decompiler, and a network runtime all rolled into one. This, is, in some
+  way... fun.
+
+### Current status / features
+
+* Pretty much feature complete / compatible with LambdaMOO 1.8.1 with a few caveats (see below)
+* Can load and run LambdaMOO 1.8.x cores.
+* Have tested against JaysHouseCore, and most of the functionality is there. Bugs are becoming increasingly rare.
+* Hosts websocket, "telnet" (classic line oriented TCP connection), and console connections. MCP clients work, with
+  remove editing, etc.
+* Objects are stored in a concurrent transactional object database -- safe, consistent and happy. The architecture 
+  allows for cleanly adding different storage backends for new scenarios.
+* Monitoring/metrics support via Prometheus-compatible export.
+* Separate network-host vs daemon process architecture means that upgrades/restarts can happen in-place without
+  dropping live connections.
+
+## How do I use it?
+
+The easiest way to get started is to run the `docker compose` setup. This will bring up a complete server with `telnet`
+and `websocket` interfaces. The server will be setup with an initial `JaysHouseCore` core import, and will be set up with
+metrics monitoring via Grafana and VictoriaMetrics.
 
-... to actually implement the backend portions on a more modern foundation, with a proper disk-based 
-transactionally safe database and full multithreaded concurrency, and replacing the classic `telnet` 
-client connectivity with websockets and such.
+To do this, take a look at the local `docker-compose.yml` file, instructions are there, but it really just amounts to:
 
-### Current status
+    `docker compose up`
 
-   * Mostly feature complete. Have tested against JaysHouseCore, and most of the functionality is there.
-   * Objects are stored in a RocksDB database, and are transactionally safe and consistent.
-   * Accepts inbound websocket connections (in lieux of telnet), accepts Basic-Auth login, attaches them to a user, 
-     and executes commands.
-     ![Screenshot of simple session](./doc/screenshot-session.png)
-   * Has an HTTP endpoint for retrieving some public properties (such as `$login.welcome_message`) etc.
-   * A Read-Eval-Print loop can be brought up to work against the database, and can be used to poke around and 
-     execute verbs and such.
-   * And what's software without dashboards?
+Then connect (with your favourite MUD/MOO client) to `localhost:8888` and follow the login instauctions. Classic 
+clients like `TinyFugue` will work fine, and there are some newer clients like [BlightMud](https://github.com/Blightmud/Blightmud) 
+that should work fine. A partial and probably outdated list of clients can be found here: https://lisdude.com/moo/#clients
 
-![Screenshot of grafana dashboard](./doc/screenshot-grafana.png)
+Once you're familiar with how the docker setup works, you can get more creative. 
+
+An actual production deployment can be fairly easily derived from the `docker-compose.yml` file, and the provided `Dockerfile`.
 
 ### Missing / Next steps before 1.0
 
-   * Bugs, bugs, bugs. Collect em' all.
-   * Missing features:
-     * Quota support. 
-     * Background tasks resumption after restart (from DB and from textdump load.)
-     * Accept old-school "telnet" (raw ASCII TCP socket) connections in addition to websocket.
-       (Mainly useful for compatibility with existing MUD clients)
-     * Dump to a backup `textdump` format.
-     * 'out of band' network command support, used by some cores/clients.
-   * Improvements:
-     * Performance improvements. Especially caching at the DB layer is missing and this thing will run dog slow 
-       without it
-     * A bit better connection management generally.
-     * Better auth (SSO, OAuth2, etc?). Better crypt/password support.
+* Bugs, bugs, bugs. Collect em' all.
+* Generally, open issues / missing features can be seen here: https://github.com/rdaum/moor/issues
+* Major missing features:
+    * Quota support.
+    * Background tasks resumption after restart (from DB and from textdump load.)
+    * Dump to a backup `textdump` format.
+    * `$do_command`; LambdaMOO has the ability to attempt execution of a command through
+      user code on `#0:do_command`; if that fails, it then dispatches through the regular
+      built-in command handler. I need to get around to this.
+    * `read`; This is used for prompts, password changes, editor, etc. It's slightly tricky
+      because of the 'transactional' nature of I/O in Moor where all verb and I/O operations
+      can be retried on transaction commit failure. Haven't decided what to do about this.
+    * Actual transaction retry on commit-conflict. (Mainly because without actual users and stress testing I haven't
+      been able to provoke this scenario to test against yet. The hooks are there, just not done.)
+* Improvements needed:
+    * Performance improvements. Especially caching at the DB layer is missing and this thing will run dog-slow
+      without it
+    * Better auth (SSO, OAuth2, etc?). Better crypt/password support.
 
 ### Unsupported features that might not get supported
 
-  * `$do_command`; LambdaMOO has the ability to attempt execution of a command through
-    user code on `#0:do_command`; if that fails, it then dispatches through the regular
-    built-in command handler. This is not supported, and is actually fairly trick to 
-    implement given the asynchronous flow of the system right now.
-  * `read`; This is used for prompts, password changes, editor, etc. It's slightly tricky
-    because of the 'transactional' nature of I/O in Moor where all verb and I/O operations
-    can be retried on transaction commit failure. Haven't decided what to do about this.
-  * `encode_binary` & `decode_binary`:  These two functions allow for escaped binary 
-     sequences along with a network option for sending them, etc. 
-     But:
+* `encode_binary` & `decode_binary`:  These two functions allow for escaped binary
+  sequences along with a network option for sending them, etc.
+  But:
     * `moor`'s strings are utf8 so arbitrary byte sequences aren't going to cut it and
     * we're on a websocket, and have better ways of doing binary than encoding it into the
-      output. 
+      output.
     * The alternative will be to provide a `binary` type that can be used for this purpose
       and to have special `notify` calls for emitting them to the client.
-  * Network connections, outbound and inbound (e.g. `open_network_connection`, `listen`,  
-    `unlisten` etc). My intent is for the network service layer to be implemented at the Rust level, in the 
-    server daemon, not in MOO code.
-
-## LambdaMOO is 30+ years old, why remain compatible?
+* Network connections, outbound and inbound (e.g. `open_network_connection`, `listen`,  
+  `unlisten` etc). My intent is for the network service layer to be implemented at the Rust level, in the
+  server daemon, not in MOO code.
 
-* Because it's easy to go into the weeds creating new things, and never finishing. By having a concrete goal, and something
-  to compare and test against, I may actually get somewhere.
-* Because the *actual* useful and hard parts of those old MOO-type systems was the "user-space" type pieces (like
-  LambdaCore/JHCore etc) and by making a new system run those old cores, there's more win.
-* Because LambdaMOO itself is actually a very *complicated system with a lot of moving parts*; there's a compiler,  
-  an object database, a virtual machine, a decompiler, and a network runtime all rolled into one. This, is, in some
-  way... fun.
-
-### Someday ...
-
-Eventual new feature goals, after full MOO backwards compatibility has been achieved:
-
-* Embedded JavaScript engine to allow implementation of MOO verbs in a more modern standard language.
-* Extended protocol support, richer output than pure text (e.g. JSON events for clients to render, etc.)
-* Distributed rather than local-only storage of objects.
-* Incremental runtime / model changes.
-
-Contributions are welcome and encouraged. 
+### But then...
 
-Ryan ([email protected])
+The following are targeted as eventual goals / additions once 1.0 (fall 2023) is out the door:
+
+   * A richer front-end experience. Support for websockets as a connection method means that the server can provide  
+     a richer narrative stream to browsers (with in-core support assistance.) A client which provides a full proper 
+     UI experience with interactive UI components, graphical elements, and so on are the end-goal here.
+   * Support for multiple programming language for programming MOO verbs/objects. The backend has been written such that
+     this is feasible. Authoring verbs in JavaScript/TypeScript will be the first target, and WebAssembly modules are
+     also a possibility. These verbs would still run within the same shared environment and use the same shared object
+     environment, but would allow for a more modern programming experience.
+   * A more scalable server architecture; the system right now is divided into separate "host" frontends for network  
+     connections, and a common backend `daemon` which manages the database, virtual machine, and task scheduler. This
+     can be further split up to permit a distributed database backend or distributing other components, to meet higher
+     scalability goals if that is needed.
+   * Enhancements to the MOO data model and language, to support a richer / smoother authoring experience. Some ideas 
+     are:
+     * Datalog-style relations / predicates; for managing logical relationships between entities. This could allow
+       bidirectional (or more) relationships like already exist with e.g. `location`/`contents`, but more generalized,
+       and to allow for making complex worlds easier to maintain.
+     * Adding a map/dictionary type. MOO predates the existence of dictionary types as a standard type in most languages.  
+       MOO's type system only has lists and uses "associative lists" for maps, which are a bit awkward. Immutable/CoW
+       maps with an explicit syntax would be a nice addition. Other MOO offshoots (Stunt, etc.) do already provided this.
+     * Adding a `binary` type. MOO's type system is very string-oriented, and there's not an elegant way to represent
+       arbitrary binary data. (There's `encode_binary` and `decode_binary` builtins, but these are not the way I'd do it
+       today.)
+     * and so on
+
+## License.
+
+Currently licensed under the Apache 2.0 license. See LICENSE file for details.
+
+## Contribute and help!
+
+Contributions are welcome and encouraged.
+
+Right now the best way to contribute is to run the system and report bugs, or to try to run your own LambdaMOO core
+and report bugs. (Or to fix bugs and submit PRs!)
+
+## Enjoy.
+
+Ryan (ryan.daum @ gmail.com)

crates/README.md

@@ -1,9 +1,6 @@
 Directory layout for `crates/`
 
-  * `values` - crate that implements the core MOO discriminated union (`Var`) value type,
-    plus all associated types and traits and interfaces used by other crates.
-  * `kernel` - the actual implementation of the system: database, compiler, virtual machine,
-    task scheduler, implementations of all builtin functions, etc.
+Binaries:
   * `daemon` - the actual server runtime. Brings up the database, VM, task scheduler, etc, and provides an interface
      to them over a 0MQ based RPC interface, exposing any external network protocol to the outside world. 
      Instead, that functionality is provided by...
@@ -14,8 +11,13 @@ Directory layout for `crates/`
      as well as various web APIs.
   * `console-host` - console host which connects as a user to the `daemon` and provides a readline-type interface to the
      system.
-  * `rpc-common` - crate providing types used by both `daemon` and `host`, for the RPC interface
-  * `regexpr-binding` - crate providing bindings to the old regular expressions library used by
-    the LambdaMOO server, for compatibility with existing cores. This is a temporary measure until
-    this can be reworked with use of the `regex` crate and some compatibility translation
-
+
+Libraries:
+  * `values` - crate that implements the core MOO discriminated union (`Var`) value type,
+     plus all associated types and traits.
+  * `kernel` - the actual implementation of the system: database, compiler, virtual machine,
+     task scheduler, implementations of all builtin functions, etc.
+  * `rpc-common` - provides types & functions used by both `daemon` and each host binary, for the RPC interface
+  * `regexpr-binding` - provides bindings to the old regular expressions library used by
+     the LambdaMOO server, for compatibility with existing cores. This is a temporary measure until
+     this can be reworked with use of the `regex` crate and some compatibility translation

doc/lambda-background.md

@@ -0,0 +1,26 @@
+## What is LambdaMOO?
+
+LambdaMOO is/was a shared virtual world system similar to a (or kind of) MUD. It was originally written by Stephen White
+(U Waterloo) and added to and maintained by Pavel Curtis (Xerox PARC) (and others) in C, and released in 1990.
+
+It was novel in that the bulk of the world's behaviour was implemented in a virtual machine, and the world itself was
+stored in a shared persistent programmable object database.
+
+The world was structured -- like other MUDs -- with some of the aspects of interactive fiction / adventure games, where
+users could move around, interact with objects, and interact with each other using a text-based interface. The focus
+was primarily social interaction and creativity, and it was/is great.
+
+There were many LambdaMOO worlds, and the most famous was LambdaMOO itself, which was a social experiment in shared
+virtual space. It was a text-based virtual world, where users could create objects, and program them in a prototype-based
+object-oriented language called "MOO".
+
+LambdaMOO is still in use today, and there are many active worlds running on it, but the community these days is much
+smaller and less active than it was in the early/mid-90s.
+
+In its essence LambdaMOO is/was a live and interactive "social network" offering a somewhat richer social experience
+than what the web page-oriented systems of today offer. It's a shame that this line of evolution was mostly abandoned
+in the 90s.
+
+The original LambdaMOO server is still available, but is showing its age, and the codebase is not easy to work with to
+add fundamentally new features like a modern database, or to support new network protocols, and most importantly, newer
+user interface modalities / presentation modes.