Developer Guide

This document is intended for people planning to make changes to, and fix bugs in, the Royale Compiler.

The Royale Compiler is a set of Java projects. Eclipse is used by most of the developers working on the compiler, but you can just use any text editor and Apache Maven or Apache Ant to compile and test your changes. For Eclipse setup instructions see this document Eclipse Setup.

The compiler uses several third-party libraries. The ActionScript parsing is handled by Antlr 3. MXML parsing uses JFlex. SWF output uses JBurg. SWF output also uses LZMA for SWF compression. JS output uses Google Closure Library. JUnit is used for testing. And Apache Commons and Guava are used throughout. Volunteers who want to consolidate the parsing to use, say, Antlr 4 are welcome.

Projects

The compiler projects are described below and listed in build dependency order.

These two projects rarely, if ever, change and thus aren't even in the main Maven build. You will probably never need to know what is in these projects.

• compiler-build-tools - utility classes that are used during the build to generate other classes
• compiler-jburg-types - classes that need to be compiled early by Maven for compiling against JBurg later

These next 4 are the main projects for the compiler.

• compiler-common - classes shared by all (or most) of the various compilers.
• compiler-externc - "Externs Compiler" classes used to generate Typedefs SWCs from Google Closure externs. Typedefs are used to represent external libraries.
• compiler - This project contains the parsers and lexers and SWF output code.
• compiler-jx - This project is the transpiler. It can be configured to output JavaScript and just about anything else.

The rest of the projects don't get touched too often.

• compiler-test-utils - This project contains utilities for testing. It abstracts the differences between where Maven and Ant like to put output files and lets you setup variables that point to other dependencies.
• swfutils - These are classes copied from the Flex SDK used by debugger and flex-compiler-oem projects to parse SWFs. These are unlikely to ever be modified.
• debugger - The command-line SWF debugger. Many IDEs use the debugger underneath a graphical user interface.
• flex-compiler-oem - This code is used to integrate the compiler with Adobe Flash Builder. Other IDEs might use it as well.
• royale-ant-tasks - The Apache Ant integration project.
• royale-maven-plugin - The Apache Maven integration project.

Terminology and Concepts

The compiler has a set of "clients" that are entry points into the compiler. They will be found in packages ending with "client". There is an mxmlc and compc client just for SWF output in order to be more backward compatible with Apache Flex. There is an externc client for generating TypeDefs SWCs. The compiler-jx project has its own clients called mxmljsc and compjsc that launch the compiler code with different output generators known as "emitters".

The compiler-jx project also introduces the concept of a "target" which is a set of emitters and an optional "publisher". A publisher takes the output and does some post-processing. The default Royale Publisher for JavaScript calls the Google Closure Compiler to optimize the output JS. There is a Cordova Publisher that further calls Apache Cordova command-line commands to package the JavaScript for use as a Cordova application. The user can specify more than one target at a time. There is a target for SWF production, one for Royale JavaScript, Royale with Cordova Support, NodeJS output, and raw JavaScript output.

Pretty much all targets start execution the same way: They build an Abstract Syntax Tree (AST). An AST is a tree data structure with nodes that represent the parsed source code. The top of the tree is a FileNode and usually the first child is a PackageNode and children of that might be ImportNodes and a ClassNode and children of the ClassNode will be FunctionNodes that contain ParameterNodes and ExpressionNodes and so on. At the bottom of the tree should be IdentifierNodes and LiteralNodes. To create the AST, a "compilation unit" is instantiated for each source files (and for each class in a SWC as well), and the compilation unit is told to generate the AST.

What happens next is different for SWF output vs JavaScript output. In the SWF output, the JBurg "reducer" is called to reduce patterns of nodes into SWF output byte code. This reducer works from the bottom of the tree upward. This supposedly makes it easier and more efficient to identify possible optimizations such as collapsing constant expressions, and identify semantic errors. However, the syntax for describing tree patterns is complex and not very approachable to people new to the code base, so a volunteer provided an AST tree-walking 'backend' instead.

The JavaScript output also runs the reducer to help identify semantic errors. But if none are found it then walks the AST from the top FileNode down, along the way "emitting" whatever output is desired. There are emitters that just output the ASDoc into JSON files which is how we generate our ASDoc today. There is an emitter that outputs ActionScript just to prove that the AST has not lost information. There is an emitter than outputs vanilla JavaScript. But the main emitter class (JSRoyaleEmitter) outputs JavaScript along with Google Closure Library calls so that the Google Closure Compiler can optimize the output.

During reduction or the tree walk, nodes are 'resolved' into 'definitions'. If there is a variable 'foo', the compiler needs to know if it is a variable (VariableDefinition) or parameter (ParameterDefinition) or a method (FunctionDefinition). There are plenty of other Definitions not mentioned in this paragraph.

Emitter Details

This section gives a brief overview of the classes involved in an Emitter.

• MXMLJSC.java This is the main entry point for the Transpiler. In the method _mainNoExit you will see a switch statement for each target. The target case statement in turn calls a client such as MXMLJSCRoyale.
• MXMLJSCRoyale.java This client then instantiates a "backend". In this case, MXMLRoyaleBackend, checks the configuration options, then calls the compile method which then calls buildArtifact which runs the reducer, looking for semantic errors. And if there are no errors, it then calls the publisher to get an output folder, then starts going through all of the files (the reachableCompilationUnits). For each file that had ActionScript or MXML source (as opposed to having been pre-transpiled and coming from a SWC), the backend is called to create a 'writer', which is given an output stream and a compilation unit and told to write itself.
• MXMLRoyaleBackend.java This backend will create a JSWriter for ActionScript files and an MXMLWriter for MXML files. When asked by a JSWriter, it will tell the JSWriter to use as ASBlockWalker to walk the AST for an ActionScript file and use JSRoyaleEmitter as the emitter. For MXML files, this backend will tell the MXMLWriter to use MXMLRoyaleBlockWalker to walk the AST for an MXML file and use MXMLRoyaleEmitter as the emitter.
• JSWriter.java In its writeTo method, this writer tells the walker to 'visit' the compilation unit. This starts the walking of the AST. Each node will be 'visited' in a standard top-down tree walk.
• MXMLWriter.java In its writeTo method, this writer will set up a JSFilterWriter for any Script block in the MXML and tell it to use ASBlockWalker and JSRoyaleEmitter as the emitter. Then it will tell MXMLRoyaleBlockWalker to 'visit' the compilation unit.
• ASBlockWalker.java This should look like a standard tree walking algorithm with 'visit' methods for each of the nodes. The 'visit' methods generally call an emitter method.
• MXMLRoyaleBlockWalker.java This should also look like a standard tree walking algorithm with 'visit' methods for each of the nodes.
• JSRoyaleEmitter.java This emitter has methods to emit JavaScript for every node. Much of the actual output code has been encapsulated in support classes like IdentifierEmitter and LiteralEmitter to make it easier to re-use these emitters when customizing output.
• MXMLRoyaleEmitter.java This emitter has methods for each MXML AST node. However, MXML is output mostly as a data structure, and not as much JavaScript. So the methods build up that data structure.

Notes

Nobody is claiming that the architecture and implementation of the compiler is optimal. The code represents what we had time to do in order to get proofs-of-concepts up and running. Ideally, the Transpiler would do its own semantic analysis so the source wouldn't get processed by both a reducer and the tree walker. Volunteers are welcome to make that happen.

The tree walk might also benefit from other optimizations. There isn't much state retained about the parent chain of nodes but many times the emitter must walk up the parent chain to better understand what to output.

Also, identifiers might be resolved too many times instead of figuring out a smart way to cache the resolved identifier's definition or optimize the resolving if you know it should be a member of a class.