YAML Configuration Files (#551)

## Misc. Bugfixes/Enhancements

* `openlane.config.Config`
* YAML 1.2 configuration files now accepted using `.yaml` or `.yml`
extensions, with the same featureset as JSON files.
* The first configuration (file/dict) supplied no longer needs to be a
complete configuration so long as any required variables are supplied in
later configurations. Missing variables are only checked on the complete
configuration.
* Internally reworked how config files and command-line overrides are
parsed.
 
## Documentation

* Added info on YAML configuration files.

.coveragerc

@@ -22,6 +22,7 @@ omit =
     */steps/netgen.py
     */steps/odb.py
     */steps/openroad.py
+    */steps/pyosys.py
     */steps/yosys.py
     */steps/verilator.py
     */steps/cvc_rv.py

default.nix

@@ -42,6 +42,7 @@
   click,
   cloup,
   pyyaml,
+  yamlcore,
   rich,
   requests,
   pcpp,
@@ -102,6 +103,7 @@
         click
         cloup
         pyyaml
+        yamlcore
         rich
         requests
         pcpp

docs/source/reference/configuration.md

@@ -2,16 +2,16 @@
 
 Unless the design uses the API directly, each OpenLane-compatible design must
 come with a configuration file. These configuration files can be written in one
-of two grammars: JSON or Tcl.
+of three grammars: JSON, YAML or Tcl.
 
-Tcl offers more flexibility at the detriment of security, while JSON is more
-straightforward at the cost of flexibility. While Tcl allows you to do all
-manner of computation on your variables, JSON has a limited expression engine
-that will be detailed later in this document. Nevertheless, for security (and
-future-proofing), we recommend you use either the JSON format or write Python
-scripts using the API.
+Tcl offers more flexibility at the detriment of security, while YAML/JSON are
+more straightforward at the cost of flexibility. While Tcl allows you to do all
+manner of computation on your variables, YAML/JSON have a limited expression
+engine that will be detailed later in this document. Nevertheless, for security
+(and future-proofing), we recommend you use either the YAML/JSON format or write
+Python scripts using the API.
 
-The folder containing your `config.tcl`/`config.json` is known as the **Design
+The folder containing your `config.{tcl/yml/yaml/json}` is known as the **Design
 Directory** -- though the design directory can also be set explicitly over the
 command-line using `--design-dir`. The design directory is special in that paths
 in the JSON configuration files can be resolved relative to this directory and
@@ -24,17 +24,17 @@ enabling you to do complex pre-processing far beyond the capacity of either
 JSON or Tcl files. You can still use `ref::` and such like JSON files though.
 ```
 
-## JSON
+## JSON/YAML
 
-The JSON files are simple key-value pairs.
+The JSON and YAML files are simple key-value pairs.
 
 <a name="scalars"></a>
 
 The values can be scalars (strings, numbers, Booleans, and `null`s), lists or
 dictionaries, subject to validation.
 
-All files must be ECMA404-compliant, i.e., pure JSON with no extensions such as
-comments or the new elements introduced in [JSON5](https://json5.org/).
+All JSON files must be ECMA404-compliant, i.e., pure JSON with no extensions
+such as comments or the new elements introduced in [JSON5](https://json5.org/).
 
 An minimal demonstrative configuration file would look as follows:
 
@@ -49,23 +49,43 @@ An minimal demonstrative configuration file would look as follows:
     "FP_CORE_UTIL": 40,
     "PL_TARGET_DENSITY_PCT": "expr::($FP_CORE_UTIL + 10.0)",
     "scl::sky130_fd_sc_hd": {
-      "CLOCK_PERIOD": 15
+      "CLOCK_PERIOD": 15.0
     }
   }
 }
 ```
 
+All YAML files must conform to the
+[YAML 1.2 specification](https://yaml.org/spec/1.2.2/). Scalar types are deduced
+according to the
+[YAML 1.2 Core Schema](https://yaml.org/spec/1.2.2/#103-core-schema).
+
+An minimal demonstrative configuration file would look as follows:
+
+```yml
+DESIGN_NAME: spm
+VERILOG_FILES: dir::src/*.v
+CLOCK_PORT: clk
+CLOCK_PERIOD: 100
+pdk::sky130A:
+  MAX_FANOUT_CONSTRAINT: 6
+  FP_CORE_UTIL: 40
+  PL_TARGET_DENSITY_PCT: expr::($FP_CORE_UTIL + 10.0)
+  scl::sky130_fd_sc_hd:
+    CLOCK_PERIOD: 15.0
+```
+
 ### Pre-processing
 
-The JSON files are pre-processed at runtime. Features include conditional
+JSON/YAML files are pre-processed at runtime. Features include conditional
 execution, a way to reference the design directory, other variables, and a basic
 numeric expression engine.
 
 #### Conditional Execution
 
-The JSON configuration files support conditional execution based on PDK or
-standard cell library (or, by nesting as shown above, a combination thereof.)
-You can do this using the `pdk::` or `scl::` key prefixes.
+The configuration files support conditional execution based on PDK or standard
+cell library (or, by nesting as shown above, a combination thereof.) You can do
+this using the `pdk::` or `scl::` key prefixes.
 
 The value for this key would be a `dict` that is only evaluated if the PDK or
 SCL matches those in the key, i.e., for `pdk::sky130A` as shown above, this
@@ -135,9 +155,9 @@ reference a variable that is declared after the current expression.
 > it is declared, but the latter is OK, where the value will be "vdd gnd" as
 > well.
 
-Do note that unlike Tcl config files, environment variables (other than
-`DESIGN_DIR`, `PDK`, `PDKPATH`, `STD_CELL_LIBRARY`) are not exposed to
-`config.json` by default.
+Do note that unlike Tcl config files, environment variables are not exposed to
+`config.{yml/yaml/json}` by default. You only have access to four specific
+variables: `DESIGN_DIR`, `PDK`, `PDKPATH`, and `STD_CELL_LIBRARY`.
 
 If the files you choose lie **inside** the design directory, a different prefix,
 `refg::`, supports non-recursive globs, i.e., you can use an asterisk as a
@@ -150,8 +170,8 @@ wildcard to pick multiple files in a specific folder.
   * If no elements were found, the glob string is returned verbatim as a single
     element in array.
 
-As shown below, `refg::$DESIGN_DIR/src/*.v` would find all files ending with `.v`
-in the `src` folder inside the design directory.
+As shown below, `refg::$DESIGN_DIR/src/*.v` would find all files ending with
+`.v` in the `src` folder inside the design directory.
 
 ```json
 {

flake.nix

@@ -82,7 +82,8 @@
               };
               sphinx-tippy = callPythonPackage ./nix/sphinx-tippy.nix {};
               sphinx-subfigure = callPythonPackage ./nix/sphinx-subfigure.nix {};
-
+              yamlcore = callPythonPackage ./nix/yamlcore.nix {};
+
               # ---
               openlane = callPythonPackage ./default.nix {};
               default = self.openlane;

nix/yamlcore.nix

@@ -0,0 +1,46 @@
+# Copyright 2024 Efabless Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+{
+  lib,
+  fetchurl,
+  buildPythonPackage,
+  fetchPypi,
+  pyyaml,
+  setuptools,
+  version ? "0.0.2",
+  sha256 ? "sha256-iy65VBy+Fq+XsSy9w2+rkqjG9Y/ImL1oZR6Vnn2Okm8=",
+}: let
+  self = buildPythonPackage {
+    pname = "yamlcore";
+    inherit version;
+    format = "pyproject";
+
+    src = fetchPypi {
+      inherit (self) pname version;
+      inherit sha256;
+    };
+
+    nativeBuildInputs = [
+      setuptools
+    ];
+
+    meta = with lib; {
+      description = "YAML 1.2 Core Schema Support for PyYAML";
+      homepage = "https://github.com/perlpunk/pyyaml-core";
+      license = licenses.mit;
+      inherit (pyyaml.meta) platforms;
+    };
+  };
+in
+  self

openlane/__main__.py

@@ -13,6 +13,7 @@
 # limitations under the License.
 import os
 import sys
+import glob
 import shutil
 import marshal
 import tempfile
@@ -49,8 +50,8 @@
 from . import common
 from .container import run_in_container
 from .plugins import discovered_plugins
-from .config import Config, InvalidConfig, PassedDirectoryError
 from .common.cli import formatter_settings
+from .config import Config, InvalidConfig, PassedDirectoryError
 from .flows import Flow, SequentialFlow, FlowException, FlowError, cloup_flow_opts
 
 
@@ -257,7 +258,7 @@ def run_included_example(
         if os.name == "posix":
             subprocess.check_call(["chmod", "-R", "755", final_path])
 
-        config_file = os.path.join(final_path, "config.json")
+        config_file = glob.glob(os.path.join(final_path, "config.*"))[0]
 
         # 3. Run
         run(

openlane/common/__init__.py

@@ -47,6 +47,7 @@
 )
 from .types import (
     is_number,
+    is_real_number,
     is_string,
     Number,
     Path,