Refactor e2e & Joules Plugin (#799)

* initial files add

* joules save/reload checkpoints & many other tweaks

* bump power tool versions

* fixed design name

* fixing sim/power clock periods

* tool fixes and config improvements

* removing redundant stuff

* more tweaks to pass gate level sims

* Update README.md

* renaming tech to pdk

* post syn and par power

* fixed env configs

* Update README.md

* Update README.md

* Update Makefile

* minor tweaks

* minor tweaks

* minor changes

* Update README.md

* tweak

* small fix

* renamed test to pass example

* many fixes, removing redundant keys

* minor tweaks

* improved report configs explanations

* adding to hammer docs

* Update Hammer-Overview.rst

* adding power tool-supplied persistent hook

* changing default toolflow from commercial to nop

* fixing tool version comments

* fixing typo

doc/Examples/e2e.rst

@@ -0,0 +1 @@
+../../e2e/README.md

doc/Hammer-Basics/Hammer-Overview.rst

@@ -77,13 +77,16 @@ Using hooks requires the designer to extend the ``CLIDriver`` class. A good exam
 
     example-vlsi -e env.yml -p config.yml --obj_dir build par
 
-In both of these commands, an environment configuration is passed to Hammer using a ``-e`` flag, which in this case is ``env.yml``.
-``env.yml`` contains pointers to the required tool licenses and environment variables.
-These environment settings will not be propagated to the output configuration files after each action.
-
-Any number of other YML or JSON files can then be passed in using the ``-p`` flag.
+Hammer configuration files must be in YML or JSON format, 
+and are divided into environment and project configurations.
+The environment configuration file, which in this case is ``env.yml``, is passed to Hammer using the ``-e`` flag.
+``env.yml`` contains pointers to the required tool licenses and environment variables. 
+The project configuration file is passed in using the ``-p`` flag.
 In this case, there is only one, ``config.yml``, and it needs to set all the required keys for the step of the flow being run.
 Passing in multiple files looks like ``-p config1.yml -p config2.yml``. Refer to the :ref:`config` section for the implications of multiple config files.
+The environment settings take precedence over all project configurations, and are not propagated to the output configuration files after each action.
+The order of precedence for the project configs reads from right to left (i.e. each file overrides all files to its left in the command line).
+
 
 ``--obj_dir build`` designates what directory Hammer should use as a working directory.
 All default action run directories and output files will be placed here.

e2e/Makefile

@@ -1 +1,45 @@
-include obj_dir/hammer.d
+vlsi_dir=$(abspath .)
+
+
+
+# minimal flow configuration variables
+design              ?= pass
+pdk                 ?= sky130
+tools               ?= nop
+env                 ?= bwrc
+
+extra               ?=  # extra configs
+args                ?=  # command-line args (including step flow control)
+
+
+OBJ_DIR             ?= $(vlsi_dir)/build-$(pdk)-$(tools)/$(design)
+
+# non-overlapping default configs
+ENV_YML             ?= configs-env/$(env)-env.yml
+PDK_CONF            ?= configs-pdk/$(pdk).yml
+TOOLS_CONF          ?= configs-tool/$(tools).yml
+
+# design-specific overrides of default configs
+DESIGN_CONF         ?= configs-design/$(design)/common.yml
+DESIGN_PDK_CONF     ?= configs-design/$(design)/$(pdk).yml
+SIM_CONF            ?= $(if $(findstring rtl,$(MAKECMDGOALS)), configs-design/$(design)/sim-rtl.yml, \
+                       $(if $(findstring syn,$(MAKECMDGOALS)), configs-design/$(design)/sim-syn.yml, \
+                       $(if $(findstring par,$(MAKECMDGOALS)), configs-design/$(design)/sim-par.yml, )))
+POWER_CONF          ?= $(if $(findstring power-rtl,$(MAKECMDGOALS)), configs-design/$(design)/power-rtl-$(pdk).yml, \
+                       $(if $(findstring power-syn,$(MAKECMDGOALS)), configs-design/$(design)/power-syn-$(pdk).yml, \
+                       $(if $(findstring power-par,$(MAKECMDGOALS)), configs-design/$(design)/power-par-$(pdk).yml, )))
+
+PROJ_YMLS           ?= $(PDK_CONF) $(TOOLS_CONF) $(DESIGN_CONF) $(DESIGN_PDK_CONF) $(SIM_CONF) $(POWER_CONF) $(extra)
+HAMMER_EXTRA_ARGS   ?= $(foreach conf, $(PROJ_YMLS), -p $(conf)) $(args)
+
+
+
+
+HAMMER_D_MK = $(OBJ_DIR)/hammer.d
+
+build: $(HAMMER_D_MK)
+
+$(HAMMER_D_MK):
+	hammer-vlsi --obj_dir $(OBJ_DIR) -e $(ENV_YML) $(HAMMER_EXTRA_ARGS) build
+
+-include $(HAMMER_D_MK)

e2e/README.md

@@ -11,50 +11,143 @@ poetry install
 poetry shell
 ```
 
-We provide configs for Berkeley EECS compute nodes: BWRC (`-bwrc`), Millennium (`-a`), and instructional machines (`-inst`).
+## Overview
 
-- Environment configs (commercial CAD tool paths and license servers) are in `env`
-- PDK configs for ASAP7 and sky130 (pointers to PDK paths and CAD tool versions) are in `pdks`
-- The common design config (which CAD tool plugins to use and the design input files) in `configs/common.yml`
-- The PDK specific design config (clock, placement, and pin constraints, and which SRAM generator to use) in `configs/{asap7,sky130}.yml`
+
+### Flow Selection
+
+The following variables in the Makefile select the target flow to run:
+
+- `design` - RTL name
+    - {`pass`, `gcd`}
+- `pdk` - PDK name
+    - {`sky130`, `asap7`}
+- `tools` - CAD tool flow
+    - {`cm` (commercial), `or` (OpenROAD)}
+- `env` - compute environment
+    - {`bwrc` (BWRC), `a` (Millenium), `inst` (instructional machines)}
+
+The outputs of the flow by default reside in `OBJ_DIR=build-<pdk>-<tools>/<design>/`
+
+### Configs
+
+The Hammer configuration files consist of environment (`ENV_YML`) and project (`PROJ_YMLS`) configurations.
+The environment configs take precedence over ALL project configs.
+The order of precedence for the project configs reads from right to left (i.e. each file overrides all files to its left).
+All configuration files are summarized below.
+
+```shell
+#         lowest precedence -------------------------------------------> highest precendence
+CONFS ?= $(PDK_CONF) $(TOOLS_CONF) $(DESIGN_CONF) $(DESIGN_PDK_CONF) $(SIM_CONF) $(POWER_CONF)
+```
+
+- `ENV_YML`- Environment configs that specify CAD tool license servers and paths are in `configs-env`.
+  This will take precedence over any other config file
+- `PDK_CONF` - PDK configs shared across all runs with this PDK are in `configs-pdk`
+- `TOOLS_CONF` - Tool configs to select which CAD tool flow to use are in `configs-tools`
+- Design-specific configs are located in `configs-design/<design>`, and are summarized below:
+    - `DESIGN_CONF` - the common design config (design input files, anything else design-specific)
+    - `DESIGN_PDK_CONF` - PDK-specific configs for this particular design (clock, placement, pin constraints)
+    - `SIM_CONF` - Simulation configs for post-RTL, Synthesis, or PnR simulation
+    - `POWER_CONF` - Power simulation configs for post-RTL, Synthesis, or PnR simulation
+      (NOTE: The Makefile expects the power config filename for each simulation level + PDK to be in the format `power-{rtl,syn,par}-<pdk>.yml`,
+      while the `joules.yml` and `voltus.yml` files serve as templates for the Cadence Joules/Voltus power tools)
+
+
+## Run the Flow
 
 First, use Hammer to construct a Makefile fragment with targets for all parts of the RTL -> GDS flow.
-Specify the configs according to which PDK and environment you are using.
+Specify the appropriate `env/tools/pdk/design` variables to select which configs will be used.
 
 ```shell
-hammer-vlsi -e env/a-env.yml -p pdks/asap7-a.yml -p configs/common.yml -p configs/asap7.yml build
+make build
+# same as: `make env=bwrc tools=cm pdk=sky130 design=pass build`
 ```
 
-Hammer will generate a Makefile fragment in `obj_dir/hammer.d`.
+Hammer will generate a Makefile fragment in `OBJ_DIR/hammer.d`.
+
+Then run the rest of the flow, making sure to set the `env/tools/pdk/design` variables as needed:
+
+```shell
+make sim-rtl
+make power-rtl
+
+make syn
+make sim-syn
+make power-syn
+
+make par
+make sim-par
+make power-par
+
+make drc
+make lvs
+```
 
-### Run the Flow
+These actions are summarized in more detail:
 
 - RTL simulation
-    - `make sim-rtl HAMMER_EXTRA_ARGS="-p configs/sim.yml"`
-    - Generated waveform in `obj_dir/sim-rtl-rundir/output.fsdb`
+    - `make sim-rtl`
+    - Generated waveform in `OBJ_DIR/sim-rtl-rundir/output.fsdb`
+- Post-RTL Power simulation
+    - `make sim-rtl-to-power`
+    - `make power-rtl`
+    - Generated power reports in `OBJ_DIR/power-rtl-rundir/reports`
 - Synthesis
     - `make syn`
-    - Gate-level netlist in `obj_dir/syn-rundir/pass.mapped.v`
+    - Gate-level netlist in `OBJ_DIR/syn-rundir/<design>.mapped.v`
 - Post-Synthesis simulation
-    - `make syn-to-sim HAMMER_EXTRA_ARGS="-p configs/syn-sim.yml"`
-    - `make sim-syn HAMMER_EXTRA_ARGS="-p configs/syn-sim.yml"`
-    - Generated waveform and register forcing ucli script in `obj_dir/sim-syn-rundir`
+    - `make syn-to-sim`
+    - `make sim-syn`
+    - Generated waveform and register forcing ucli script in `OBJ_DIR/sim-syn-rundir`
+- Post-Synthesis Power simulation
+    - `make syn-to-power`
+    - `make sim-syn-to-power`
+    - `make power-syn`
+    - Generated power reports in `OBJ_DIR/power-syn-rundir/reports`
 - PnR
     - `make syn-to-par`
     - `make par`
-    - LVS netlist (`pass.lvs.v`) and GDS (`pass.gds`) in `obj_dir/par-rundir`
+    - LVS netlist (`<design>.lvs.v`) and GDS (`<design>.gds`) in `OBJ_DIR/par-rundir`
 - Post-PnR simulation
-    - `make par-to-sim HAMMER_EXTRA_ARGS="-p configs/par-sim.yml"`
-    - `make sim-par HAMMER_EXTRA_ARGS="-p configs/par-sim.yml"`
+    - `make par-to-sim`
+    - `make sim-par`
+- Post-PnR Power simulation
+    - `make par-to-power`
+    - `make sim-par-to-power`
+    - `make power-par`
+    - Generated power reports in `OBJ_DIR/power-par-rundir`
 
-### Custom Setups
+### Flow Customization
+
+If at any point you would like to use custom config files (that will override any previous configs), assign the `extra` Make variable to a space-separated list of these files.
+For example, to run the `pass` design with `sky130` through the commercial flow, but run LVS with Cadence Pegasus instead of the default Siemens Calibre,
+simply run the following:
+
+```shell
+make extra="configs-tool/pegasus.yml" build
+```
+
+To use the [Hammer step flow control](https://hammer-vlsi.readthedocs.io/en/stable/Hammer-Use/Flow-Control.html), prepend `redo-` to any VLSI flow action,
+then assign the `args` Make variable to the appropriate Hammer command line args.
+For example, to only run the `report_power` step of the `power-rtl` action (i.e. bypass synthesis), run the following:
+
+```shell
+make args="--only_step report_power" redo-power-rtl
+```
+
+## Custom Setups
 
 If you're not using a Berkeley EECS compute node, you can create your own environment setup.
 
-- Create an environment config for your node to specify the location of the CAD tools, modeled after the yaml files in `env`
-- Create a PDK config for your node to specify the PDK paths and versions, modeled after the yaml files in `pdks`
-- Point to your custom configs when running `hammer-vlsi`. The rest of the flow should be identical
+- Create an environment config similar to those in `configs-env` for your node to specify the CAD tool license servers and paths/versions of CAD tools and the PDK, and set the `ENV_YML` variable to this file.
+- The rest of the flow should be identical
+
+### ASAP7 Install
+
+Clone the [asap7 repo](https://github.com/The-OpenROAD-Project/asap7) somewhere and reference the path in your `ENV_YML` config.
 
-#### ASAP7 Install
+### Sky130 Install
 
-Clone the [asap7 repo](https://github.com/The-OpenROAD-Project/asap7) somewhere and reference the path in your PDK yaml config.
+Refer to the [Hammer Sky130 plugin README](https://hammer-vlsi.readthedocs.io/en/stable/Technology/Sky130.html)
+to install the Sky130 PDK, then reference the path in your `ENV_YML` config (only the `technology.sky130.sky130A` key is required).

e2e/configs-design/gcd/asap7.yml

@@ -0,0 +1,35 @@
+# Specify clock signals
+vlsi.inputs.clocks: [
+  {name: "clk", period: "5ns", uncertainty: "0.1ns"}
+]
+
+sim.inputs:
+  defines: ["CLOCK_PERIOD=5"]
+  defines_meta: "append"
+  options: ["-timescale=1ns/10ps"]
+  options_meta: append
+
+# Placement Constraints
+vlsi.inputs.placement_constraints:
+  - path: "gcd"
+    type: toplevel
+    x: 0
+    y: 0
+    width: 100
+    height: 100
+    margins:
+      left: 0
+      right: 0
+      top: 0
+      bottom: 0
+
+vlsi.inputs.delays: [
+  {name: "reset",           clock: "clk", delay: "1", direction: "input"},
+  {name: "operands_bits_A", clock: "clk", delay: "1", direction: "input"},
+  {name: "operands_bits_B", clock: "clk", delay: "1", direction: "input"},
+  {name: "operands_val",    clock: "clk", delay: "1", direction: "input"},
+  {name: "operands_rdy",    clock: "clk", delay: "1", direction: "output"},
+  {name: "result_bits_data",clock: "clk", delay: "1", direction: "output"},
+  {name: "result_val",      clock: "clk", delay: "1", direction: "output"},
+  {name: "result_rdy",      clock: "clk", delay: "1", direction: "input"} 
+]

e2e/configs-design/gcd/common.yml

@@ -0,0 +1,19 @@
+# Generate Make include to aid in tlow
+vlsi.core.build_system: make
+
+vlsi.inputs.power_spec_type: "cpf"
+vlsi.inputs.power_spec_mode: "auto"
+
+synthesis.inputs:
+  top_module: "gcd"
+  input_files: ["src/gcd.v"]
+
+sim.inputs:
+  top_module: "gcd"
+  tb_name: "gcd_tb"
+  tb_dut: "gcd_dut"
+
+power.inputs:
+  top_module: "gcd"
+  tb_name: "gcd_tb"
+  tb_dut: "gcd_dut"

e2e/configs-design/gcd/joules.yml

@@ -0,0 +1,6 @@
+# Test Joules Flow
+
+vlsi.core.power_tool: "hammer.power.joules"
+
+power.inputs.input_files: ["/path/to/pass.v"]
+power.inputs.waveforms: ["path/to/waveform"]

e2e/configs-design/gcd/power-rtl-sky130.yml

@@ -0,0 +1,31 @@
+# Test Joules Flow
+
+vlsi.core.power_tool: "hammer.power.joules"
+
+power.inputs:
+  level: "rtl"
+  input_files: ["src/gcd.v"]
+  waveforms: ["build-sky130-cm/gcd/sim-rtl-rundir/output.fsdb"]
+  report_configs:
+    - waveform_path: "build-sky130-cm/gcd/sim-rtl-rundir/output.fsdb"
+      levels: 1
+      toggle_signal: /gcd/clk
+      num_toggles: 1
+      report_name: levels-1
+      output_formats:
+      - all
+    # the following two should produce equivalent results:
+    - waveform_path: "build-sky130-cm/gcd/sim-rtl-rundir/output.fsdb"
+      inst: /gcd/GCDctrl0
+      toggle_signal: /gcd/clk
+      num_toggles: 1
+      report_name: inst-GCDctrl0
+      output_formats:
+      - all
+    - waveform_path: "build-sky130-cm/gcd/sim-rtl-rundir/output.fsdb"
+      module: gcd_control
+      interval_size: 5ns
+      report_name: module-gcd_control
+      output_formats:
+      - all
+

e2e/configs-design/gcd/sim-par.yml

@@ -0,0 +1,7 @@
+# Post-PAR Simulation
+
+sim.inputs:
+  level: "par"
+  input_files: ["src/gcd_tb.v"]
+  input_files_meta: append
+  timing_annotated: true

e2e/configs-design/gcd/sim-rtl.yml

@@ -0,0 +1,7 @@
+# RTL Simulation
+
+sim.inputs:
+  level: "rtl"
+  input_files: ["src/gcd.v", "src/gcd_tb.v"]
+
+# sim.inputs.execute_sim: false

e2e/configs-design/gcd/sim-syn.yml

@@ -0,0 +1,8 @@
+# Post-Synthesis Simulation
+
+sim.inputs:
+  level: "syn"
+  input_files: ["src/gcd_tb.v"]
+  input_files_meta: append
+  timing_annotated: true
+