Spike: Comprehensive Manifest File Version Handling

[MattToast]

Description

Currently, the way SmartSim and SmartDashboard inter-op is:

• A user writes a driver script that calls Experiment.start
• SmartSim creates a {experiment_path}/.smartsim directory
• SmartSim will save information about the launched entities in a manifest.json file under the .smartsim directory
• SmartSim will collect additional information about a run running entities (start times, stop times, etc.) and save that info additional sub-directories and files under the .smartsim directory
• SmartDashboard will be invoked by the user (either during or after the execution of the driver script) and look for the manifest.json file
• SmartDashboard will parse the file and build and display the GUI

In order to make sure that the manifest.json file that is output by SmartSim and understandable by SmartDashboard, SmartSim adds some metadata about the file, including a version number the schema of manifest. SmartDashboard then checks the version of the file, and reports if it cannot parse the file to the user. Currently that is done with this block in smartdashboard.utils.ManifestReader.

if version not in ("0.0.2", "0.0.3"):
    version_exception = Exception(
        "SmartDashboard version 0.0.3 is unable to parse manifest "
        f"file at version {version}."
    )

As can be observed, the manifest reader is currently just checking to see if the version field of the manifest.json is some string that it cannot recognize. We should consider upgrading to a comprehensive version check.

Justification

One of the main problems with this is that users cannot use an older versions of the dashboard with manifests that come from newer versions of SmartSim when they add new information to the manifest.json EVEN WHEN THOSE CHANGES ARE NON-BREAKING as the patch/minor release bump in the manifest.json means that the version number will not be in the list of "recognized" version strings in the dashboard. This effectively means that every time a user upgrade SmartSim, they must also upgrade SmartDashboard even though the older version could still parse information in the manifest (albeit, none of the new information).

Furthermore, as SmartSim begins outputting more and more information about a job, it is becoming more and more likely that a change will need to be made that is non-addative. When this needs to be done, SmartSim will start providing more complex version numbers, and it is unlikely that SmartDashboard will be able effectively maintain its list of "recognized" version strings. It would be far more effective if the dashboard could specify a range of supported versions.

Implementation Strategy

The simplest possible way to allow the dashboard to start checking if a version number in a manifest.json is within a range of numbers is to simply have the ManifestFileReader do a little bit of parsing of the version string on its own to coerce the version into a tuple of ints that can then be checked if it is within some range

if (0, 0, 2) <= tuple(int(i) for i in version.split('.')) < (1, 0 ,0):
    version_exception = Exception(...)

This may be effective enough, but does limit the version number of the manifest to simple integers (e.g. we cannot do 1.0.0rc2, 0.5.0-alpha, etc.). Its unclear whether or not we need this functionality, so this aspect should be explored.

Another option, of course, would be to simply bring in a third party dependency to handle this for us (although I am not aware of one offhand, suggestions welcome below!!).

The ticket will look at other repositories and strategies for handling versioning of external contracts/artifacts and present options to the dev group.