Robust software principles for device and client code

[sgbaird]

The device that runs the experiments as well as the software package that is meant to interact with the device should be robust to bad user inputs and other fringe cases that might lock up the device or cause damage.

[sgbaird]

Avoiding unrecoverable on-device errors via bad user input

• Should be able to handle a bad payload A bad MQTT payload can throw an unrecoverable exception in main.py #99
• Should give an informative exception, i.e. with filename, line no. and error message
  • This should be done using temporary filestreams (e.g. StringIO)
  • don't write to a local file in response to user input, otherwise, this might cause an out-of-memory error eventually
  • writing to an SD card should only be for optional things, like a running log of experimental details as a secondary backup to a database write backup data to SD card slot if present #95
• Should floats be rounded and converted to integers, and if so, should this occur at the microcontroller level, the Python package level, or both?
• making host and client software robust to user inputs (avoid device lock) #101
• Secure data transfer to avoid instructions from bad actors (encrypted and authenticated)
  • How can data be securely transferred? #84
  • How to differentiate between which hardware the experiment was run on? #127 (comment)
  • How to differentiate between which hardware the experiment was run on? #127 (reply in thread)

[sgbaird]

Avoiding experimental failure/hazards

• The microcontroller should verify that inputs are within excepted ranges and throw an exception otherwise (i.e. don't let it move on to code that interfaces with the equipment)
• There should be some cleanup after an error is thrown.
  • In the RGB example, this involves setting the RGB values back to (0, 0, 0)
  • The "cleanup" steps might be different depending on where in the experiment the error was thrown
  • If the cleanup step is unable to complete successfully, then it may be necessary to put everyone's experiments on hold until the proper manual intervention has been carried out
• Tracking resource-dependent conditions #156
  • e.g., using the database to track resource constraints, a separate collection with the calculated history, and backup logging to the device
  • notification when resources are projected to be running low
  • Devices that monitor resources (flowmeters, liquid level gauges, mass balances) #172
  • stopping further experiments when resources have reached a critical level

[sgbaird]

Ensure that the devices are functioning normally

• Run a GitHub actions cron that checks for signs of life from the device and sends an email and/or text if the system is unresponsive github actions cron for sending and receiving data to the publicly accessible test demo #79
• Use an auditory signal for experiment start and stop buzzer chimes for when an experiment starts or ends #96
• Blink an onboard LED every few seconds to demonstrate that the device is online
• Integrate tests into the Python package test suite
  • These can be simulated tests that use the same basic structure as the functions that call the real experiments
  • To avoid clogging up the experimental queue, tests on the experimental setup can be specified to run only on request by GitHub users with adequate permissions or when a new Python package release is being provisioned. This is similar to the build farm setup that Numba uses. In the latter case, if there are breaking changes, then the on-device software should be updated before the new release.
  • There can also be a GitHub cron (see above) that performs a complete experiment at a specified interval (e.g. every several weeks)
  • EDIT: there can also be a cron that runs more frequently if you have a way of communicating with the device and requesting a "dummy" experiment (kind of like the simulator class). While this more frequent cron doesn't guarantee the full experiment will run, it will at least help out in quickly identifying other issues that might come up like network connection issues or forgetting to change a particular line (e.g. hardcoding "test" as the PICO_ID for the public demo) Enable requesting a dummy experiment from the device #155
• Use the experiments yourself for real applications
• Check that no extraneous parameters are passed (e.g. if default is getting used due to bad key)

[sgbaird]

Minimizing the amount of code that lives on the device

• from @rekumar How to differentiate between which hardware the experiment was run on? #127 (reply in thread):

  In my mind all access control/input checking/etc should be handled at the application level. As long as your hardware only accepts instructions from your application, this should be fine. Not hard to achieve this isolation using an encrypted handshake or an internal (or even dedicated) LAN for communication. The minor risk is that a developer could change the input checking step, though this seems reckless and unlikely. I'd even argue that moving hardware limits to the application level can improve transparency + avoid duplication (ie repeat code at API and hardware levels). However the riskiest cases are probably easy to catch on hardware without something like pydantic (ie if temp>500 raise TooHotError) Additionally, you can imagine limits that involve multiple steps of the SDL (ie temperature limits change based on your solvent's boiling point), which have to be caught at the application level.

  Anecdotally, all beamlines I've used as an external user rely on (often easily bypassed) software limits to avoid instrument collisions etc, and they seem to function well enough.

• I think the codebase could use a refactor so that the device-side code is more minimalistic. However, an outstanding issue is still getting that secure handshake for sending the device instructions How to differentiate between which hardware the experiment was run on? #127 (reply in thread). Using MongoDB as the queue system (suggestion by @rekumar) would simplify this, but requires continuous "listening" to the database by the Pico W (EDIT: maybe not a problem for MongoDB Atlas free tier shared cluster). Maybe a capped collection would work for this (stackoverflow). The amount of data transfer could be minimized by deleting entries on the collection once the experiment has been run. EDIT: figured out secure MQTT (using free instance of HiveMQ)
• "The data is precious", so maybe redundancy between device and client in logging results to MongoDB Atlas is an exception Robust software principles for device and client code #100 (comment)

[sgbaird]

Preserving Data

• Write data backup to an SD card Best practices for backup data logging #122
• Device (tries to) log to the database Persistent and accessible storage of data #117
• Client logs to the database if _experiment_id is not present in the database - perhaps with an extra flag to indicate it's the client doing the uploading rather than the device
• Throw an error on the client if neither the device nor the client can log to the database MongoDB logging stopped working on the device after some time #157