Persistent and accessible storage of data

[sgbaird]

One idea I've had is to create a notebook that does a walk-through of setting up a free SQL database (e.g. https://bit.io/) and using Ax's SQL backend for storing experiments. I think it would also be informative to have similar notebooks focused on storing experimental data to other platforms: MongoDB Atlas, Foundry, Figshare or Zenodo, and Matminer.

EDIT: See https://github.com/cloudcommunity/Cloud-Free-Tier-Comparison for info on the free-tiers of different cloud services.

Related:

• How can data be securely transferred? #84
• What does access restriction look like for a scaled up cloud-accessible lab? #91
• Best practices for backup data logging #122

@blaiszik curious to hear your thoughts.

EDIT: I've been using MongoDB with the shared tier. It gets pretty expensive moving to a serverless option if using find_one without an index https://qr.ae/prMzyj, so it would likely be much better to avoid serverless and just choose the appropriate shared or dedicated cluster options.

EDIT: DynamoDB is another option. See comparisons/limitations at https://chat.openai.com/share/ee16a785-e09d-4fa6-80f4-583690a4c514 (in particular, max 400 kB vs. 16 MB per document, though much more data storage allowed (25 GB vs. 500 MB)

[sgbaird]

A relevant snippet from the NOMAD JOSS paper (see paper review for link):

Materials scientists require effective solutions for managing their research data, but they cannot and should not develop their own individual solutions.

Some options:

• NOMAD

  • NOMAD is a free web-service that lets you share your data or use comprehensive data that others provide. You can use NOMAD to organize, analyze, share, and publish your materials science data, as well as explore, download, and analyze your colleagues' data.

  • The Novel Materials Discovery (NOMAD) Laboratory maintains the largest worldwide repository for input and output files of all important computational materials science computer programs. Based on its open-access data, it builds several Big-Data Services helping to advance materials science and engineering by recognizing that data is a crucial raw material of the 21st century.

• Foundry by Globus

  • Here, we introduce Foundry, a simple way to publish and discover datasets for machine learning and to link these datasets to predictive models. Foundry is a synthesis of service capabilities from the Materials Data Facility and DLHub, layered with Python tooling, standardized metadata, and a file structure specification to meet the needs of the machine learning community.

• MPContribs

  • MPContribs provides a platform and advanced programming interface (API) to contribute computational as well as experimental data to Materials Project. Data on MPContribs is collectively maintained as annotations to existing MP materials (or formulas and chemical systems), and automatically exposed to over 200,000 MP users.

• NOMAD OASIS (needs to be hosted somewhere)

  • Originally, NOMAD Central Repository is a service run at Max-Planck's computing facility in Garching, Germany. However, the NOMAD software is Open-Source, and everybody can run it. Any service that uses NOMAD software independently is called a NOMAD OASIS. A NOMAD OASIS does not need to be fully isolated. For example, you can publish uploads from your OASIS to the central NOMAD installation.