diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 30b9a2ab..39e998e3 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,5 +1,9 @@ -* [ ] ❗ I have followed the [Contributing to DVCLive](https://github.com/iterative/dvclive/blob/master/CONTRIBUTING.md) guide. +- \[ \] ❗ I have followed the + [Contributing to DVCLive](https://github.com/iterative/dvclive/blob/master/CONTRIBUTING.md) + guide. -* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/iterative/dvc.org) and linked it here. +- \[ \] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I + have created a separate PR (or issue, at least) in + [dvc.org](https://github.com/iterative/dvc.org) and linked it here. Thank you for the contribution - we'll try to review it as soon as possible. 🙏 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fba0267e..ea024b63 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -37,3 +37,8 @@ repos: rev: 'v0.0.261' hooks: - id: ruff + - repo: https://github.com/executablebooks/mdformat + rev: 0.7.16 + hooks: + - id: mdformat + args: ["--wrap=80"] diff --git a/README.md b/README.md new file mode 100644 index 00000000..da161b7c --- /dev/null +++ b/README.md @@ -0,0 +1,164 @@ +# DVCLive + +[![PyPI](https://img.shields.io/pypi/v/dvclive.svg)](https://pypi.org/project/dvclive/) +[![Status](https://img.shields.io/pypi/status/dvclive.svg)](https://pypi.org/project/dvclive/) +[![Python Version](https://img.shields.io/pypi/pyversions/dvclive)](https://pypi.org/project/dvclive) +[![License](https://img.shields.io/pypi/l/dvclive)](https://opensource.org/licenses/Apache-2.0) + +[![Tests](https://github.com/iterative/dvclive/workflows/Tests/badge.svg?branch=main)](https://github.com/iterative/dvclive/actions?workflow=Tests) +[![Codecov](https://codecov.io/gh/iterative/dvclive/branch/main/graph/badge.svg)](https://app.codecov.io/gh/iterative/dvclive) +[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit) +[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) + +DVCLive is a Python library for logging machine learning metrics and other +metadata in simple file formats, which is fully compatible with DVC. + +# [Documentation](https://dvc.org/doc/dvclive) + +- [Get Started](https://dvc.org/doc/start/experiments) +- [How it Works](https://dvc.org/doc/dvclive/how-it-works) +- [API Reference](https://dvc.org/doc/dvclive/live) +- [Integrations](https://dvc.org/doc/dvclive/ml-frameworks) + +______________________________________________________________________ + +# Quickstart + +## Install *dvclive* + +```console +$ pip install dvclive +``` + +## Initialize DVC Repository + +```console +$ git init +$ dvc init +$ git commit -m "DVC init" +``` + +## Example code + +Copy the snippet below as a basic example of the API usage: + +```python +# train.py +import random +import sys +from dvclive import Live + +with Live(save_dvc_exp=True) as live: + epochs = int(sys.argv[1]) + live.log_param("epochs", epochs) + for epoch in range(epochs): + live.log_metric("train/accuracy", epoch + random.random()) + live.log_metric("train/loss", epochs - epoch - random.random()) + live.log_metric("val/accuracy",epoch + random.random() ) + live.log_metric("val/loss", epochs - epoch - random.random()) + live.next_step() +``` + +See [Integrations](https://dvc.org/doc/dvclive/ml-frameworks) for examples using +DVCLive alongside different ML Frameworks. + +## Running + +Run couple of times passing different values: + +```console +$ python train.py 5 +$ python train.py 5 +$ python train.py 7 +``` + +## Comparing + +DVCLive outputs can be rendered in different ways: + +### DVC CLI + +You can use [dvc exp show](https://dvc.org/doc/command-reference/exp/show) and +[dvc plots](https://dvc.org/doc/command-reference/plots) to compare and +visualize metrics, parameters and plots across experiments: + +```console +$ dvc exp show +``` + +``` +───────────────────────────────────────────────────────────────────────────────────────────────────────────── +Experiment Created train.accuracy train.loss val.accuracy val.loss step epochs +───────────────────────────────────────────────────────────────────────────────────────────────────────────── +workspace - 6.0109 0.23311 6.062 0.24321 6 7 +master 08:50 PM - - - - - - +├── 4475845 [aulic-chiv] 08:56 PM 6.0109 0.23311 6.062 0.24321 6 7 +├── 7d4cef7 [yarer-tods] 08:56 PM 4.8551 0.82012 4.5555 0.033533 4 5 +└── d503f8e [curst-chad] 08:56 PM 4.9768 0.070585 4.0773 0.46639 4 5 +───────────────────────────────────────────────────────────────────────────────────────────────────────────── +``` + +```console +$ dvc plots diff $(dvc exp list --names-only) --open +``` + +![dvc plots diff](./docs/dvc_plots_diff.png) + +### DVC Extension for VS Code + +Inside the +[DVC Extension for VS Code](https://marketplace.visualstudio.com/items?itemName=Iterative.dvc), +you can compare and visualize results using the +[Experiments](https://github.com/iterative/vscode-dvc/blob/main/extension/resources/walkthrough/experiments-table.md) +and +[Plots](https://github.com/iterative/vscode-dvc/blob/main/extension/resources/walkthrough/plots.md) +views: + +![VSCode Experiments](./docs/vscode_experiments.png) + +![VSCode Plots](./docs/vscode_plots.png) + +While experiments are running, live updates will be displayed in both views. + +### DVC Studio + +If you push the results to [DVC Studio](https://dvc.org/doc/studio), you can +compare experiments against the entire repo history: + +![Studio Compare](./docs/studio_compare.png) + +You can enable +[Studio Live Experiments](https://dvc.org/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots) +to see live updates while experiments are running. + +______________________________________________________________________ + +# Comparison to related technologies + +**DVCLive** is an *ML Logger*, similar to: + +- [MLFlow](https://mlflow.org/) +- [Weights & Biases](https://wandb.ai/site) +- [Neptune](https://neptune.ai/) + +The main difference with those *ML Loggers* is that **DVCLive** does not +**require** any additional services or servers to run. + +Logged metrics, parameters, and plots are stored as plain text files that can be +versioned by tools like Git or tracked as pointers to files in DVC storage. + +You can then use different [options](#comparing) to visualize the metrics, +parameters, and plots across experiments. + +______________________________________________________________________ + +# Contributing + +Contributions are very welcome. To learn more, see the +[Contributor Guide](CONTRIBUTING.rst). + +# License + +Distributed under the terms of the +[Apache 2.0 license](https://opensource.org/licenses/Apache-2.0), *dvclive* is +free and open source software. diff --git a/README.rst b/README.rst deleted file mode 100644 index cb48a08d..00000000 --- a/README.rst +++ /dev/null @@ -1,110 +0,0 @@ -DVCLive -------- - -|PyPI| |Status| |Python Version| |License| - -|Tests| |Codecov| |pre-commit| |Black| - -.. |PyPI| image:: https://img.shields.io/pypi/v/dvclive.svg - :target: https://pypi.org/project/dvclive/ - :alt: PyPI -.. |Status| image:: https://img.shields.io/pypi/status/dvclive.svg - :target: https://pypi.org/project/dvclive/ - :alt: Status -.. |Python Version| image:: https://img.shields.io/pypi/pyversions/dvclive - :target: https://pypi.org/project/dvclive - :alt: Python Version -.. |License| image:: https://img.shields.io/pypi/l/dvclive - :target: https://opensource.org/licenses/Apache-2.0 - :alt: License -.. |Tests| image:: https://github.com/iterative/dvclive/workflows/Tests/badge.svg?branch=main - :target: https://github.com/iterative/dvclive/actions?workflow=Tests - :alt: Tests -.. |Codecov| image:: https://codecov.io/gh/iterative/dvclive/branch/main/graph/badge.svg - :target: https://app.codecov.io/gh/iterative/dvclive - :alt: Codecov -.. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white - :target: https://github.com/pre-commit/pre-commit - :alt: pre-commit -.. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg - :target: https://github.com/psf/black - :alt: Black - -DVCLive is a Python library for logging machine learning metrics and other metadata in simple file formats, which is fully compatible with DVC. - -`Documentation `_ ----------------------------------------------- - -- `Get Started `_ -- `How it Works `_ -- `API Reference `_ - -Installation ------------- - -You can install *dvclive* via pip_ from PyPI_: - -.. code:: console - - $ pip install dvclive - -Depending on the *ML framework* you plan to use to train your model, you might need to specify -one of the optional dependencies: ``mmcv``, ``tf``, ``xgb``. Or ``all`` to include them all. -For example, for TensorFlow the command should look like this: - -.. code-block:: bash - - pip install dvclive[tf] - -TensorFlow and its dependencies will be installed automatically. - - -To install the development version, run: - -.. code-block:: bash - - pip install git+https://github.com/iterative/dvclive - -Comparison to related technologies ----------------------------------- - -**DVCLive** is an *ML Logger*, similar to: - -- `MLFlow `_ -- `Weights & Biases `_ -- `Neptune `_ - -The main difference with those *ML Loggers* is that **DVCLive** does not require any additional services or servers to run. - -Logged metrics and metadata are stored as plain text files that can be versioned by version control tools (i.e Git) or tracked as pointers to files in DVC storage. - ------ - - -Contributing ------------- - -Contributions are very welcome. -To learn more, see the `Contributor Guide`_. - - -License -------- - -Distributed under the terms of the `Apache 2.0 license`_, -*dvclive* is free and open source software. - - -Issues ------- - -If you encounter any problems, -please `file an issue`_ along with a detailed description. - - -.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0 -.. _PyPI: https://pypi.org/ -.. _file an issue: https://github.com/iterative/dvclive/issues -.. _pip: https://pip.pypa.io/ -.. github-only -.. _Contributor Guide: CONTRIBUTING.rst diff --git a/docs/dvc_plots_diff.png b/docs/dvc_plots_diff.png new file mode 100644 index 00000000..7de45f45 Binary files /dev/null and b/docs/dvc_plots_diff.png differ diff --git a/docs/studio_compare.png b/docs/studio_compare.png new file mode 100644 index 00000000..47624b2e Binary files /dev/null and b/docs/studio_compare.png differ diff --git a/docs/vscode_experiments.png b/docs/vscode_experiments.png new file mode 100644 index 00000000..9cd6c785 Binary files /dev/null and b/docs/vscode_experiments.png differ diff --git a/docs/vscode_plots.png b/docs/vscode_plots.png new file mode 100644 index 00000000..a3c856d4 Binary files /dev/null and b/docs/vscode_plots.png differ