Add experimental metrics package following the array API standard (#499)

* update dependencies

* Add lightning dependency for testing distributed metrics

* Update numpy version and add torchmetrics and cupy
dependencies

* Add optional module import utility and tests

* Add utilities for experimental metrics module

* Add distributed backends package

* Add remove_ignore_index utility function to ops.py

* Add base class for metrics

* Update dependencies and fix tests involving cupy arrays

* Ignore import errors for cupy and torch modules in
test_ops.py

* Fix import statement in test_metric.py

* Add torchmetrics as dependency to tests and fix tests

* Fix code ops and update test

* Add confusion matrix

* Fix import statements

* Refactor flatten function to use copy parameter.

* Disable mypy 'no-any-return' error code in mpi4py backend module

* Add MPI and cupy support for integration tests

* Fix MPI installation in integration tests workflow

* Check openmpi installation

* [WIP] integration test workflow fix

* [WIP] integration test workflow fix

* [WIP] integration test workflow fix

* Remove unnecessary dependency installation and
update poetry add command.

* Add cupy to test group in poetry

* use pip instead of poetry to install cupy

* Add openmpi installation check and export paths

* update integration tests workflow config

* use conftest for `metrics` tests

* Add function to get open port for PyTorch DDP setup

* Add tests for torch distributed backend

* Add __init__.py file for distributed backend tests

* Fix flatten_seq function to handle single character strings.

* Add mpi4py dependency for integration tests

* Add MPICC environment variable to mpi4py installation.

* Restructure package for distributed backends

* cleanup + improve tests

* Mark integration test for sync on compute list state

* mark as integration test

.github/workflows/integration_tests.yml

@@ -38,6 +38,11 @@ jobs:
   integration-tests:
     runs-on: [self-hosted, gpu, db]
     steps:
+      - name: Export OpenMPI PATH
+        run: |
+          export PATH=/opt/openmpi-4.1.5/bin:$PATH
+          export LD_LIBRARY_PATH=/opt/openmpi-4.1.5/lib:$LD_LIBRARY_PATH
+          ompi_info
       - uses: actions/checkout@v3
       - name: Install poetry
         run: pip install poetry
@@ -49,6 +54,8 @@ jobs:
           poetry env use '3.10'
           source $(poetry env info --path)/bin/activate
           poetry install --with test
+          pip install cupy-cuda12x
+          env MPICC=/opt/openmpi-4.1.5/bin/mpicc python -m pip install git+https://github.com/mpi4py/mpi4py
           coverage run -m pytest -m integration_test && coverage xml && coverage report -m
       - name: Upload coverage to Codecov
         uses: Wandalen/[email protected]

cyclops/evaluate/metrics/experimental/__init__.py

@@ -0,0 +1,6 @@
+"""Array-API-compatible metrics."""
+from cyclops.evaluate.metrics.experimental.confusion_matrix import (
+    BinaryConfusionMatrix,
+    MulticlassConfusionMatrix,
+    MultilabelConfusionMatrix,
+)