Skip to content

Commit

Permalink
Potential v0.5.1 PyPi release (#347)
Browse files Browse the repository at this point in the history
* A candidate for the v0.5 pypi release; added metadata to pyproject.toml; limited python versions to 3.10 variants (until we can include a version of modelbench that supports higher); updated readme with new options

* A candidate for the v0.5.1 pypi release; works with pythons from 3.10 to 3.12.
  • Loading branch information
dhosterman authored Apr 29, 2024
1 parent 5a5cf65 commit 6d0d95c
Show file tree
Hide file tree
Showing 3 changed files with 75 additions and 28 deletions.
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ can get lower-fidelity reports in minutes by running a benchmark with fewer item

## Installation

Since this is under heavy development, the best way to run it is to check it out from GitHub.
Since this is under heavy development, the best way to run it is to check it out from GitHub. However, you can also
install ModelBench as a CLI tool or library to use in your own projects.

### Install ModelBench with [Poetry](https://python-poetry.org/) for local development.

Expand All @@ -55,7 +56,14 @@ poetry install
```

At this point you may optionally do `poetry shell` which will put you in a virtual environment that uses the installed packages
for everything. If you do that, you don't have to explictly say `poetry run` in the commands below.
for everything. If you do that, you don't have to explicitly say `poetry run` in the commands below.

### Install ModelBench from PyPi.

1. Install ModelBench into your local environment or project the way you normally would. For example:
```shell
pip install modelbench
```

## Running Tests

Expand All @@ -74,7 +82,8 @@ tests and benchmarks.
### Running Your First Benchmark

Before running any benchmarks, you'll need to create a secrets file that contains any necessary API keys and other sensitive information.
Create a file at `config/secrets.toml`. You can use the following as a template.
Create a file at `config/secrets.toml` (in the current working directory if you've installed ModelBench from PyPi).
You can use the following as a template.

```toml
[together]
Expand All @@ -84,6 +93,7 @@ api_key = "<your key here>"
To obtain an API key for Together, you can create an account [here](https://api.together.xyz/).

With your keys in place, you are now ready to run your first benchmark!
Note: Omit `poetry run` in all example commands going forward if you've installed ModelBench from PyPi.

```shell
poetry run modelbench benchmark -m 10
Expand Down
47 changes: 25 additions & 22 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 37 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,51 @@ requires-python = ">=3.10, <3.13"

[tool.poetry]
name = "modelbench"
version = "0.5.0"
description = ""
version = "0.5.1"
description = "Run benchmarks and generate reports measuring the behavior of many AI Systems."
license = "Apache-2.0"
authors = ["MLCommons AI Safety <[email protected]>"]
readme = "README.md"
repository = "https://github.com/mlcommons/modelbench"
keywords = [
"AI",
"GenAI",
"LLM",
"NLP",
"evaluate",
"measure",
"quality",
"testing",
"prompt",
"safety",
"compare",
"artificial",
"intelligence",
"Large",
"Language",
"Models",
]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Intended Audience :: Information Technology",
"Intended Audience :: Science/Research",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: System :: Benchmark",
"Typing :: Typed",
]
packages = [
{ include = "modelbench", from = "src" }
]

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
modelgauge = { version = "^0.5.0", extras = ["all_plugins"] }
modelgauge = { version = ">=0.5.1", extras = ["all_plugins"] }
jq = "^1.6.0"
click = "^8.1.7"
casefy = "^0.1.7"
Expand Down

0 comments on commit 6d0d95c

Please sign in to comment.