basin-provider

Data ingestion for Tableland Basin

Table of Contents

• basin-provider
• Table of Contents
• Background
• Usage
  • Basin Worker
• Development
  • Running
  • Run Tests
• Contributing
• License

Background

Tableland Basin is a secure and verifiable open data platform. The Basin Provider is a collection of daemons and libraries that work in conjuction with basin-cli and basin-storage.

basin-provider is responsible for ingesting Basin Publications that are archived on Filecoin by basin-storage. Publication data can be ingested in two forms:

1. Continuously published data from a connected database (currently, only PostgreSQL is supported)
2. Raw parquet files

See basin-cli for usage info.

This repo contains the following crates:

• basin_worker: Daemon that exposes a Cap’n Proto RPC API for creating publications and ingesting publication data. [basin-storage`](https://github.com/tablelandnetwork/basin-storage.git).
• basin_evm: Rust bindings for the basin-storage EVM contract.
• basin_protocol: Cap’n Proto protocol definitions and Rust bindings.
• basin_common: Shared types and utilities.

🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧

Usage

Basin Worker

basin_worker --help

Ingest daemon for Tableland Basin

Usage: basin_worker [OPTIONS] --export-bucket <EXPORT_BUCKET> --export-credentials <EXPORT_CREDENTIALS> --database-url <DATABASE_URL>

Options:
      --evm-type <EVM_TYPE>
          EVM type (other EVM flags are ignored when this is 'mem')

          [env: EVM_TYPE=]
          [default: remote]

          Possible values:
          - mem:    Use an in-memory ephemeral EVM w/ random wallet (a BasinStorage contract will be deployed)
          - remote: Requires wallet private key, contract address, provider URL, and chain ID

      --evm-wallet-pk <EVM_WALLET_PK>
          Wallet private key (ECDSA, secp256k1) to use with the EVM (must have PUB_ADMIN_ROLE)

          [env: EVM_WALLET_PK=]

      --evm-contract-address <EVM_CONTRACT_ADDRESS>
          BasinStorage EVM contract address (ECDSA, secp256k1)

          [env: EVM_CONTRACT_ADDRESS=]

      --evm-provider-url <EVM_PROVIDER_URL>
          EVM provider URL

          [env: EVM_PROVIDER_URL=]
          [default: http://127.0.0.1:8545]

      --evm-chain-id <EVM_CHAIN_ID>
          EVM chain ID

          [env: EVM_CHAIN_ID=]
          [default: 31337]

      --export-bucket <EXPORT_BUCKET>
          Parquet export GCS bucket

          [env: EXPORT_BUCKET=]

      --export-credentials <EXPORT_CREDENTIALS>
          Parquet export sink credentials

          [env: EXPORT_CREDENTIALS=]

      --database-url <DATABASE_URL>
          Postgres-style database URL

          [env: DATABASE_URL=]

      --bind-address <BIND_ADDRESS>
          Host and port to bind the RPC API to

          [env: BIND_ADDRESS=]
          [default: 127.0.0.1:3000]

      --bind-health-address <BIND_HEALTH_ADDRESS>
          Host and port to bind the Health API to

          [env: BIND_HEALTH_ADDRESS=]
          [default: 127.0.0.1:3001]

  -v, --verbosity...
          Logging verbosity (repeat for more verbose logging)
          
          [env: VERBOSITY=]

  -q, --quiet
          Silence logging
          
          [env: QUIET=]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Development

Running

docker compose up

Run Tests

docker compose up crdb
cargo test

Contributing

PRs accepted.

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors