Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change max block size for rooted splitter #21

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Change max block size for rooted splitter #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f3a75d8
Change max block size for rooted splitter
aschmahmann Apr 1, 2024
73fe19b
Update README.md
aschmahmann Apr 1, 2024
49f6456
Update root-node.js
aschmahmann Apr 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ The API is the same as for [`CarSplitter`](#class-carsplitter).

The root node is a `dag-cbor` node that is a tuple of the string `/carbites/1`, an array of root CIDs (only seen in first CAR) and an array of block CIDs (all the blocks in the CAR). e.g. `['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']]`.

Note: The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.
Note: The root node is limited to 2MiB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 15GiB.

### `class RootedCarJoiner`

Expand Down
4 changes: 2 additions & 2 deletions lib/rooted/root-node.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { Block } from 'multiformats/block'

/** @typedef {['/carbites/1', CID[], CID[]]} RootNode */

const MAX_SIZE = 1024 * 1024 * 4
const MAX_SIZE = 1024 * 1024 * 2

/**
* Make a carbites root node. Format: ['/carbites/1', roots, blocks]
Expand All @@ -18,7 +18,7 @@ export async function mkRootNode (roots, blocks) {
const value = ['/carbites/1', Array.from(roots), Array.from(blocks)]
const bytes = encode(value)
// FIXME: Given a V1 CID of ~36 bytes and the default IPFS chunk size of
// 262,144 bytes you'd need to be splitting at 30GiB or more to experience
// 262,144 bytes you'd need to be splitting at 15GiB or more to experience
// this error.
if (bytes.length >= MAX_SIZE) {
throw new Error('root node too big. The root node is bigger than 4MiB: the biggest message IPFS will bitswap. Split the CAR into smaller chunks.')
Expand Down