Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark Piece Indexer: find piece info in ContextID #194

Open
bajtos opened this issue Nov 18, 2024 · 0 comments
Open

Spark Piece Indexer: find piece info in ContextID #194

bajtos opened this issue Nov 18, 2024 · 0 comments

Comments

@bajtos
Copy link
Contributor

bajtos commented Nov 18, 2024

Rework spark-piece-indexer as follows:

  • Ingest only advertisements where ContextID contains Piece Info
  • The REST API for querying should accept ContextID instead of PieceCID

How does Curio map Piece Info to ContextID:

  • The PieceInfo is a structure defined as {PieceCID, PieceSize}
  • The piece info is serialised using DAG-CBOR and base64 encoded

Example IPNI advertisement:
https://http.f02620.devtty.eu/ipni-provider/12D3KooWBpYWsD4DuzTTVQdS6N5mt6rc5HtY11p7KLfhyVLw83d6/ipni/v1/ad/baguqeera6g3t7nianlk2kplejqyzl5wvbaq6h4vdqrhhgm2ctid7q4sgcx4a

{
  "Addresses":["/dns/cid.contact/https/http-path/%2Fingest%2Fannounce"],
  "ContextID":{"/":{"bytes":"ghsAAAAIAAAAANgqWCgAAYHiA5IgIPxmN381s3gKZTaNDaP+GGLv+fs22x1BbOe5TC4QYugO"}},
  // etc.
}

Decoded ContextID (via https://cbor.me):

82                                      # array(2)
   1B 0000000800000000                  # unsigned(34359738368)
   D8 2A                                # tag(42)
      58 28                             # bytes(40)
         000181E203922020FC66377F35B3780A65368D0DA3FE1862EFF9FB36DB1D416CE7B94C2E1062E80E # 

Node.js script to inspect and reconstruct the ContextID:

import { encode, decode } from '@ipld/dag-cbor'
import { CID } from 'multiformats'

const contextId = 'ghsAAAAIAAAAANgqWCgAAYHiA5IgIPxmN381s3gKZTaNDaP+GGLv+fs22x1BbOe5TC4QYugO'
console.log('Input ContextID', contextId)

const contextIdBytes = Buffer.from(contextId, 'base64')
const data = decode(contextIdBytes)
console.log('decoded', data)

console.log('PieceSize', data[0].toString())
console.log('PieceCID', data[1].toString())

const contextId2Bytes = encode([
  Number(data[0].toString()),
  CID.parse(data[1].toString())
])
const contextId2 = Buffer.from(contextId2Bytes).toString('base64')
console.log('Reconstructed ContextID', contextId2)
console.log('Match?', contextId === contextId2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant