Spark v2: DDO, Allocators, Data Quality Improvements

[bajtos]

ETA: 2025-01-09

TL;DR: Each miner storing publicly retrievable data should show on the Spark public dashboard.

Spark is checking whether Storage Providers are offering public data retrievals for deals that are expected to be retrievable. To do so, we need to understand which deals are expected to be retriable. Earlier this year, the introduction of Direct Data Onboarding (DDO) and FIL+ Allocator Pathways significantly changed the Filecoin landscape, and our previous solution is no longer sufficient. The goal of this milestone is to improve Spark to support DDO & Allocators.

The second objective is to improve quality of data produced by Spark and filter out more misbehaving nodes.

Resources:

• https://www.notion.so/spacemeridian/Spark-v2-40138e6879a04e8589925b3ee9ae11b1
• https://docs.google.com/document/d/1YvFYYUwP6VibKyTeHLd89mrkB1S5sVjq9ydFCGbBfrw/edit

children:

• Support New Allocators:
• Feat: treat all FIL+ deals as publicly retrievable filecoin-station/fil-deal-ingester#16
• Data Quality Improvements:
• Retrieve CIDs directly using fetch filecoin-station/spark#46
• Report retrieval network errors filecoin-station/spark#43
• Deterministic randomness for each SPARK round filecoin-station/spark#27
• Deterministic task<->node assignment filecoin-station/spark#38
• Find majorities in Spark measurements #59
• Deal Retrievability Score #156
• DDO deals:
• Spark: IPNI Reverse Index Grant Application Write Up #128
• Lightweight DDO Compatibility: Spark Piece Indexer #143
• Spark Piece Indexer: find piece info in ContextID #194
• Real-time view of all storage deals (including DDO) #144
• Use Advertisement.ContextID as the lookup key filecoin-station/piece-indexer#33
• Integrate real-time view of deals and piece index to Spark #169
• DDO Compatibility: IPNI Reverse Index and related features #147
• Write FRC describing Spark's requirements on miner SW (Curio, Venus) #189
• Tooling & Data Access:
• Public API/website to check if a miner/client/allocator is verified by Spark filecoin-station/spark#78
• Break down retrieval success rate by Deals, Clients, Allocators, MetaAllocators #139
• Spark on-chain oracle #145
• Incentives:
• Spark long term value flow proposal #142
• PoC for long term Spark funding #146

[MikeH1999]

Sorry, I'm wondering how long this will take.

[juliangruber]

It's currently planned to land this year #113