Storage Market

Storage Market in Filecoin

Storage Market subsystem is the data entry point into the network. Storage miners only earn power from data stored in a storage deal and all deals live on the Filecoin network. Specific deal negotiation process happens off chain, clients and miners enter a storage deal after an agreement has been reached and post storage deals on the Filecoin network to earn block rewards and get paid for storing the data in the storage deal. A deal is only valid when it is posted on chain with signatures from both parties and at the time of posting, there are sufficient balances for both parties locked up to honor the deal in terms of deal price and deal collateral.


  • StorageClient - The party that wants to make a deal to store data
  • StorageProvider - The party that will store the data in exchange for payment. A storage miner.
  • StorageMarketActor - The on-chain component of deals. The StorageMarketActor is analogous to an escrow and a ledger for all deals made.
  • StorageAsk - The current price and parameters a miner is currently offering for storage (analogous to an Ask in a financial market)
  • StorageDealProposal - A proposal for a storage deal, signed only by the - Storage client
  • StorageDeal - A storage deal proposal with a counter signature from the Provider, which then goes on-chain.

Deal Flow

The lifecycle for a deal within the storage market contains distinct phases:

  1. Discovery - The client identifies miners and determines their current asks.
  2. Negotiation (out of band) - Both parties come to an agreement about the terms of the deal, each party commits funds to the deal and data is transferred from the client to the provider.
  3. Publishing - The deal is published on chain, making the storage provider publicly accountable for the deal.
  4. Handoff - Once the deal is published, it is handed off and handled by the Storage Mining Subsystem. The Storage Mining Subsystem will add the data corresponding to the deal to a sector, seal the sector, and tell the Storage Market Actor that the deal is in a sector, thereby marking the deal as active.

From that point on, the deal is handled by the Storage Mining Subsystem, which communicates with the Storage Market Actor in order to process deal payments. See Storage Mining Subsystem for more details.

The following diagram outlines the phases of deal flow within the storage market in detail:

Storage Market Deal Flow


Discovery is the client process of identifying storage providers (i.e. a miner) who (subject to agreement on the deal’s terms) are offering to store the client’s data. There are many ways which a client can use to identify a provider to store their data. The list below outlines the minimum discovery services a filecoin implementation MUST provide. As the network evolves, third parties may build systems that supplement or enhance these services.

Discovery involves identifying providers and determining their current StorageAsk. The steps are as follows:

  1. A client queries the chain to retrieve a list of Storage Miner Actors who have registerd as miners with the StoragePowerActor.
  2. A client may perform additional queries to each Storage Miner Actor to determine their properties. Among others, these properties can include worker address, sector size, libp2p Multiaddress etc.
  3. Once the client identifies potentially suitable providers, it sends a direct libp2p message using the Storage Query Protocol to get each potential provider’s current StorageAsk.
  4. Miners respond on the AskProtocol with a signed version of their current StorageAsk.

A StorageAsk contains all the properties that a client will need to determine if a given provider will meet its needs for storage at this moment. Providers should update their asks frequently to ensure the information they are providing to clients is up to date.


Negotiation is the out-of-band process during which a storage client and a storage provider come to an agreement about a storage deal and reach the point where a deal is published on chain.

Negotiation begins once a client has discovered a miner whose StorageAsk meets their desired criteria. The recommended order of operations for negotiating and publishing a deal is as follows:

  1. In order to propose a storage deal, the StorageClient calculates the piece commitment (CommP) for the data it intends to store. This is neccesary so that the StorageProvider can verify that the data the StorageClient sends to be stored matches the CommP in the StorageDealProposal. For more detail about the relationship between payloads, pieces, and CommP see Piece.
  2. Before sending a proposal to the provider, the StorageClient adds funds for a deal, as necessary, to the StorageMarketActor (by calling AddBalance).
  3. The StorageClient now creates a StorageDealProposal and sends the proposal and the CID for the root of the data payload to be stored to the StorageProvider using the Storage Deal Protocol.

From this point onwards, execution moves to the StorageProvider.

  1. The StorageProvider inspects the deal to verify that the deal’s parameters match its own internal criteria (such as price, piece size, deal duration, etc). The StorageProvider rejects the proposal if the parameters don’t match its own criteria by sending a rejection to the client over the Storage Deal Protocol.
  2. The StorageProvider queries the StorageMarketActor to verify the StorageClient has deposited enough funds to make the deal (i.e. the client’s balance is greater than the total storage price) and rejects the proposal if it hasn’t.
  3. If all criteria are met, the StorageProvider responds using the Storage Deal Protocol to indicate an intent to accept the deal.

From this point onwards execution moves back to the StorageClient.

  1. The StorageClient opens a push request for the payload data using the Data Transfer Module, and sends the request to the provider along with a voucher containing the CID for the StorageDealProposal.
  2. The StorageProvider checks the voucher and verifies that the CID matches the storage deal proposal it has received and verified but not put on chain already. If so, it accepts the data transfer request from the StorageClient.
  3. The Data Transfer Module now transfers the payload data to be stored from the StorageClient to the StorageProvider using GraphSync.
  4. Once complete, the Data Transfer Module notifies the StorageProvider.
  5. The StorageProvider recalculates the piece commitment (CommP) from the data transfer that just completed and verifies it matches the piece commitment in the StorageDealProposal.


Data is now transferred, both parties have agreed, and it’s time to publish the deal. Given that the counter signature on a deal proposal is a standard message signature by the provider and the signed deal is an on-chain message, it is usually the StorageProvider that publishes the deal. However, if StorageProvider decides to send this signed on-chain message to the client before calling PublishStorageDeal then the client can publish the deal on-chain. The client’s funds are not locked until the deal is published and a published deal that is not activated within some pre-defined window will result in an on-chain penalty.

  1. First, the StorageProvider adds collateral for the deal as needed to the StorageMarketActor (using AddBalance).
  2. Then, the StorageProvider prepares and signs the on-chain StorageDeal message with the StorageDealProposal signed by the client and its own signature. It can now either send this message back to the client or call PublishStorageDeals on the StorageMarketActor to publish the deal. It is recommended for StorageProvider to send back the signed message before PublishStorageDeals is called.
  3. After calling PublishStorageDeals, the StorageProvider sends a message to the StorageClient on the Storage Deal Protocol with the CID of the message that it is putting on chain for convenience.
  4. If all goes well, the StorageMarketActor responds with an on-chain DealID for the published deal.

Finally, the StorageClient verifies the deal.

  1. The StorageClient queries the node for the CID of the message published on chain (sent by the provider). It then inspects the message parameters to make sure they match the previously agreed deal.


Now that a deal is published, it needs to be stored, sealed, and proven in order for the provider to be paid. See Storage Deal for more information about how deal payments are made. These later stages of a deal are handled by the Storage Mining Subsystem. So the final task for the Storage Market is to handoff to the Storage Mining Subsystem.

  1. The StorageProvider writes the serialized, padded piece to a shared Filestore.
  2. The StorageProvider calls HandleStorageDeal on the StorageMiner with the published StorageDeal and filestore path (in Go this is the io.Reader).

A note regarding the order of operations: the only requirement to publish a storage deal with the StorageMarketActor is that the StorageDealProposal is signed by the StorageClient, the publish message is signed by the StorageProvider, and both parties have deposited adequate funds/collateral in the StorageMarketActor. As such, it’s not required that the steps listed above happen in this exact order. However, the above order is recommended because it generally minimizes the ability of either party to act maliciously.

Data Representation in the Storage Market

Data submitted to the Filecoin network go through several transformations before they come to the format at which the StorageProvider stores it. Here we provide a summary of these transformations.

  1. When a piece of data, or file is submitted to Filecoin (in some raw system format) it is transformed into a UnixFS DAG style data representation (in case it is not in this format already, e.g., from IPFS-based applications). The hash that represents the root of the IPLD DAG of the UnixFS file is the Payload CID, which is used in the Retrieval Market.
  2. In order to make a Filecoin Piece the UnixFS IPLD DAG is serialised into a .car file, which is also raw bytes.
  3. The resulting .car file is padded with some extra data.
  4. The next step is to calculate the Merkle root out of the hashes of individual Pieces. The resulting root of the Merkle tree is the Piece CID. This is also referred to as CommP. Note that at this stage the data is still unsealed.
  5. At this point, the Piece is included in a Sector together with data from other deals. The StorageProvider then calculates Merkle root for all the Pieces inside the sector. The root of this tree is CommD. This is the unsealed sector CID.
  6. The StorageProvider is then sealing the sector and the root of the resulting Merkle root is the CommR.

The following data types are unique to the Storage Market:

package storagemarket

import (

	logging ""
	ma ""
	cbg ""

	datatransfer ""


var log = logging.Logger("storagemrkt")

//go:generate cbor-gen-for --map-encoding ClientDeal MinerDeal Balance SignedStorageAsk StorageAsk DataRef ProviderDealState DealStages DealStage Log

// DealProtocolID is the ID for the libp2p protocol for proposing storage deals.
const OldDealProtocolID = "/fil/storage/mk/1.0.1"
const DealProtocolID = "/fil/storage/mk/1.1.0"

// AskProtocolID is the ID for the libp2p protocol for querying miners for their current StorageAsk.
const OldAskProtocolID = "/fil/storage/ask/1.0.1"
const AskProtocolID = "/fil/storage/ask/1.1.0"

// DealStatusProtocolID is the ID for the libp2p protocol for querying miners for the current status of a deal.
const OldDealStatusProtocolID = "/fil/storage/status/1.0.1"
const DealStatusProtocolID = "/fil/storage/status/1.1.0"

// Balance represents a current balance of funds in the StorageMarketActor.
type Balance struct {
	Locked    abi.TokenAmount
	Available abi.TokenAmount

// StorageAsk defines the parameters by which a miner will choose to accept or
// reject a deal. Note: making a storage deal proposal which matches the miner's
// ask is a precondition, but not sufficient to ensure the deal is accepted (the
// storage provider may run its own decision logic).
type StorageAsk struct {
	// Price per GiB / Epoch
	Price         abi.TokenAmount
	VerifiedPrice abi.TokenAmount

	MinPieceSize abi.PaddedPieceSize
	MaxPieceSize abi.PaddedPieceSize
	Miner        address.Address
	Timestamp    abi.ChainEpoch
	Expiry       abi.ChainEpoch
	SeqNo        uint64

// SignedStorageAsk is an ask signed by the miner's private key
type SignedStorageAsk struct {
	Ask       *StorageAsk
	Signature *crypto.Signature

// SignedStorageAskUndefined represents the empty value for SignedStorageAsk
var SignedStorageAskUndefined = SignedStorageAsk{}

// StorageAskOption allows custom configuration of a storage ask
type StorageAskOption func(*StorageAsk)

// MinPieceSize configures a minimum piece size of a StorageAsk
func MinPieceSize(minPieceSize abi.PaddedPieceSize) StorageAskOption {
	return func(sa *StorageAsk) {
		sa.MinPieceSize = minPieceSize

// MaxPieceSize configures maxiumum piece size of a StorageAsk
func MaxPieceSize(maxPieceSize abi.PaddedPieceSize) StorageAskOption {
	return func(sa *StorageAsk) {
		sa.MaxPieceSize = maxPieceSize

// StorageAskUndefined represents an empty value for StorageAsk
var StorageAskUndefined = StorageAsk{}

// MinerDeal is the local state tracked for a deal by a StorageProvider
type MinerDeal struct {
	ProposalCid           cid.Cid
	AddFundsCid           *cid.Cid
	PublishCid            *cid.Cid
	Miner                 peer.ID
	Client                peer.ID
	State                 StorageDealStatus
	PiecePath             filestore.Path
	MetadataPath          filestore.Path
	SlashEpoch            abi.ChainEpoch
	FastRetrieval         bool
	Message               string
	FundsReserved         abi.TokenAmount
	Ref                   *DataRef
	AvailableForRetrieval bool

	DealID       abi.DealID
	CreationTime cbg.CborTime

	TransferChannelId *datatransfer.ChannelID
	SectorNumber      abi.SectorNumber

	InboundCAR string

// NewDealStages creates a new DealStages object ready to be used.
// EXPERIMENTAL; subject to change.
func NewDealStages() *DealStages {
	return &DealStages{}

// DealStages captures a timeline of the progress of a deal, grouped by stages.
// EXPERIMENTAL; subject to change.
type DealStages struct {
	// Stages contains an entry for every stage that the deal has gone through.
	// Each stage then contains logs.
	Stages []*DealStage

// DealStages captures data about the execution of a deal stage.
// EXPERIMENTAL; subject to change.
type DealStage struct {
	// Human-readable fields.
	// TODO: these _will_ need to be converted to canonical representations, so
	//  they are machine readable.
	Name             string
	Description      string
	ExpectedDuration string

	// Timestamps.
	// TODO: may be worth adding an exit timestamp. It _could_ be inferred from
	//  the start of the next stage, or from the timestamp of the last log line
	//  if this is a terminal stage. But that's non-determistic and it relies on
	//  assumptions.
	CreatedTime cbg.CborTime
	UpdatedTime cbg.CborTime

	// Logs contains a detailed timeline of events that occurred inside
	// this stage.
	Logs []*Log

// Log represents a point-in-time event that occurred inside a deal stage.
// EXPERIMENTAL; subject to change.
type Log struct {
	// Log is a human readable message.
	// TODO: this _may_ need to be converted to a canonical data model so it
	//  is machine-readable.
	Log string

	UpdatedTime cbg.CborTime

// GetStage returns the DealStage object for a named stage, or nil if not found.
// TODO: the input should be a strongly-typed enum instead of a free-form string.
// TODO: drop Get from GetStage to make this code more idiomatic. Return a
//  second ok boolean to make it even more idiomatic.
// EXPERIMENTAL; subject to change.
func (ds *DealStages) GetStage(stage string) *DealStage {
	if ds == nil {
		return nil

	for _, s := range ds.Stages {
		if s.Name == stage {
			return s

	return nil

// AddStageLog adds a log to the specified stage, creating the stage if it
// doesn't exist yet.
// EXPERIMENTAL; subject to change.
func (ds *DealStages) AddStageLog(stage, description, expectedDuration, msg string) {
	if ds == nil {

	log.Debugf("adding log for stage <%s> msg <%s>", stage, msg)

	now := curTime()
	st := ds.GetStage(stage)
	if st == nil {
		st = &DealStage{
			CreatedTime: now,
		ds.Stages = append(ds.Stages, st)

	st.Name = stage
	st.Description = description
	st.ExpectedDuration = expectedDuration
	st.UpdatedTime = now
	if msg != "" && (len(st.Logs) == 0 || st.Logs[len(st.Logs)-1].Log != msg) {
		// only add the log if it's not a duplicate.
		st.Logs = append(st.Logs, &Log{msg, now})

// AddLog adds a log inside the DealStages object of the deal.
// EXPERIMENTAL; subject to change.
func (d *ClientDeal) AddLog(msg string, a ...interface{}) {
	if len(a) > 0 {
		msg = fmt.Sprintf(msg, a...)

	stage := DealStates[d.State]
	description := DealStatesDescriptions[d.State]
	expectedDuration := DealStatesDurations[d.State]

	d.DealStages.AddStageLog(stage, description, expectedDuration, msg)

// ClientDeal is the local state tracked for a deal by a StorageClient
type ClientDeal struct {
	ProposalCid       cid.Cid
	AddFundsCid       *cid.Cid
	State             StorageDealStatus
	Miner             peer.ID
	MinerWorker       address.Address
	DealID            abi.DealID
	DataRef           *DataRef
	Message           string
	DealStages        *DealStages
	PublishMessage    *cid.Cid
	SlashEpoch        abi.ChainEpoch
	PollRetryCount    uint64
	PollErrorCount    uint64
	FastRetrieval     bool
	FundsReserved     abi.TokenAmount
	CreationTime      cbg.CborTime
	TransferChannelID *datatransfer.ChannelID
	SectorNumber      abi.SectorNumber

// StorageProviderInfo describes on chain information about a StorageProvider
// (use QueryAsk to determine more specific deal parameters)
type StorageProviderInfo struct {
	Address    address.Address // actor address
	Owner      address.Address
	Worker     address.Address // signs messages
	SectorSize uint64
	PeerID     peer.ID
	Addrs      []ma.Multiaddr

// ProposeStorageDealResult returns the result for a proposing a deal
type ProposeStorageDealResult struct {
	ProposalCid cid.Cid

// ProposeStorageDealParams describes the parameters for proposing a storage deal
type ProposeStorageDealParams struct {
	Addr          address.Address
	Info          *StorageProviderInfo
	Data          *DataRef
	StartEpoch    abi.ChainEpoch
	EndEpoch      abi.ChainEpoch
	Price         abi.TokenAmount
	Collateral    abi.TokenAmount
	Rt            abi.RegisteredSealProof
	FastRetrieval bool
	VerifiedDeal  bool

const (
	// TTGraphsync means data for a deal will be transferred by graphsync
	TTGraphsync = "graphsync"

	// TTManual means data for a deal will be transferred manually and imported
	// on the provider
	TTManual = "manual"

// DataRef is a reference for how data will be transferred for a given storage deal
type DataRef struct {
	TransferType string
	Root         cid.Cid

	PieceCid     *cid.Cid              // Optional for non-manual transfer, will be recomputed from the data if not given
	PieceSize    abi.UnpaddedPieceSize // Optional for non-manual transfer, will be recomputed from the data if not given
	RawBlockSize uint64                // Optional: used as the denominator when calculating transfer %

// ProviderDealState represents a Provider's current state of a deal
type ProviderDealState struct {
	State         StorageDealStatus
	Message       string
	Proposal      *market.DealProposal
	ProposalCid   *cid.Cid
	AddFundsCid   *cid.Cid
	PublishCid    *cid.Cid
	DealID        abi.DealID
	FastRetrieval bool

func curTime() cbg.CborTime {
	now := time.Now()
	return cbg.CborTime(time.Unix(0, now.UnixNano()).UTC())

Details about StorageDealProposal and StorageDeal (which are used in the Storage Market and elsewhere) specifically can be found in Storage Deal.


Name: Storage Query Protocol
Protocol ID: /fil/<network-name>/storage/ask/1.0.1

Request: CBOR Encoded AskProtocolRequest Data Structure Response: CBOR Encoded AskProtocolResponse Data Structure

Name: Storage Deal Protocol
Protocol ID: /fil/<network-name>/storage/mk/1.0.1

Request: CBOR Encoded DealProtocolRequest Data Structure Response: CBOR Encoded DealProtocolResponse Data Structure

Storage Provider

The StorageProvider is a module that handles incoming queries for Asks and proposals for Deals from a StorageClient. It also tracks deals as they move through the deal flow, handling off chain actions during the negotiation phases of the deal and ultimately telling the StorageMarketActor to publish on chain. The StorageProvider’s last action is to handoff a published deal for storage and sealing to the Storage Mining Subsystem. Note that any address registered as a StorageMarketParticipant with the StorageMarketActor can be used with the StorageClient.

It is worth highlighting that a single participant can be a StorageClient, StorageProvider, or both at the same time.

Because most of what a Storage Provider does is respond to actions initiated by a StorageClient, most of its public facing methods relate to getting current status on deals, as opposed to initiating new actions. However, a user of the StorageProvider module can update the current Ask for the provider.

package storagemarket

import (




// ProviderSubscriber is a callback that is run when events are emitted on a StorageProvider
type ProviderSubscriber func(event ProviderEvent, deal MinerDeal)

// StorageProvider provides an interface to the storage market for a single
// storage miner.
type StorageProvider interface {

	// Start initializes deal processing on a StorageProvider and restarts in progress deals.
	// It also registers the provider with a StorageMarketNetwork so it can receive incoming
	// messages on the storage market's libp2p protocols
	Start(ctx context.Context) error

	// OnReady registers a listener for when the provider comes on line

	// Stop terminates processing of deals on a StorageProvider
	Stop() error

	// SetAsk configures the storage miner's ask with the provided prices (for unverified and verified deals),
	// duration, and options. Any previously-existing ask is replaced.
	SetAsk(price abi.TokenAmount, verifiedPrice abi.TokenAmount, duration abi.ChainEpoch, options ...StorageAskOption) error

	// GetAsk returns the storage miner's ask, or nil if one does not exist.
	GetAsk() *SignedStorageAsk

	// ListLocalDeals lists deals processed by this storage provider
	ListLocalDeals() ([]MinerDeal, error)

	// AddStorageCollateral adds storage collateral
	AddStorageCollateral(ctx context.Context, amount abi.TokenAmount) error

	// GetStorageCollateral returns the current collateral balance
	GetStorageCollateral(ctx context.Context) (Balance, error)

	// ImportDataForDeal manually imports data for an offline storage deal
	ImportDataForDeal(ctx context.Context, propCid cid.Cid, data io.Reader) error

	// SubscribeToEvents listens for events that happen related to storage deals on a provider
	SubscribeToEvents(subscriber ProviderSubscriber) shared.Unsubscribe

	RetryDealPublishing(propCid cid.Cid) error

Storage Client

The StorageClient is a module that discovers miners, determines their asks, and proposes deals to StorageProviders. It also tracks deals as they move through the deal flow. Note that any address registered as a StorageMarketParticipant with the StorageMarketActor can be used with the StorageClient.

Recall that a single participant can be a StorageClient, StorageProvider, or both at the same time.

package storagemarket

import (

	bstore ""



type PayloadCID = cid.Cid

// BlockstoreAccessor is used by the storage market client to get a
// blockstore when needed, concretely to send the payload to the provider.
// This abstraction allows the caller to provider any blockstore implementation:
// a CARv2 file, an IPFS blockstore, or something else.
// They key is a payload CID because this is the unique top-level key of a
// client-side data import.
type BlockstoreAccessor interface {
	Get(PayloadCID) (bstore.Blockstore, error)
	Done(PayloadCID) error

// ClientSubscriber is a callback that is run when events are emitted on a StorageClient
type ClientSubscriber func(event ClientEvent, deal ClientDeal)

// StorageClient is a client interface for making storage deals with a StorageProvider
type StorageClient interface {

	// Start initializes deal processing on a StorageClient and restarts
	// in progress deals
	Start(ctx context.Context) error

	// OnReady registers a listener for when the client comes on line

	// Stop ends deal processing on a StorageClient
	Stop() error

	// ListProviders queries chain state and returns active storage providers
	ListProviders(ctx context.Context) (<-chan StorageProviderInfo, error)

	// ListLocalDeals lists deals initiated by this storage client
	ListLocalDeals(ctx context.Context) ([]ClientDeal, error)

	// GetLocalDeal lists deals that are in progress or rejected
	GetLocalDeal(ctx context.Context, cid cid.Cid) (ClientDeal, error)

	// GetAsk returns the current ask for a storage provider
	GetAsk(ctx context.Context, info StorageProviderInfo) (*StorageAsk, error)

	// GetProviderDealState queries a provider for the current state of a client's deal
	GetProviderDealState(ctx context.Context, proposalCid cid.Cid) (*ProviderDealState, error)

	// ProposeStorageDeal initiates deal negotiation with a Storage Provider
	ProposeStorageDeal(ctx context.Context, params ProposeStorageDealParams) (*ProposeStorageDealResult, error)

	// GetPaymentEscrow returns the current funds available for deal payment
	GetPaymentEscrow(ctx context.Context, addr address.Address) (Balance, error)

	// AddStorageCollateral adds storage collateral
	AddPaymentEscrow(ctx context.Context, addr address.Address, amount abi.TokenAmount) error

	// SubscribeToEvents listens for events that happen related to storage deals on a provider
	SubscribeToEvents(subscriber ClientSubscriber) shared.Unsubscribe