siafile

package
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2019 License: MIT Imports: 25 Imported by: 0

README

SiaFile

The SiaFile contains all the information about an uploaded file that is required to download it plus additional metadata about the file. The SiaFile is split up into 4kib pages. The header of the SiaFile is located within the first page of the SiaFile. More pages will be allocated should the header outgrow the page. The metadata and host public key table are kept in memory for as long as the siafile is open, and the chunks are loaded and unloaded as they are accessed.

Since SiaFile's are rapidly accessed during downloads and repairs, the SiaFile was built with the requirement that all reads and writes must be able to happen in contant time, knowing only the offset of thte logical data within the SiaFile. To achieve that, all the data is page-aligned which also improves disk performance. Overall the SiaFile package is designed to minimize disk I/O operations and to keep the memory footprint as small as possible without sacrificing performance.

Benchmarks

  • Writing to a random chunk of a SiaFile
    • i9-9900K with Intel SSDPEKNW010T8 -> 200 writes/second
  • Writing to a random chunk of a SiaFile (multithreaded)
    • i9-9900K with Intel SSDPEKNW010T8 -> 200 writes/second
  • Reading a random chunk of a SiaFile
    • i9-9900K with Intel SSDPEKNW010T8 -> 50,000 reads/second
  • Loading a a SiaFile's header into memory
    • i9-9900K with Intel SSDPEKNW010T8 -> 20,000 reads/second

Structure of the SiaFile:

Metadata

The metadata contains all the information about a SiaFile that is not specific to a single chunk of the file. This includes keys, timestamps, erasure coding etc. The definition of the Metadata type which contains all the persisted fields is located within metadata.go. The metadata is the only part of the SiaFile that is JSON encoded for easier compatibility and readability. The encoded metadata is written to the beginning of the header.

Host Public Key Table

The host public key table uses the Sia Binary Encoding and is written to the end of the header. As the table grows, it will grow towards the front of the header while the metadata grows towards the end. Should metadata and host public key table ever overlap, a new page will be allocated for the header. The host public key table is a table of all the hosts that contain pieces of the corresponding SiaFile.

Chunks

The chunks are written to disk starting at the first 4kib page after the header. For each chunk, the SiaFile reserves a full page on disk. That way the SiaFile always knows at which offset of the file to look for a chunk and can therefore read and write chunks in constant time. A chunk only consists of its pieces and each piece contains its merkle root and an offset which can be resolved to a host's public key using the host public key table. The chunk and piece types can be found in siafile.go.

Subsystems

The SiaFile is split up into the following subsystems.

Erasure Coding Subsystem

Key Files

File Format Subsystem

Key Files

The file format subsystem contains the type definitions for the SiaFile format and most of the exported methods of the package.

Persistence Subsystem

Key Files

The persistence subsystem handles all of the disk I/O and marshaling of datatypes. It provides helper functions to read the SiaFile from disk and atomically write to disk using the writeaheadlog package.

SiaFileSet Subsystem

Key Files

While a SiaFile object is threadsafe by itself, it's not safe to load a SiaFile into memory multiple times as this will cause corruptions on disk. Only one instance of a specific SiaFile can exist in memory at once. To ensure that, the siafileset was created as a pool for SiaFiles which is used by other packages to get access to SiaFileEntries which are wrappers for SiaFiles containing some extra information about how many threads are using it at a certain time. If a SiaFile was already loaded the siafileset will hand out the existing object, otherwise it will try to load it from disk.

Snapshot Subsystem

Key Files

The snapshot subsystem allows a user to create a readonly snapshot of a SiaFile. A snapshot contains most of the information a SiaFile does but can't be used to modify the underlying SiaFile directly. It is used to reduce locking contention within parts of the codebase where readonly access is good enough like the download code for example.

Documentation

Index

Constants

View Source 
const (
	CombinedChunkStatusInvalid    = iota // status wasn't initialized
	CombinedChunkStatusInComplete        // partial chunk is included in an incomplete combined chunk.
	CombinedChunkStatusCompleted         // partial chunk is included in a completed combined chunk.
)

Constants to indicate which part of the partial upload the combined chunk is currently at.

Variables

View Source 
var (
	// ErrPathOverload is an error when a file already exists at that location
	ErrPathOverload = errors.New("a file already exists at that location")
	// ErrUnknownPath is an error when a file cannot be found with the given path
	ErrUnknownPath = errors.New("no file known with that path")
	// ErrUnknownThread is an error when a SiaFile is trying to be closed by a
	// thread that is not in the threadMap
	ErrUnknownThread = errors.New("thread should not be calling Close(), does not have control of the siafile")
)

Functions

func ApplyUpdates 

func ApplyUpdates(updates ...writeaheadlog.Update) error

ApplyUpdates is a wrapper for applyUpdates that uses the production dependencies.

func CombinedChunkIndex added in v1.4.1 

func CombinedChunkIndex(numChunks, chunkIndex uint64, numCombinedChunks int) int

CombinedChunkIndex is a helper method which translates a chunk's index to the corresponding combined chunk index dependng on the number of combined chunks.

func ExtractSegment 

func ExtractSegment(pieces [][]byte, segmentIndex int, segmentSize uint64) [][]byte

ExtractSegment is a convenience method that extracts the data of the segment at segmentIndex from pieces.

func IsSiaFileUpdate 

func IsSiaFileUpdate(update writeaheadlog.Update) bool

IsSiaFileUpdate is a helper method that makes sure that a wal update belongs to the SiaFile package.

func NewRSCode 

func NewRSCode(nData, nParity int) (modules.ErasureCoder, error)

NewRSCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

func NewRSSubCode 

func NewRSSubCode(nData, nParity int, segmentSize uint64) (modules.ErasureCoder, error)

NewRSSubCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

Types

type BubbledMetadata 

type BubbledMetadata struct {
	Health              float64
	LastHealthCheckTime time.Time
	ModTime             time.Time
	NumStuckChunks      uint64
	Redundancy          float64
	Size                uint64
	StuckHealth         float64
}

BubbledMetadata is the metadata of a siafile that gets bubbled

type CachedHealthMetadata 

type CachedHealthMetadata struct {
	Health      float64
	Redundancy  float64
	StuckHealth float64
}

CachedHealthMetadata is a healper struct that contains the siafile health metadata fields that are cached

type Chunk 

type Chunk struct {
	Pieces [][]Piece
}

Chunk is an exported chunk. It contains exported pieces.

type FileChunk 

type FileChunk struct {
	Pieces [][]Piece
}

FileChunk is a helper struct that contains data about a chunk.

type FileData 

type FileData struct {
	Name        string
	FileSize    uint64
	MasterKey   [crypto.EntropySize]byte
	ErasureCode modules.ErasureCoder
	RepairPath  string
	PieceSize   uint64
	Mode        os.FileMode
	Deleted     bool
	UID         SiafileUID
	Chunks      []FileChunk
}

FileData is a helper struct that contains all the relevant information of a file. It simplifies passing the necessary data between modules and keeps the interface clean.

type HostPublicKey 

type HostPublicKey struct {
	PublicKey types.SiaPublicKey // public key of host
	Used      bool               // indicates if we currently use this host
}

HostPublicKey is an entry in the HostPubKey table.

func (HostPublicKey) MarshalSia 

func (hpk HostPublicKey) MarshalSia(w io.Writer) error

MarshalSia implements the encoding.SiaMarshaler interface.

func (*HostPublicKey) UnmarshalSia 

func (hpk *HostPublicKey) UnmarshalSia(r io.Reader) error

UnmarshalSia implements the encoding.SiaUnmarshaler interface.

type Metadata added in v1.4.1 

type Metadata struct {
	UniqueID SiafileUID `json:"uniqueid"` // unique identifier for file

	StaticPagesPerChunk uint8    `json:"pagesperchunk"` // number of pages reserved for storing a chunk.
	StaticVersion       [16]byte `json:"version"`       // version of the sia file format used
	FileSize            int64    `json:"filesize"`      // total size of the file
	StaticPieceSize     uint64   `json:"piecesize"`     // size of a single piece of the file
	LocalPath           string   `json:"localpath"`     // file to the local copy of the file used for repairing

	// Fields for encryption
	StaticMasterKey      []byte            `json:"masterkey"` // masterkey used to encrypt pieces
	StaticMasterKeyType  crypto.CipherType `json:"masterkeytype"`
	StaticSharingKey     []byte            `json:"sharingkey"` // key used to encrypt shared pieces
	StaticSharingKeyType crypto.CipherType `json:"sharingkeytype"`

	// Fields for partial uploads
	DisablePartialChunk bool               `json:"disablepartialchunk"` // determines whether the file should be treated like legacy files
	PartialChunks       []PartialChunkInfo `json:"partialchunks"`       // information about the partial chunk.
	HasPartialChunk     bool               `json:"haspartialchunk"`     // indicates whether this file is supposed to have a partial chunk or not

	// The following fields are the usual unix timestamps of files.
	ModTime    time.Time `json:"modtime"`    // time of last content modification
	ChangeTime time.Time `json:"changetime"` // time of last metadata modification
	AccessTime time.Time `json:"accesstime"` // time of last access
	CreateTime time.Time `json:"createtime"` // time of file creation

	// Cached fields. These fields are cached fields and are only meant to be used
	// to create FileInfos for file related API endpoints. There is no guarantee
	// that these fields are up-to-date. Neither in memory nor on disk. Updates to
	// these fields aren't persisted immediately. Instead they will only be
	// persisted whenever another method persists the metadata or when the SiaFile
	// is closed.
	//
	// CachedRedundancy is the redundancy of the file on the network and is
	// updated within the 'Redundancy' method which is periodically called by the
	// repair code.
	//
	// CachedUserRedundancy is the redundancy of the file on the network as
	// visible to the user and is updated within the 'Redundancy' method which is
	// periodically called by the repair code.
	//
	// CachedHealth is the health of the file on the network and is also
	// periodically updated by the health check loop whenever 'Health' is called.
	//
	// CachedStuckHealth is the health of the stuck chunks of the file. It is
	// updated by the health check loop. CachedExpiration is the lowest height at
	// which any of the file's contracts will expire. Also updated periodically by
	// the health check loop whenever 'Health' is called.
	//
	// CachedUploadedBytes is the number of bytes of the file that have been
	// uploaded to the network so far. Is updated every time a piece is added to
	// the siafile.
	//
	// CachedUploadProgress is the upload progress of the file and is updated
	// every time a piece is added to the siafile.
	//
	CachedRedundancy     float64           `json:"cachedredundancy"`
	CachedUserRedundancy float64           `json:"cacheduserredundancy"`
	CachedHealth         float64           `json:"cachedhealth"`
	CachedStuckHealth    float64           `json:"cachedstuckhealth"`
	CachedExpiration     types.BlockHeight `json:"cachedexpiration"`
	CachedUploadedBytes  uint64            `json:"cacheduploadedbytes"`
	CachedUploadProgress float64           `json:"cacheduploadprogress"`

	// Repair loop fields
	//
	// Health is the worst health of the file's unstuck chunks and
	// represents the percent of redundancy missing
	//
	// LastHealthCheckTime is the timestamp of the last time the SiaFile's
	// health was checked by Health()
	//
	// NumStuckChunks is the number of all the SiaFile's chunks that have
	// been marked as stuck by the repair loop. This doesn't include a potential
	// partial chunk at the end of the file though. Use 'numStuckChunks()' for
	// that instead.
	//
	// Redundancy is the cached value of the last time the file's redundancy
	// was checked
	//
	// StuckHealth is the worst health of any of the file's stuck chunks
	//
	Health              float64   `json:"health"`
	LastHealthCheckTime time.Time `json:"lasthealthchecktime"`
	NumStuckChunks      uint64    `json:"numstuckchunks"`
	Redundancy          float64   `json:"redundancy"`
	StuckHealth         float64   `json:"stuckhealth"`

	// File ownership/permission fields.
	Mode    os.FileMode `json:"mode"`    // unix filemode of the sia file - uint32
	UserID  int         `json:"userid"`  // id of the user who owns the file
	GroupID int         `json:"groupid"` // id of the group that owns the file

	// The following fields are the offsets for data that is written to disk
	// after the pubKeyTable. We reserve a generous amount of space for the
	// table and extra fields, but we need to remember those offsets in case we
	// need to resize later on.
	//
	// chunkOffset is the offset of the first chunk, forced to be a factor of
	// 4096, default 4kib
	//
	// pubKeyTableOffset is the offset of the publicKeyTable within the
	// file.
	//
	ChunkOffset       int64 `json:"chunkoffset"`
	PubKeyTableOffset int64 `json:"pubkeytableoffset"`

	// erasure code settings.
	//
	// StaticErasureCodeType specifies the algorithm used for erasure coding
	// chunks. Available types are:
	//   0 - Invalid / Missing Code
	//   1 - Reed Solomon Code
	//
	// erasureCodeParams specifies possible parameters for a certain
	// StaticErasureCodeType. Currently params will be parsed as follows:
	//   Reed Solomon Code - 4 bytes dataPieces / 4 bytes parityPieces
	//
	StaticErasureCodeType   [4]byte `json:"erasurecodetype"`
	StaticErasureCodeParams [8]byte `json:"erasurecodeparams"`
	// contains filtered or unexported fields
}

Metadata is the metadata of a SiaFile and is JSON encoded.

func LoadSiaFileMetadata added in v1.4.1 

func LoadSiaFileMetadata(path string) (Metadata, error)

LoadSiaFileMetadata is a wrapper for loadSiaFileMetadata that uses the production dependencies.

type PartialChunkInfo added in v1.4.1 

type PartialChunkInfo struct {
	ID     modules.CombinedChunkID `json:"id"`     // ID of the combined chunk
	Index  uint64                  `json:"index"`  // Index of the combined chunk within partialsSiaFile
	Offset uint64                  `json:"offset"` // Offset of partial chunk within combined chunk
	Length uint64                  `json:"length"` // Length of partial chunk within combined chunk
	Status uint8                   `json:"status"` // Status of combined chunk
}

PartialChunkInfo contains all the essential information about a partial chunk relevant to SiaFiles. A SiaFile with a partial chunk may contain 1 or 2 PartialChunkInfos since the partial chunk might be split between 2 combined chunks.

type Piece 

type Piece struct {
	HostPubKey types.SiaPublicKey // public key of the host
	MerkleRoot crypto.Hash        // merkle root of the piece
}

Piece is an exported piece. It contains a resolved public key instead of the table offset.

type RSCode 

type RSCode struct {
	// contains filtered or unexported fields
}

RSCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface.

func (*RSCode) Encode 

func (rs *RSCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSCode) EncodeShards 

func (rs *RSCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards creates the parity shards for an already sharded input.

func (*RSCode) Identifier added in v1.4.1 

func (rs *RSCode) Identifier() modules.ErasureCoderIdentifier

Identifier returns an identifier for an erasure coder which can be used to identify erasure coders of the same type, dataPieces and parityPieces.

func (*RSCode) MinPieces 

func (rs *RSCode) MinPieces() int

MinPieces return the minimum number of pieces that must be present to recover the original data.

func (*RSCode) NumPieces 

func (rs *RSCode) NumPieces() int

NumPieces returns the number of pieces returned by Encode.

func (*RSCode) Reconstruct added in v1.4.1 

func (rs *RSCode) Reconstruct(pieces [][]byte) error

Reconstruct recovers the full set of encoded shards from the provided pieces, of which at least MinPieces must be non-nil.

func (*RSCode) Recover 

func (rs *RSCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover recovers the original data from pieces and writes it to w. pieces should be identical to the slice returned by Encode (length and order must be preserved), but with missing elements set to nil.

func (*RSCode) SupportsPartialEncoding 

func (rs *RSCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns false for the basic reed-solomon encoder.

func (*RSCode) Type 

func (rs *RSCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type RSSubCode 

type RSSubCode struct {
	RSCode
	// contains filtered or unexported fields
}

RSSubCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface in a way that every crypto.SegmentSize bytes of encoded data can be recovered separately.

func (*RSSubCode) Encode 

func (rs *RSSubCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSSubCode) EncodeShards 

func (rs *RSSubCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards encodes data in a way that every segmentSize bytes of the encoded data can be decoded independently.

func (*RSSubCode) Identifier added in v1.4.1 

func (rs *RSSubCode) Identifier() modules.ErasureCoderIdentifier

Identifier returns an identifier for an erasure coder which can be used to identify erasure coders of the same type, dataPieces and parityPieces.

func (*RSSubCode) Reconstruct added in v1.4.1 

func (rs *RSSubCode) Reconstruct(pieces [][]byte) error

Reconstruct recovers the full set of encoded shards from the provided pieces, of which at least MinPieces must be non-nil.

func (*RSSubCode) Recover 

func (rs *RSSubCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover accepts encoded pieces and decodes the segment at segmentIndex. The size of the decoded data is segmentSize * dataPieces.

func (*RSSubCode) SupportsPartialEncoding 

func (rs *RSSubCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns true for the custom reed-solomon encoder.

func (*RSSubCode) Type 

func (rs *RSSubCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type SiaFile 

type SiaFile struct {
	// contains filtered or unexported fields
}

SiaFile is the disk format for files uploaded to the Sia network. It contains all the necessary information to recover a file from its hosts and allows for easy constant-time updates of the file without having to read or write the whole file.

func LoadSiaFile 

func LoadSiaFile(path string, wal *writeaheadlog.WAL) (*SiaFile, error)

LoadSiaFile is a wrapper for loadSiaFile that uses the production dependencies.

func LoadSiaFileFromReader added in v1.4.1 

func LoadSiaFileFromReader(r io.ReadSeeker, path string, wal *writeaheadlog.WAL) (*SiaFile, error)

LoadSiaFileFromReader allows loading a SiaFile from a different location that directly from disk as long as the source satisfies the SiaFileSource interface.

func LoadSiaFileFromReaderWithChunks added in v1.4.1 

func LoadSiaFileFromReaderWithChunks(r io.ReadSeeker, path string, wal *writeaheadlog.WAL) (*SiaFile, []chunk, error)

LoadSiaFileFromReaderWithChunks does not only read the header of the Siafile from disk but also the chunks which it returns separately. This is useful if the file is read from a buffer in-memory and the chunks can't be read from disk later.

func New 

func New(siaFilePath, source string, wal *writeaheadlog.WAL, erasureCode modules.ErasureCoder, masterKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode, partialsSiaFile *SiaFileSetEntry, disablePartialUpload bool) (*SiaFile, error)

New create a new SiaFile.

func (*SiaFile) AccessTime 

func (sf *SiaFile) AccessTime() time.Time

AccessTime returns the AccessTime timestamp of the file.

func (*SiaFile) AddPiece 

func (sf *SiaFile) AddPiece(pk types.SiaPublicKey, chunkIndex, pieceIndex uint64, merkleRoot crypto.Hash) error

AddPiece adds an uploaded piece to the file. It also updates the host table if the public key of the host is not already known.

func (*SiaFile) ChangeTime 

func (sf *SiaFile) ChangeTime() time.Time

ChangeTime returns the ChangeTime timestamp of the file.

func (*SiaFile) Chunk added in v1.4.1 

func (sf *SiaFile) Chunk(chunkIndex uint64) (chunk, error)

Chunk returns the chunk of a SiaFile at a given index.

func (*SiaFile) ChunkHealth 

func (sf *SiaFile) ChunkHealth(index int, offlineMap map[string]bool, goodForRenewMap map[string]bool) (float64, float64, error)

ChunkHealth returns the health of the chunk which is defined as the percent of parity pieces remaining.

func (*SiaFile) ChunkSize 

func (sf *SiaFile) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*SiaFile) CreateTime 

func (sf *SiaFile) CreateTime() time.Time

CreateTime returns the CreateTime timestamp of the file.

func (*SiaFile) Delete 

func (sf *SiaFile) Delete() error

Delete removes the file from disk and marks it as deleted. Once the file is deleted, certain methods should return an error.

func (*SiaFile) Deleted 

func (sf *SiaFile) Deleted() bool

Deleted indicates if this file has been deleted by the user.

func (*SiaFile) ErasureCode 

func (sf *SiaFile) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*SiaFile) Expiration 

func (sf *SiaFile) Expiration(contracts map[string]modules.RenterContract) types.BlockHeight

Expiration updates CachedExpiration with the lowest height at which any of the file's contracts will expire and returns the new value.

func (*SiaFile) GoodPieces added in v1.4.1 

func (sf *SiaFile) GoodPieces(chunkIndex int, offlineMap map[string]bool, goodForRenewMap map[string]bool) (uint64, uint64)

GoodPieces loops over the pieces of a chunk and tracks the number of unique pieces that are good for upload, meaning the host is online, and the number of unique pieces that are good for renew, meaning the contract is set to renew.

func (*SiaFile) GrowNumChunks added in v1.4.1 

func (sf *SiaFile) GrowNumChunks(numChunks uint64) (err error)

GrowNumChunks increases the number of chunks in the SiaFile to numChunks. If the file already contains >= numChunks chunks then GrowNumChunks is a no-op.

func (*SiaFile) HasPartialChunk added in v1.4.1 

func (sf *SiaFile) HasPartialChunk() bool

HasPartialChunk returns whether this file is supposed to have a partial chunk or not.

func (*SiaFile) Health 

func (sf *SiaFile) Health(offline map[string]bool, goodForRenew map[string]bool) (h float64, sh float64, uh float64, ush float64, nsc uint64)

Health calculates the health of the file to be used in determining repair priority. Health of the file is the lowest health of any of the chunks and is defined as the percent of parity pieces remaining. The NumStuckChunks will be calculated for the SiaFile and returned.

NOTE: The cached values of the health and stuck health will be set but not saved to disk as Health() does not write to disk. If the cached values need to be updated on disk then a metadata save method should be called in conjunction with Health()

health = 0 is full redundancy, health <= 1 is recoverable, health > 1 needs to be repaired from disk

func (*SiaFile) HostPublicKeys 

func (sf *SiaFile) HostPublicKeys() (spks []types.SiaPublicKey)

HostPublicKeys returns all the public keys of hosts the file has ever been uploaded to. That means some of those hosts might no longer be in use.

func (*SiaFile) IsIncludedPartialChunk added in v1.4.1 

func (sf *SiaFile) IsIncludedPartialChunk(chunkIndex uint64) bool

IsIncludedPartialChunk returns 'true' if the provided index points to a partial chunk which has been added to the partials sia file already.

func (*SiaFile) IsIncompletePartialChunk added in v1.4.1 

func (sf *SiaFile) IsIncompletePartialChunk(chunkIndex uint64) bool

IsIncompletePartialChunk returns 'true' if the provided index points to a partial chunk which hasn't been added to a partials siafile yet.

func (*SiaFile) LastHealthCheckTime 

func (sf *SiaFile) LastHealthCheckTime() time.Time

LastHealthCheckTime returns the LastHealthCheckTime timestamp of the file

func (*SiaFile) LocalPath 

func (sf *SiaFile) LocalPath() string

LocalPath returns the path of the local data of the file.

func (*SiaFile) MasterKey 

func (sf *SiaFile) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*SiaFile) Merge added in v1.4.1 

func (sf *SiaFile) Merge(newFile *SiaFile) (map[uint64]uint64, error)

Merge merges two PartialsSiafiles into one, returning a map which translates chunk indices in newFile to indices in sf.

func (*SiaFile) Metadata added in v1.4.1 

func (sf *SiaFile) Metadata() Metadata

Metadata returns the SiaFile's metadata, resolving any fields related to partial chunks.

func (*SiaFile) ModTime 

func (sf *SiaFile) ModTime() time.Time

ModTime returns the ModTime timestamp of the file.

func (*SiaFile) Mode 

func (sf *SiaFile) Mode() os.FileMode

Mode returns the FileMode of the SiaFile.

func (*SiaFile) NumChunks 

func (sf *SiaFile) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*SiaFile) NumStuckChunks 

func (sf *SiaFile) NumStuckChunks() uint64

NumStuckChunks returns the Number of Stuck Chunks recorded in the file's metadata

func (*SiaFile) PartialChunks added in v1.4.1 

func (sf *SiaFile) PartialChunks() []PartialChunkInfo

PartialChunks returns the partial chunk infos of the siafile.

func (*SiaFile) PieceSize 

func (sf *SiaFile) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*SiaFile) Pieces 

func (sf *SiaFile) Pieces(chunkIndex uint64) ([][]Piece, error)

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*SiaFile) Redundancy 

func (sf *SiaFile) Redundancy(offlineMap map[string]bool, goodForRenewMap map[string]bool) (r, ur float64, err error)

Redundancy returns the redundancy of the least redundant chunk. A file becomes available when this redundancy is >= 1. Assumes that every piece is unique within a file contract. -1 is returned if the file has size 0. It takes two arguments, a map of offline contracts for this file and a map that indicates if a contract is goodForRenew. The first redundancy returned is the one that should be used by the repair code and is more accurate. The other one is the redundancy presented to users.

func (*SiaFile) RemoveLastChunk added in v1.4.1 

func (sf *SiaFile) RemoveLastChunk() error

RemoveLastChunk removes the last chunk of the SiaFile and truncates the file accordingly.

func (*SiaFile) Rename 

func (sf *SiaFile) Rename(newSiaFilePath string) error

Rename changes the name of the file to a new one. To guarantee that renaming the file is atomic across all operating systems, we create a wal transaction that moves over all the chunks one-by-one and deletes the src file.

func (*SiaFile) SaveHeader added in v1.4.1 

func (sf *SiaFile) SaveHeader() error

SaveHeader saves the file's header to disk.

func (*SiaFile) SaveMetadata added in v1.4.1 

func (sf *SiaFile) SaveMetadata() error

SaveMetadata saves the file's metadata to disk.

func (*SiaFile) SaveWithChunks added in v1.4.1 

func (sf *SiaFile) SaveWithChunks(chunks []chunk) error

SaveWithChunks saves the file's header to disk and appends the raw chunks provided at the end of the file.

func (*SiaFile) SetAllStuck added in v1.4.1 

func (sf *SiaFile) SetAllStuck(stuck bool) (err error)

SetAllStuck sets the Stuck field of all chunks to stuck.

func (*SiaFile) SetChunkStatusCompleted added in v1.4.1 

func (sf *SiaFile) SetChunkStatusCompleted(pci uint64) error

SetChunkStatusCompleted sets the CombinedChunkStatus field of the metadata to completed.

func (*SiaFile) SetFileSize added in v1.4.1 

func (sf *SiaFile) SetFileSize(fileSize uint64) error

SetFileSize changes the fileSize of the SiaFile.

func (*SiaFile) SetLastHealthCheckTime added in v1.4.1 

func (sf *SiaFile) SetLastHealthCheckTime()

SetLastHealthCheckTime sets the LastHealthCheckTime in memory to the current time but does not update and write to disk.

NOTE: This call should be used in conjunction with a method that saves the SiaFile metadata

func (*SiaFile) SetLocalPath 

func (sf *SiaFile) SetLocalPath(path string) error

SetLocalPath changes the local path of the file which is used to repair the file from disk.

func (*SiaFile) SetMode 

func (sf *SiaFile) SetMode(mode os.FileMode) error

SetMode sets the filemode of the sia file.

func (*SiaFile) SetPartialChunks added in v1.4.1 

func (sf *SiaFile) SetPartialChunks(combinedChunks []modules.PartialChunk, updates []writeaheadlog.Update) error

SetPartialChunks informs the SiaFile about a partial chunk that has been saved by the partial chunk set. As such it should be exclusively called by the partial chunk set. It updates the metadata of the SiaFile and also adds a new chunk to the partial SiaFile if necessary. At the end it applies the updates of the partial chunk set, the SiaFile and the partial SiaFile atomically.

func (*SiaFile) SetPartialsSiaFile added in v1.4.1 

func (sf *SiaFile) SetPartialsSiaFile(partialsSiaFile *SiaFileSetEntry)

SetPartialsSiaFile sets the partialsSiaFile field of the SiaFile. This is usually done for non-partials SiaFiles after loading them from disk.

func (*SiaFile) SetSiaFilePath added in v1.4.1 

func (sf *SiaFile) SetSiaFilePath(path string)

SetSiaFilePath sets the path of the siafile on disk.

func (*SiaFile) SetStuck 

func (sf *SiaFile) SetStuck(index uint64, stuck bool) (err error)

SetStuck sets the Stuck field of the chunk at the given index

func (*SiaFile) SiaFilePath added in v1.4.1 

func (sf *SiaFile) SiaFilePath() string

SiaFilePath returns the siaFilePath field of the SiaFile.

func (*SiaFile) Size 

func (sf *SiaFile) Size() uint64

Size returns the file's size.

func (*SiaFile) SnapshotReader 

func (sf *SiaFile) SnapshotReader() (*SnapshotReader, error)

SnapshotReader creates a io.ReadCloser that can be used to read the raw Siafile from disk. Note that the underlying siafile holds a readlock until the SnapshotReader is closed, which means that no operations can be called to the underlying siafile which may cause it to grab a lock, because that will cause a deadlock.

Operations which require grabbing a readlock on the underlying siafile are also not okay, because if some other thread has attempted to grab a writelock on the siafile, the readlock will block and then the Close() statement may never be reached for the SnapshotReader.

TODO: Things upstream would be a lot easier if we could drop the requirement to hold a lock for the duration of the life of the snapshot reader.

func (*SiaFile) StuckChunkByIndex 

func (sf *SiaFile) StuckChunkByIndex(index uint64) (bool, error)

StuckChunkByIndex returns if the chunk at the index is marked as Stuck or not

func (*SiaFile) UID 

func (sf *SiaFile) UID() SiafileUID

UID returns a unique identifier for this file.

func (*SiaFile) UpdateAccessTime 

func (sf *SiaFile) UpdateAccessTime() error

UpdateAccessTime updates the AccessTime timestamp to the current time.

func (*SiaFile) UpdateUniqueID added in v1.4.1 

func (sf *SiaFile) UpdateUniqueID()

UpdateUniqueID creates a new random uid for the SiaFile.

func (*SiaFile) UpdateUsedHosts 

func (sf *SiaFile) UpdateUsedHosts(used []types.SiaPublicKey) error

UpdateUsedHosts updates the 'Used' flag for the entries in the pubKeyTable of the SiaFile. The keys of all used hosts should be passed to the method and the SiaFile will update the flag for hosts it knows of to 'true' and set hosts which were not passed in to 'false'.

func (*SiaFile) UploadProgressAndBytes added in v1.4.1 

func (sf *SiaFile) UploadProgressAndBytes() (float64, uint64, error)

UploadProgressAndBytes is the exported wrapped for uploadProgressAndBytes.

type SiaFileSet 

type SiaFileSet struct {
	// contains filtered or unexported fields
}

SiaFileSet is a helper struct responsible for managing the renter's siafiles in memory

func NewSiaFileSet 

func NewSiaFileSet(filesDir string, wal *writeaheadlog.WAL) *SiaFileSet

NewSiaFileSet initializes and returns a SiaFileSet

func (*SiaFileSet) AddExistingPartialsSiaFile added in v1.4.1 

func (sfs *SiaFileSet) AddExistingPartialsSiaFile(sf *SiaFile, chunks []chunk) (map[uint64]uint64, error)

AddExistingPartialsSiaFile adds an existing partial SiaFile to the set and stores it on disk. If the file already exists this will produce an error since we can't just add a suffix to it.

func (*SiaFileSet) AddExistingSiaFile added in v1.4.1 

func (sfs *SiaFileSet) AddExistingSiaFile(sf *SiaFile, chunks []chunk) error

AddExistingSiaFile adds an existing SiaFile to the set and stores it on disk. If the exact same file already exists, this is a no-op. If a file already exists with a different UID, the UID will be updated and a unique path will be chosen. If no file exists, the UID will be updated but the path remains the same.

func (*SiaFileSet) CachedFileInfo added in v1.4.1 

func (sfs *SiaFileSet) CachedFileInfo(siaPath modules.SiaPath, offline map[string]bool, goodForRenew map[string]bool, contracts map[string]modules.RenterContract) (modules.FileInfo, error)

CachedFileInfo returns a modules.FileInfo for a given file like FileInfo but instead of computing redundancy, health etc. it uses cached values.

func (*SiaFileSet) Delete 

func (sfs *SiaFileSet) Delete(siaPath modules.SiaPath) error

Delete deletes the SiaFileSetEntry's SiaFile

func (*SiaFileSet) DeleteDir added in v1.4.1 

func (sfs *SiaFileSet) DeleteDir(siaPath modules.SiaPath, deleteDir siadir.DeleteDirFunc) error

DeleteDir deletes a siadir and all the siadirs and siafiles within it recursively.

func (*SiaFileSet) Exists 

func (sfs *SiaFileSet) Exists(siaPath modules.SiaPath) bool

Exists checks to see if a file with the provided siaPath already exists in the renter

func (*SiaFileSet) FileInfo added in v1.4.1 

func (sfs *SiaFileSet) FileInfo(siaPath modules.SiaPath, offline map[string]bool, goodForRenew map[string]bool, contracts map[string]modules.RenterContract) (modules.FileInfo, error)

FileInfo returns information on a siafile. As a performance optimization, the fileInfo takes the maps returned by renter.managedContractUtilityMaps for many files at once.

func (*SiaFileSet) FileList added in v1.4.1 

func (sfs *SiaFileSet) FileList(siaPath modules.SiaPath, recursive, cached bool, offlineMap map[string]bool, goodForRenewMap map[string]bool, contractsMap map[string]modules.RenterContract) ([]modules.FileInfo, error)

FileList returns all of the files that the renter has in the folder specified by siaPath. If cached is true, this method will used cached values for health, redundancy etc.

func (*SiaFileSet) LoadPartialSiaFile added in v1.4.1 

func (sfs *SiaFileSet) LoadPartialSiaFile(siaPath modules.SiaPath) (*SiaFileSetEntry, error)

LoadPartialSiaFile loads a SiaFile containing combined chunks.

func (*SiaFileSet) Metadata added in v1.4.1 

func (sfs *SiaFileSet) Metadata(siaPath modules.SiaPath) (Metadata, error)

Metadata returns the metadata of a SiaFile.

func (*SiaFileSet) NewFromLegacyData 

func (sfs *SiaFileSet) NewFromLegacyData(fd FileData) (*SiaFileSetEntry, error)

NewFromLegacyData creates a new SiaFile from data that was previously loaded from a legacy file.

func (*SiaFileSet) NewSiaFile 

func (sfs *SiaFileSet) NewSiaFile(up modules.FileUploadParams, masterKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode) (*SiaFileSetEntry, error)

NewSiaFile create a new SiaFile, adds it to the SiaFileSet, adds the thread to the threadMap, and returns the SiaFileSetEntry. Since this method returns the SiaFileSetEntry, wherever NewSiaFile is called there should be a Close called on the SiaFileSetEntry to avoid the file being stuck in memory due the thread never being removed from the threadMap

func (*SiaFileSet) Open 

func (sfs *SiaFileSet) Open(siaPath modules.SiaPath) (*SiaFileSetEntry, error)

Open returns the siafile from the SiaFileSet for the corresponding key and adds the thread to the entry's threadMap. If the siafile is not in memory it will load it from disk

func (*SiaFileSet) OpenPartialsSiaFile added in v1.4.1 

func (sfs *SiaFileSet) OpenPartialsSiaFile(ec modules.ErasureCoder) (*SiaFileSetEntry, error)

OpenPartialsSiaFile opens a partials SiaFile and creates it if it doesn't exist yet.

func (*SiaFileSet) Rename 

func (sfs *SiaFileSet) Rename(siaPath, newSiaPath modules.SiaPath) error

Rename will move a siafile from one path to a new path. Existing entries that are already open at the old path will continue to be valid.

func (*SiaFileSet) RenameDir added in v1.4.1 

func (sfs *SiaFileSet) RenameDir(oldPath, newPath modules.SiaPath, rename siadir.RenameDirFunc) error

RenameDir renames a siadir and all the siadirs and siafiles within it recursively.

func (*SiaFileSet) SiaPath added in v1.4.1 

func (sfs *SiaFileSet) SiaPath(entry *SiaFileSetEntry) modules.SiaPath

SiaPath returns the siapath of a siafile.

type SiaFileSetEntry 

type SiaFileSetEntry struct {
	// contains filtered or unexported fields
}

SiaFileSetEntry is the exported struct that is returned to the thread accessing the SiaFile and the Entry

func (*SiaFileSetEntry) Close 

func (entry *SiaFileSetEntry) Close() error

Close will close the set entry, removing the entry from memory if there are no other entries using the siafile.

Note that 'Close' grabs a lock on the SiaFileSet, do not call this function while holding a lock on the SiafileSet - standard concurrency conventions though dictate that you should not be calling exported / capitalized functions while holding a lock anyway, but this function is particularly sensitive to that.

func (*SiaFileSetEntry) CopyEntry 

func (entry *SiaFileSetEntry) CopyEntry() (*SiaFileSetEntry, error)

CopyEntry returns a copy of the SiaFileSetEntry

func (*SiaFileSetEntry) FileSet added in v1.4.1 

func (entry *SiaFileSetEntry) FileSet() *SiaFileSet

FileSet returns the SiaFileSet of the entry.

func (SiaFileSetEntry) Snapshot added in v1.4.1 

func (sf SiaFileSetEntry) Snapshot() (*Snapshot, error)

Snapshot creates a snapshot of the SiaFile.

type SiafileUID added in v1.4.1 

type SiafileUID string

SiafileUID is a unique identifier for siafile which is used to track siafiles even after renaming them.

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is a snapshot of a SiaFile. A snapshot is a deep-copy and can be accessed without locking at the cost of being a frozen readonly representation of a siafile which only exists in memory.

func (*Snapshot) ChunkIndexByOffset 

func (s *Snapshot) ChunkIndexByOffset(offset uint64) (chunkIndex uint64, off uint64)

ChunkIndexByOffset will return the chunkIndex that contains the provided offset of a file and also the relative offset within the chunk. If the offset is out of bounds, chunkIndex will be equal to NumChunk().

func (*Snapshot) ChunkSize 

func (s *Snapshot) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*Snapshot) ErasureCode 

func (s *Snapshot) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*Snapshot) IsIncludedPartialChunk added in v1.4.1 

func (s *Snapshot) IsIncludedPartialChunk(chunkIndex uint64) (PartialChunkInfo, bool)

IsIncludedPartialChunk returns 'true' if the provided index points to a partial chunk which has been added to the partials sia file already.

func (*Snapshot) IsIncompletePartialChunk added in v1.4.1 

func (s *Snapshot) IsIncompletePartialChunk(chunkIndex uint64) bool

IsIncompletePartialChunk returns 'true' if the provided index points to a partial chunk which hasn't been added to a partials siafile yet.

func (*Snapshot) MasterKey 

func (s *Snapshot) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*Snapshot) Mode 

func (s *Snapshot) Mode() os.FileMode

Mode returns the FileMode of the file.

func (*Snapshot) NumChunks 

func (s *Snapshot) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*Snapshot) PartialChunks added in v1.4.1 

func (s *Snapshot) PartialChunks() []PartialChunkInfo

PartialChunks returns the snapshot's PartialChunks.

func (*Snapshot) PieceSize 

func (s *Snapshot) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*Snapshot) Pieces 

func (s *Snapshot) Pieces(chunkIndex uint64) [][]Piece

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*Snapshot) SiaPath 

func (s *Snapshot) SiaPath() modules.SiaPath

SiaPath returns the SiaPath of the file.

func (*Snapshot) Size 

func (s *Snapshot) Size() uint64

Size returns the size of the file.

func (*Snapshot) UID added in v1.4.1 

func (s *Snapshot) UID() SiafileUID

UID returns the UID of the file.

type SnapshotReader 

type SnapshotReader struct {
	// contains filtered or unexported fields
}

SnapshotReader is a helper type that allows reading a raw SiaFile from disk while keeping the file in memory locked.

func (*SnapshotReader) Close 

func (sfr *SnapshotReader) Close() error

Close closes the underlying file.

func (*SnapshotReader) Read 

func (sfr *SnapshotReader) Read(b []byte) (int, error)

Read calls Read on the underlying file.

func (*SnapshotReader) Stat 

func (sfr *SnapshotReader) Stat() (os.FileInfo, error)

Stat returns the FileInfo of the underlying file.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.