siafile

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 1, 2019 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrPathOverload is an error when a file already exists at that location
	ErrPathOverload = errors.New("a file already exists at that location")
	// ErrUnknownPath is an error when a file cannot be found with the given path
	ErrUnknownPath = errors.New("no file known with that path")
	// ErrUnknownThread is an error when a SiaFile is trying to be closed by a
	// thread that is not in the threadMap
	ErrUnknownThread = errors.New("thread should not be calling Close(), does not have control of the siafile")
)
View Source 
var (

	// RemoteRepairDownloadThreshold defines the threshold in percent under
	// which the renter starts repairing a file that is not available on disk.
	RemoteRepairDownloadThreshold = build.Select(build.Var{
		Dev:      0.25,
		Standard: 0.25,
		Testing:  0.25,
	}).(float64)
)

Functions

func ApplyUpdates 

func ApplyUpdates(updates ...writeaheadlog.Update) error

ApplyUpdates is a wrapper for applyUpdates that uses the production dependencies.

func ExtractSegment 

func ExtractSegment(pieces [][]byte, segmentIndex int, segmentSize uint64) [][]byte

ExtractSegment is a convenience method that extracts the data of the segment at segmentIndex from pieces.

func IsSiaFileUpdate 

func IsSiaFileUpdate(update writeaheadlog.Update) bool

IsSiaFileUpdate is a helper method that makes sure that a wal update belongs to the SiaFile package.

func NewRSCode 

func NewRSCode(nData, nParity int) (modules.ErasureCoder, error)

NewRSCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

func NewRSSubCode 

func NewRSSubCode(nData, nParity int, segmentSize uint64) (modules.ErasureCoder, error)

NewRSSubCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

Types

type BubbledMetadata 

type BubbledMetadata struct {
	Health              float64
	LastHealthCheckTime time.Time
	ModTime             time.Time
	NumStuckChunks      uint64
	RecentRepairTime    time.Time
	Redundancy          float64
	Size                uint64
	StuckHealth         float64
}

BubbledMetadata is the metadata of a siafile that gets bubbled

type CachedHealthMetadata 

type CachedHealthMetadata struct {
	Health      float64
	Redundancy  float64
	StuckHealth float64
}

CachedHealthMetadata is a healper struct that contains the siafile health metadata fields that are cached

type Chunk 

type Chunk struct {
	Pieces [][]Piece
}

Chunk is an exported chunk. It contains exported pieces.

type FileChunk 

type FileChunk struct {
	Pieces [][]Piece
}

FileChunk is a helper struct that contains data about a chunk.

type FileData 

type FileData struct {
	Name        string
	FileSize    uint64
	MasterKey   [crypto.EntropySize]byte
	ErasureCode modules.ErasureCoder
	RepairPath  string
	PieceSize   uint64
	Mode        os.FileMode
	Deleted     bool
	UID         string
	Chunks      []FileChunk
}

FileData is a helper struct that contains all the relevant information of a file. It simplifies passing the necessary data between modules and keeps the interface clean.

type HostPublicKey 

type HostPublicKey struct {
	PublicKey types.SiaPublicKey // public key of host
	Used      bool               // indicates if we currently use this host
}

HostPublicKey is an entry in the HostPubKey table.

func (HostPublicKey) MarshalSia 

func (hpk HostPublicKey) MarshalSia(w io.Writer) error

MarshalSia implements the encoding.SiaMarshaler interface.

func (*HostPublicKey) UnmarshalSia 

func (hpk *HostPublicKey) UnmarshalSia(r io.Reader) error

UnmarshalSia implements the encoding.SiaUnmarshaler interface.

type Piece 

type Piece struct {
	HostPubKey types.SiaPublicKey // public key of the host
	MerkleRoot crypto.Hash        // merkle root of the piece
}

Piece is an exported piece. It contains a resolved public key instead of the table offset.

type RSCode 

type RSCode struct {
	// contains filtered or unexported fields
}

RSCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface.

func (*RSCode) Encode 

func (rs *RSCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSCode) EncodeShards 

func (rs *RSCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards creates the parity shards for an already sharded input.

func (*RSCode) MinPieces 

func (rs *RSCode) MinPieces() int

MinPieces return the minimum number of pieces that must be present to recover the original data.

func (*RSCode) NumPieces 

func (rs *RSCode) NumPieces() int

NumPieces returns the number of pieces returned by Encode.

func (*RSCode) Recover 

func (rs *RSCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover recovers the original data from pieces and writes it to w. pieces should be identical to the slice returned by Encode (length and order must be preserved), but with missing elements set to nil.

func (*RSCode) SupportsPartialEncoding 

func (rs *RSCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns false for the basic reed-solomon encoder.

func (*RSCode) Type 

func (rs *RSCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type RSSubCode 

type RSSubCode struct {
	RSCode
	// contains filtered or unexported fields
}

RSSubCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface in a way that every crypto.SegmentSize bytes of encoded data can be recovered separately.

func (*RSSubCode) Encode 

func (rs *RSSubCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSSubCode) EncodeShards 

func (rs *RSSubCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards encodes data in a way that every segmentSize bytes of the encoded data can be decoded independently.

func (*RSSubCode) Recover 

func (rs *RSSubCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover accepts encoded pieces and decodes the segment at segmentIndex. The size of the decoded data is segmentSize * dataPieces.

func (*RSSubCode) SupportsPartialEncoding 

func (rs *RSSubCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns true for the custom reed-solomon encoder.

func (*RSSubCode) Type 

func (rs *RSSubCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type SiaFile 

type SiaFile struct {
	// contains filtered or unexported fields
}

SiaFile is the disk format for files uploaded to the Sia network. It contains all the necessary information to recover a file from its hosts and allows for easy constant-time updates of the file without having to read or write the whole file.

func LoadSiaFile 

func LoadSiaFile(path string, wal *writeaheadlog.WAL) (*SiaFile, error)

LoadSiaFile is a wrapper for loadSiaFile that uses the production dependencies.

func New 

func New(siaPath modules.SiaPath, siaFilePath, source string, wal *writeaheadlog.WAL, erasureCode modules.ErasureCoder, masterKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode) (*SiaFile, error)

New create a new SiaFile.

func (*SiaFile) AccessTime 

func (sf *SiaFile) AccessTime() time.Time

AccessTime returns the AccessTime timestamp of the file.

func (*SiaFile) AddPiece 

func (sf *SiaFile) AddPiece(pk types.SiaPublicKey, chunkIndex, pieceIndex uint64, merkleRoot crypto.Hash) error

AddPiece adds an uploaded piece to the file. It also updates the host table if the public key of the host is not already known.

func (*SiaFile) ChangeTime 

func (sf *SiaFile) ChangeTime() time.Time

ChangeTime returns the ChangeTime timestamp of the file.

func (*SiaFile) ChunkHealth 

func (sf *SiaFile) ChunkHealth(index int, offlineMap map[string]bool, goodForRenewMap map[string]bool) float64

ChunkHealth returns the health of the chunk which is defined as the percent of parity pieces remaining.

func (*SiaFile) ChunkIndexByOffset 

func (sf *SiaFile) ChunkIndexByOffset(offset uint64) (chunkIndex uint64, off uint64)

ChunkIndexByOffset will return the chunkIndex that contains the provided offset of a file and also the relative offset within the chunk. If the offset is out of bounds, chunkIndex will be equal to NumChunk().

func (*SiaFile) ChunkSize 

func (sf *SiaFile) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*SiaFile) CreateTime 

func (sf *SiaFile) CreateTime() time.Time

CreateTime returns the CreateTime timestamp of the file.

func (*SiaFile) Delete 

func (sf *SiaFile) Delete() error

Delete removes the file from disk and marks it as deleted. Once the file is deleted, certain methods should return an error.

func (*SiaFile) Deleted 

func (sf *SiaFile) Deleted() bool

Deleted indicates if this file has been deleted by the user.

func (*SiaFile) ErasureCode 

func (sf *SiaFile) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*SiaFile) Expiration 

func (sf *SiaFile) Expiration(contracts map[string]modules.RenterContract) types.BlockHeight

Expiration returns the lowest height at which any of the file's contracts will expire.

func (*SiaFile) Health 

func (sf *SiaFile) Health(offline map[string]bool, goodForRenew map[string]bool) (float64, float64, uint64)

Health calculates the health of the file to be used in determining repair priority. Health of the file is the lowest health of any of the chunks and is defined as the percent of parity pieces remaining. Additionally the NumStuckChunks will be updated for the SiaFile and returned

health = 0 is full redundancy, health <= 1 is recoverable, health > 1 needs to be repaired from disk

func (*SiaFile) HealthPercentage 

func (sf *SiaFile) HealthPercentage(health float64) float64

HealthPercentage returns the health in a more human understandable format out of 100%

func (*SiaFile) HostPublicKeys 

func (sf *SiaFile) HostPublicKeys() (spks []types.SiaPublicKey)

HostPublicKeys returns all the public keys of hosts the file has ever been uploaded to. That means some of those hosts might no longer be in use.

func (*SiaFile) LastHealthCheckTime 

func (sf *SiaFile) LastHealthCheckTime() time.Time

LastHealthCheckTime returns the LastHealthCheckTime timestamp of the file

func (*SiaFile) LocalPath 

func (sf *SiaFile) LocalPath() string

LocalPath returns the path of the local data of the file.

func (*SiaFile) MarkAllHealthyChunksAsUnstuck 

func (sf *SiaFile) MarkAllHealthyChunksAsUnstuck(offline map[string]bool, goodForRenew map[string]bool) (err error)

MarkAllHealthyChunksAsUnstuck marks all health chunks as unstuck in the siafile

func (*SiaFile) MarkAllUnhealthyChunksAsStuck 

func (sf *SiaFile) MarkAllUnhealthyChunksAsStuck(offline map[string]bool, goodForRenew map[string]bool) (err error)

MarkAllUnhealthyChunksAsStuck marks all unhealthy chunks as stuck in the siafile

func (*SiaFile) MasterKey 

func (sf *SiaFile) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*SiaFile) ModTime 

func (sf *SiaFile) ModTime() time.Time

ModTime returns the ModTime timestamp of the file.

func (*SiaFile) Mode 

func (sf *SiaFile) Mode() os.FileMode

Mode returns the FileMode of the SiaFile.

func (*SiaFile) NumChunks 

func (sf *SiaFile) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*SiaFile) NumStuckChunks 

func (sf *SiaFile) NumStuckChunks() uint64

NumStuckChunks returns the Number of Stuck Chunks recorded in the file's metadata

func (*SiaFile) PieceSize 

func (sf *SiaFile) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*SiaFile) Pieces 

func (sf *SiaFile) Pieces(chunkIndex uint64) ([][]Piece, error)

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*SiaFile) RecentRepairTime 

func (sf *SiaFile) RecentRepairTime() time.Time

RecentRepairTime returns the RecentRepairTime timestamp of the file

func (*SiaFile) Redundancy 

func (sf *SiaFile) Redundancy(offlineMap map[string]bool, goodForRenewMap map[string]bool) float64

Redundancy returns the redundancy of the least redundant chunk. A file becomes available when this redundancy is >= 1. Assumes that every piece is unique within a file contract. -1 is returned if the file has size 0. It takes two arguments, a map of offline contracts for this file and a map that indicates if a contract is goodForRenew.

func (*SiaFile) Rename 

func (sf *SiaFile) Rename(newSiaPath modules.SiaPath, newSiaFilePath string) error

Rename changes the name of the file to a new one.

func (*SiaFile) SetLocalPath 

func (sf *SiaFile) SetLocalPath(path string) error

SetLocalPath changes the local path of the file which is used to repair the file from disk.

func (*SiaFile) SetMode 

func (sf *SiaFile) SetMode(mode os.FileMode) error

SetMode sets the filemode of the sia file.

func (*SiaFile) SetStuck 

func (sf *SiaFile) SetStuck(index uint64, stuck bool) (err error)

SetStuck sets the Stuck field of the chunk at the given index

func (*SiaFile) SiaPath 

func (sf *SiaFile) SiaPath() modules.SiaPath

SiaPath returns the file's sia path.

func (*SiaFile) Size 

func (sf *SiaFile) Size() uint64

Size returns the file's size.

func (*SiaFile) Snapshot 

func (sf *SiaFile) Snapshot() *Snapshot

Snapshot creates a snapshot of the SiaFile.

func (*SiaFile) SnapshotReader 

func (sf *SiaFile) SnapshotReader() (*SnapshotReader, error)

SnapshotReader creates a io.ReadCloser that can be used to read the raw Siafile from disk.

func (*SiaFile) StuckChunkByIndex 

func (sf *SiaFile) StuckChunkByIndex(index uint64) bool

StuckChunkByIndex returns if the chunk at the index is marked as Stuck or not

func (*SiaFile) UID 

func (sf *SiaFile) UID() string

UID returns a unique identifier for this file.

func (*SiaFile) UpdateAccessTime 

func (sf *SiaFile) UpdateAccessTime() error

UpdateAccessTime updates the AccessTime timestamp to the current time.

func (*SiaFile) UpdateCachedHealthMetadata 

func (sf *SiaFile) UpdateCachedHealthMetadata(metadata CachedHealthMetadata) error

UpdateCachedHealthMetadata updates the siafile metadata fields that are the cached health values

func (*SiaFile) UpdateLastHealthCheckTime 

func (sf *SiaFile) UpdateLastHealthCheckTime() error

UpdateLastHealthCheckTime updates the LastHealthCheckTime timestamp to the current time.

func (*SiaFile) UpdateRecentRepairTime 

func (sf *SiaFile) UpdateRecentRepairTime() error

UpdateRecentRepairTime updates the RecentRepairTime timestamp to the current time.

func (*SiaFile) UpdateUsedHosts 

func (sf *SiaFile) UpdateUsedHosts(used []types.SiaPublicKey) error

UpdateUsedHosts updates the 'Used' flag for the entries in the pubKeyTable of the SiaFile. The keys of all used hosts should be passed to the method and the SiaFile will update the flag for hosts it knows of to 'true' and set hosts which were not passed in to 'false'.

func (*SiaFile) UploadProgress 

func (sf *SiaFile) UploadProgress() float64

UploadProgress indicates what percentage of the file (plus redundancy) has been uploaded. Note that a file may be Available long before UploadProgress reaches 100%, and UploadProgress may report a value greater than 100%.

func (*SiaFile) UploadedBytes 

func (sf *SiaFile) UploadedBytes() uint64

UploadedBytes indicates how many bytes of the file have been uploaded via current file contracts. Note that this includes padding and redundancy, so uploadedBytes can return a value much larger than the file's original filesize.

type SiaFileSet 

type SiaFileSet struct {
	// contains filtered or unexported fields
}

SiaFileSet is a helper struct responsible for managing the renter's siafiles in memory

func NewSiaFileSet 

func NewSiaFileSet(filesDir string, wal *writeaheadlog.WAL) *SiaFileSet

NewSiaFileSet initializes and returns a SiaFileSet

func (*SiaFileSet) Delete 

func (sfs *SiaFileSet) Delete(siaPath modules.SiaPath) error

Delete deletes the SiaFileSetEntry's SiaFile

func (*SiaFileSet) Exists 

func (sfs *SiaFileSet) Exists(siaPath modules.SiaPath) bool

Exists checks to see if a file with the provided siaPath already exists in the renter

func (*SiaFileSet) NewFromLegacyData 

func (sfs *SiaFileSet) NewFromLegacyData(fd FileData) (*SiaFileSetEntry, error)

NewFromLegacyData creates a new SiaFile from data that was previously loaded from a legacy file.

func (*SiaFileSet) NewSiaFile 

func (sfs *SiaFileSet) NewSiaFile(up modules.FileUploadParams, masterKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode) (*SiaFileSetEntry, error)

NewSiaFile create a new SiaFile, adds it to the SiaFileSet, adds the thread to the threadMap, and returns the SiaFileSetEntry. Since this method returns the SiaFileSetEntry, wherever NewSiaFile is called there should be a Close called on the SiaFileSetEntry to avoid the file being stuck in memory due the thread never being removed from the threadMap

func (*SiaFileSet) Open 

func (sfs *SiaFileSet) Open(siaPath modules.SiaPath) (*SiaFileSetEntry, error)

Open returns the siafile from the SiaFileSet for the corresponding key and adds the thread to the entry's threadMap. If the siafile is not in memory it will load it from disk

func (*SiaFileSet) Rename 

func (sfs *SiaFileSet) Rename(siaPath, newSiaPath modules.SiaPath) error

Rename will move a siafile from one path to a new path. Existing entries that are already open at the old path will continue to be valid.

type SiaFileSetEntry 

type SiaFileSetEntry struct {
	// contains filtered or unexported fields
}

SiaFileSetEntry is the exported struct that is returned to the thread accessing the SiaFile and the Entry

func (*SiaFileSetEntry) Close 

func (entry *SiaFileSetEntry) Close() error

Close will close the set entry, removing the entry from memory if there are no other entries using the siafile.

Note that 'Close' grabs a lock on the SiaFileSet, do not call this function while holding a lock on the SiafileSet - standard concurrency conventions though dictate that you should not be calling exported / capitalized functions while holding a lock anyway, but this function is particularly sensitive to that.

func (*SiaFileSetEntry) CopyEntry 

func (entry *SiaFileSetEntry) CopyEntry() *SiaFileSetEntry

CopyEntry returns a copy of the SiaFileSetEntry

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is a snapshot of a SiaFile. A snapshot is a deep-copy and can be accessed without locking at the cost of being a frozen readonly representation of a siafile which only exists in memory.

func (*Snapshot) ChunkIndexByOffset 

func (s *Snapshot) ChunkIndexByOffset(offset uint64) (chunkIndex uint64, off uint64)

ChunkIndexByOffset will return the chunkIndex that contains the provided offset of a file and also the relative offset within the chunk. If the offset is out of bounds, chunkIndex will be equal to NumChunk().

func (*Snapshot) ChunkSize 

func (s *Snapshot) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*Snapshot) ErasureCode 

func (s *Snapshot) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*Snapshot) MasterKey 

func (s *Snapshot) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*Snapshot) Mode 

func (s *Snapshot) Mode() os.FileMode

Mode returns the FileMode of the file.

func (*Snapshot) NumChunks 

func (s *Snapshot) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*Snapshot) PieceSize 

func (s *Snapshot) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*Snapshot) Pieces 

func (s *Snapshot) Pieces(chunkIndex uint64) ([][]Piece, error)

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*Snapshot) SiaPath 

func (s *Snapshot) SiaPath() modules.SiaPath

SiaPath returns the SiaPath of the file.

func (*Snapshot) Size 

func (s *Snapshot) Size() uint64

Size returns the size of the file.

type SnapshotReader 

type SnapshotReader struct {
	// contains filtered or unexported fields
}

SnapshotReader is a helper type that allows reading a raw SiaFile from disk while keeping the file in memory locked.

func (*SnapshotReader) Close 

func (sfr *SnapshotReader) Close() error

Close closes the underlying file.

func (*SnapshotReader) Read 

func (sfr *SnapshotReader) Read(b []byte) (int, error)

Read calls Read on the underlying file.

func (*SnapshotReader) Stat 

func (sfr *SnapshotReader) Stat() (os.FileInfo, error)

Stat returns the FileInfo of the underlying file.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.