Documentation
¶
Overview ¶
The sync package implements Kelda's sync algorithm. It handles syncing local files to a staging location within the container, and syncing from the staging files to the source final destination.
There are three types of files:
- SourceFiles -- These are files that exist on the user's machine.
- MirrorFiles -- These files keep mirror the SourceFiles in the container. This way, the sync to DestinationFiles occurs on the local filesystem, so the length of the "in between" state won't be impacted by network latency between the user's machine and the container.
- DestinationFiles -- These are the final synced files in the container.
The sync server running on the user's laptop is responsible for syncing SourceFiles to MirrorFiles, as well as sending the SyncConfig to the container.
The container then uses the MirrorFiles and SyncConfig to install the files into the proper places within the container.
The sync algorithm only deals with files. Empty directories aren't synced. This lets us cleanly sync files into directories that contain files that shouldn't be affected by syncing.
Index ¶
- func AppliesTo(rule dev.SyncRule, path string) bool
- func HashFile(path string) (string, error)
- func VersionSyncConfig(cfg dev.SyncConfig) string
- type DestinationFile
- type FileAttributes
- type LocalSnapshot
- type MirrorFile
- type MirrorSnapshot
- type MirrorTracker
- type SourceFile
- type SyncedTracker
- func (tracker SyncedTracker) Diff(mirror *MirrorTracker, syncConfig dev.SyncConfig) (toCopy []DestinationFile, toRemove []string)
- func (tracker SyncedTracker) Files() (files []DestinationFile)
- func (tracker SyncedTracker) Removed(path string)
- func (tracker SyncedTracker) Synced(f DestinationFile)
- func (tracker SyncedTracker) Version(syncConfig dev.SyncConfig) Version
- type Version
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AppliesTo ¶
AppliesTo returns whether the given path applies to the sync rule. In other words, whether it matches `rule.From`, and isn't ignored.
func VersionSyncConfig ¶
func VersionSyncConfig(cfg dev.SyncConfig) string
VersionSyncConfig returns a string representing the contents of the sync config.
Types ¶
type DestinationFile ¶
type DestinationFile struct {
// Path is the path of the synced file on the filesystem after the sync
// rules have been applied to the source file.
SyncDestinationPath string
// SyncSourcePath is the normalized path to the file on the user's machine.
// This path doesn't exist in the container, but is used to calculate the
// SyncDestinationPath and the version of the synced files.
SyncSourcePath string
FileAttributes
// SyncRules are the sync rules that were applied to sync this file.
// Multiple sync rules can apply to the same destination if they have the
// same source. This pattern is common as a way of setting `triggerInit` on
// a specific file within a directory.
SyncRules []*dev.SyncRule
}
DestinationFile contains metadata tracking files that were synced from user machines.
type FileAttributes ¶
type FileAttributes struct {
// ContentsHash is the sha512 hash of the contents of the file.
ContentsHash string
// Mode is the file mode of the file.
Mode os.FileMode
// ModTime is the time of the last file modification.
ModTime time.Time
}
FileAttributes contains some metadata used to compare whether two files are equal.
func (FileAttributes) Equal ¶
func (f FileAttributes) Equal(otherFile FileAttributes) bool
Equal returns whether two files are equal (i.e. whether a sync is necessary).
func (FileAttributes) Version ¶
func (f FileAttributes) Version() string
Version returns a string representing the contents of the metadata.
type LocalSnapshot ¶
type LocalSnapshot map[string]SourceFile
LocalSnapshot is a collection of all source files.
func SnapshotSource ¶
func SnapshotSource(syncConfig dev.SyncConfig, relativeTo string) (LocalSnapshot, error)
SnapshotSource returns the information on the files that are tracked by the syncConfig.
func (LocalSnapshot) Diff ¶
func (local LocalSnapshot) Diff(mirror MirrorSnapshot) (toMirror []SourceFile, toRemove []string)
Diff returns the files that need to be create, updated, or removed from the file mirror.
type MirrorFile ¶
type MirrorFile SourceFile
A MirrorFile is a copy of a SourceFile that lives in the container rather than on the user's machine. The SyncSourcePath and FileAttributes of a properly mirrored file are exactly equal. The ContentsPath may differ since the file is living on a different filesystem.
type MirrorSnapshot ¶
type MirrorSnapshot map[string]MirrorFile
MirrorSnapshot is a collection of all mirrored files.
func UnmarshalMirrorSnapshot ¶
func UnmarshalMirrorSnapshot(pbSnapshot dev.MirrorSnapshot) (MirrorSnapshot, error)
UnmarshalMirrorSnapshot parses the protobuf version of the mirror into the go type that's used in the rest of the code.
func (MirrorSnapshot) Add ¶
func (mirror MirrorSnapshot) Add(f MirrorFile)
Add updates the MirrorSnapshot.
func (MirrorSnapshot) Marshal ¶
func (mirror MirrorSnapshot) Marshal() (dev.MirrorSnapshot, error)
Marshal converts the MirrorSnapshot into the protobuf format.
type MirrorTracker ¶
type MirrorTracker struct {
// contains filtered or unexported fields
}
MirrorTracker tracks the files that have been mirrored from the user's local machine to the container.
func NewMirrorTracker ¶
func NewMirrorTracker() *MirrorTracker
NewMirrorTracker returns a new MirrorTracker.
func (*MirrorTracker) Get ¶
func (tracker *MirrorTracker) Get(localPath string) (MirrorFile, bool)
func (*MirrorTracker) GetSnapshot ¶
func (tracker *MirrorTracker) GetSnapshot() MirrorSnapshot
func (*MirrorTracker) Mirrored ¶
func (tracker *MirrorTracker) Mirrored(f MirrorFile)
Mirrored updates the information for f.
func (*MirrorTracker) Removed ¶
func (tracker *MirrorTracker) Removed(path string)
Mirrored updates the tracker to reflect that `path` was removed from the user's machine.
type SourceFile ¶
type SourceFile struct {
// ContentsPath is the path to the file that can be opened by the Kelda
// process. It may be either a relative path or an absolute path.
ContentsPath string
// SyncSourcePath is the normalized path in terms of the rules in the
// SyncConfig.
// For example, a file would have a ContentsPath of `/service/index.js` and
// SyncSourcePath of `index.js` if the file is being synced by a SyncConfig
// at `/service/kelda.yaml` with a rule.From of `index.js`.
SyncSourcePath string
// FileAttributes contains metadata that's used for comparing equality
// between SourceFiles and MirrorFiles.
FileAttributes
}
A SourceFile is a file that exists on the user's machine.
type SyncedTracker ¶
type SyncedTracker map[string]DestinationFile
SyncedTracker tracks files that have been synced from the mirror files to their final destination according to the user's sync config.
func NewSyncedTracker ¶
func NewSyncedTracker() *SyncedTracker
NewSyncedTracker creates a new SyncedTracker.
func (SyncedTracker) Diff ¶
func (tracker SyncedTracker) Diff(mirror *MirrorTracker, syncConfig dev.SyncConfig) ( toCopy []DestinationFile, toRemove []string)
Diff returns the file operations necessary to make the tracked files compliant with the sync config. * Files that are newly added should be copied. * Files that have been updated (i.e. have the wrong version) should be resynced. * And files that are no longer tracked on the user's machine should be removed.
func (SyncedTracker) Files ¶
func (tracker SyncedTracker) Files() (files []DestinationFile)
Files returns the tracked files as a list.
func (SyncedTracker) Removed ¶
func (tracker SyncedTracker) Removed(path string)
Removed updates SyncedTracker to reflect that `path` should no longer be tracked. This can happen if the source file was removed on the user's machine, or if the sync config was changed so that the source file is no longer tracked.
func (SyncedTracker) Synced ¶
func (tracker SyncedTracker) Synced(f DestinationFile)
Synced updates SyncedTracker to reflect that `f` was added or updated.
func (SyncedTracker) Version ¶
func (tracker SyncedTracker) Version(syncConfig dev.SyncConfig) Version
Version returns the version of the files that have been synced. It assumes that the `syncConfig` was properly applied to the tracked SyncSourcePaths.
type Version ¶
type Version struct {
SyncConfig dev.SyncConfig
LocalFiles LocalSnapshot
}
Version represents the version of a set of files. This is used to calculate the local version of a set of files, and the running version. When used as the running version, we assume that the sync rules in the SyncConfig were properly applied to the LocalFiles.
Source Files
Directories