xm

package module
v0.0.0-...-2396041 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 7, 2015 License: BSD-3-Clause Imports: 9 Imported by: 0

README 

Experiment manager

An experiment consists of a bunch of jobs
Each job corresponds to some settings of hyperparameters for the experiment

Jobs are distributed across many machines, each of which can handle multiple jobs at once

Each job is just an executable run by a wrapper, which periodically reports progress of the wrapped executable to a central server
The server stores progress information from all jobs in a MongoDB database. This information includes time series data (e.g. loss functions, train/validation error, etc)
Each job also can give more "complete" progress information, such as snapshotted weights or optimization states. This information is not reported automatically due to size concerns, but can be downloaded if requested.

Information can be viewed by a web viewer, which should allow the user to see up-to-date progress reports, filter jobs by various criteria (e.g. lowest error or loss), and perform actions on running jobs (e.g. cancel jobs, restart jobs, and download progress)


Workflow for xm
xm -spec spec.yaml
- Enumerate jobs
- Allocate jobs to machines
- Spawn jobs
  - For each machine
    - Start SSH session for all jobs

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeWorkerArgstr 

func DecodeWorkerArgstr(s string) (*workerArgs, error)

func MakeDbSession 

func MakeDbSession(host string) (*dbSession, error)

func MakeExperiment 

func MakeExperiment(config *ExperimentConfig) *experiment

func MakeWorkerArgstr 

func MakeWorkerArgstr(job *experimentJob) string

func RunReaders 

func RunReaders(db *dbSession, jobId JobId, jobChannels *OutputChannels, readerConfigs []*JobOutputConfig)

Types

type DeploymentConfig 

type DeploymentConfig struct {
	Hosts                    []string
	JobsPerHost              int    `yaml:"jobs_per_host"`
	HostUsername             string `yaml:"host_username"`
	HostWorkerExecutablePath string `yaml:"host_worker_executable_path"`
	ControllerHost           string `yaml:"controller_host"`
}

type ExperimentConfig 

type ExperimentConfig struct {
	Name       string
	Deployment *DeploymentConfig
	Jobs       *JobConfig
}

type ExperimentId 

type ExperimentId struct {
	bson.ObjectId
}

type JobConfig 

type JobConfig struct {
	CommandTemplate string `yaml:"command_template"`
	Params          []*JobParamConfig
	Outputs         []*JobOutputConfig
}

type JobHostAssignment 

type JobHostAssignment [][]*experimentJob

func AssignJobsToHosts 

func AssignJobsToHosts(
	jobs []*experimentJob,
	deployment *DeploymentConfig) JobHostAssignment

type JobId 

type JobId struct {
	bson.ObjectId
}

type JobOutputConfig 

type JobOutputConfig struct {
	Name   string
	Type   string
	Source string
}

type JobParamConfig 

type JobParamConfig struct {
	Name   string
	Values []string
}

type OutputChannels 

type OutputChannels struct {
	Stdout chan string
	Stderr chan string
	Result chan error
}

func MakeOutputChannels 

func MakeOutputChannels() *OutputChannels

type OutputReader 

type OutputReader interface {
	Consume(line string) error
}

type ParamInstantiation 

type ParamInstantiation map[string]string

func InstantiateParams 

func InstantiateParams(paramSpecs []*JobParamConfig) []ParamInstantiation

Takes parameter specifications (names and possible values) and produces a list of instantiations of those parameters

type ReaderSource 

type ReaderSource int
const (
	INVALID ReaderSource = iota
	STDOUT
	STDERR
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
xm
xmworker

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.