README
¶
Docker Tracker
Introduction
Docker Tracker implements the JobTracker interface from Go drmaa2os. It allows to use Docker as a backend for managing jobs as containers from the DRMAA2 interface. It can also be used directly. The package also contains an implementation of the Monitorer interface so that it can be used in a DRMAA2 monitoring session.
Functionality
Docker Tracker is an API that enables the use of Docker as a backend for managing jobs as containers through the DRMAA2 interface. It provides an implementation of the JobTracker interface from Go drmaa2os, allowing easy job control and management within Docker.
The functionality of Docker Tracker includes:
-
Starting Docker containers using the DRMAA2 JobTemplate which requires a JobCategory (corresponding to a Docker image) and a RemoteCommand (the command to be executed within the Docker image).
-
Providing job control functions such as suspend, resume, and terminate for managing Docker containers.
-
Mapping DRMAA2 Job Control commands to corresponding Docker commands for seamless integration.
-
Mapping DRMAA2 State to Docker State to provide a consistent view of the container's status.
-
Allowing the removal of installed containers through the DeleteJob command, which is equivalent to docker rm.
-
Supporting Job Template Mapping to efficiently map between the JobTemplate and the Docker container configuration request.
-
Implementing Job Array functionality by creating multiple tasks sequentially in a loop, since Docker does not support Array Jobs natively.
Please note that Docker Tracker does not pull container images automatically, and the required images must be pulled before using the tool. Additionally, some DRMAA2 functionalities, such as Hold and Release, are not supported in Docker Tracker due to limitations in Docker.
For the case a Docker image needs to be pulled programmatically the OS process backend can be used.
Basic Usage
A JobTemplate requires:
* JobCategory -> which maps to an installed Docker image
* RemoteCommand -> which is the command executed within the given Docker image
Job Control Mapping
| DRMAA2 Job Control | Docker |
|---|---|
| Suspend | Signal: SIGSTOP |
| Resume | Signal: SIGCONT |
| Terminate | Signal: SIGKILL |
| Hold | Unsupported |
| Release | Unsupported |
State Mapping
| DRMAA2 State | Docker State |
|---|---|
| Failed | OOMKilled |
| Failed or Done depending on exit code | Exited |
| Failed or Done depending on exit code | Dead |
| Suspended | Paused |
| Running | Running |
| Queued | Restarting |
| Undetermined | other |
DeleteJob
DeleteJob equals docker rm and is removing an installed container. It must be terminated / finished before.
Job Template Mapping
Mapping between the job template and the Docker container config request:
| DRMAA2 JobTemplate | Docker Container Config Request |
|---|---|
| RemoteCommand | Cmd[0] |
| Args | Cmd[1:] |
| JobCategory | Image |
| CandidateMachines[0] | Hostname |
| WorkingDir | WorkingDir |
| JobEnvironment (k: v) | Env ("k=v") |
| StageInFiles | key: hostPath -v hostPath:containerPath |
| ErrorPath | Writes stderr into a local file (not a file in the container). |
| OutputPath | Writes stdout into a local file (not a file in the container). |
| Extension: "user" | User / must exist in container if set |
| Extension: "exposedPorts" | -p / multiple entries are splitted with "," |
| Extension: "net" | --net / like "host" |
| Extension: "privileged" | --privileged / "true" when enabled, default "false" |
| Extension: "restart" | --restart / like "unless-stopped", default "no" / use with care |
| Extension: "ipc" | Value like "host": --ipc "host" |
| Extension: "uts" | --uts "host" |
| Extension: "pid" | --pid "host" |
| Extension: "rm" | --rm "true" or "TRUE" |
| Extension: "gpus" | Sets GPU device request for nvidia driver. Value must be "all" or a number |
| Extension: "shm-size" | Human-readable string representing an amount of RAM in bytes, kibibytes, mebibytes, gibibytes, or tebibytes |
| Extension: "ulimit" | Comma separted list of ulimits: "memlock=-1,stack=67108864" |
Check convert.go for more details. Add your own extention and create a pull request.
If more extensions needed just open an issue.
Note that the image must be available (pulled already)!
Job Info Mapping
| DRMAA2 JobInfo | Docker Container Information |
|---|---|
| ID | Container ID |
| Slots | 1 (fixed value) |
| AllocatedMachines | Config.Hostname |
| ExitStatus | State.ExitCode |
| FinishTime | State.FinishedAt |
| DispatchTime | State.StartedAt |
| State | Mapped from Container State |
| SubmissionTime | Container Creation Time |
| JobOwner | Config.User |
| ExtensionList (workingdir) | Config.WorkingDir |
| ExtensionList (commandline) | Config.Cmd (joined as a string) |
| ExtensionList (category) | Config.Image |
Job Arrays
Since Array Jobs are not supported by Docker the job array functionality is implemented by creating n tasks sequentially in a loop. The array job ID contains all IDs of the created Docker containers.
Documentation
¶
Index ¶
- Constants
- func NewAllocator() *allocator
- func ReadJobTemplateFromLabel(containerID string) (drmaa2interface.JobTemplate, error)
- type DockerTracker
- func (dt *DockerTracker) AddArrayJob(jt drmaa2interface.JobTemplate, begin int, end int, step int, maxParallel int) (string, error)
- func (dt *DockerTracker) AddJob(jt drmaa2interface.JobTemplate) (string, error)
- func (dt *DockerTracker) CloseMonitoringSession(name string) error
- func (dt *DockerTracker) DeleteJob(jobid string) error
- func (dt *DockerTracker) GetAllJobIDs(filter *drmaa2interface.JobInfo) ([]string, error)
- func (dt *DockerTracker) GetAllMachines(filter []string) ([]drmaa2interface.Machine, error)
- func (dt *DockerTracker) GetAllQueueNames(filter []string) ([]string, error)
- func (dt *DockerTracker) JobControl(jobid, state string) error
- func (dt *DockerTracker) JobInfo(jobid string) (ji drmaa2interface.JobInfo, err error)
- func (dt *DockerTracker) JobInfoFromMonitor(id string) (ji drmaa2interface.JobInfo, err error)
- func (dt *DockerTracker) JobState(jobid string) (drmaa2interface.JobState, string, error)
- func (dt *DockerTracker) JobTemplate(jobID string) (drmaa2interface.JobTemplate, error)
- func (dt *DockerTracker) ListArrayJobs(id string) ([]string, error)
- func (dt *DockerTracker) ListJobCategories() ([]string, error)
- func (dt *DockerTracker) ListJobs() ([]string, error)
- func (dt *DockerTracker) OpenMonitoringSession(name string) error
- func (dt *DockerTracker) Wait(jobid string, timeout time.Duration, states ...drmaa2interface.JobState) error
Constants ¶
const (
ContainerLabelJobTemplate = "drmaa2jobtemplate"
)
Variables ¶
This section is empty.
Functions ¶
func NewAllocator ¶ added in v0.3.0
func NewAllocator() *allocator
func ReadJobTemplateFromLabel ¶ added in v0.3.26
func ReadJobTemplateFromLabel(containerID string) (drmaa2interface.JobTemplate, error)
ReadJobTemplateFromLabel reads the "drmaa2jobtemplate" label from the specified container. Then it decodes the base64/json encoded JobTemplate and returns it. Fo encoding see jobTemplateToContainerConfig().
Types ¶
type DockerTracker ¶
type DockerTracker struct {
// contains filtered or unexported fields
}
func New ¶
func New(jobsession string) (*DockerTracker, error)
New creates a new DockerTracker. How the Docker client is configured can be influenced by (from the Docker Documentation (https://github.com/moby/moby/blob/master/client/client.go)): "Use DOCKER_HOST to set the url to the docker server.
Use DOCKER_API_VERSION to set the version of the API to reach, leave empty for latest. Use DOCKER_CERT_PATH to load the TLS certificates from. Use DOCKER_TLS_VERIFY to enable or disable TLS verification, off by default."
func (*DockerTracker) AddArrayJob ¶
func (dt *DockerTracker) AddArrayJob(jt drmaa2interface.JobTemplate, begin int, end int, step int, maxParallel int) (string, error)
func (*DockerTracker) AddJob ¶
func (dt *DockerTracker) AddJob(jt drmaa2interface.JobTemplate) (string, error)
func (*DockerTracker) CloseMonitoringSession ¶ added in v0.3.16
func (dt *DockerTracker) CloseMonitoringSession(name string) error
func (*DockerTracker) DeleteJob ¶
func (dt *DockerTracker) DeleteJob(jobid string) error
DeleteJob removes a container so it is no longer in docker ps -a (and therefore not in the job list).
func (*DockerTracker) GetAllJobIDs ¶ added in v0.3.16
func (dt *DockerTracker) GetAllJobIDs(filter *drmaa2interface.JobInfo) ([]string, error)
func (*DockerTracker) GetAllMachines ¶ added in v0.3.16
func (dt *DockerTracker) GetAllMachines(filter []string) ([]drmaa2interface.Machine, error)
func (*DockerTracker) GetAllQueueNames ¶ added in v0.3.16
func (dt *DockerTracker) GetAllQueueNames(filter []string) ([]string, error)
func (*DockerTracker) JobControl ¶
func (dt *DockerTracker) JobControl(jobid, state string) error
func (*DockerTracker) JobInfo ¶
func (dt *DockerTracker) JobInfo(jobid string) (ji drmaa2interface.JobInfo, err error)
func (*DockerTracker) JobInfoFromMonitor ¶ added in v0.3.16
func (dt *DockerTracker) JobInfoFromMonitor(id string) (ji drmaa2interface.JobInfo, err error)
JobInfoFromMonitor might collect job state and job info in a different way as a JobSession with persistent storage does
func (*DockerTracker) JobState ¶
func (dt *DockerTracker) JobState(jobid string) (drmaa2interface.JobState, string, error)
func (*DockerTracker) JobTemplate ¶ added in v0.3.26
func (dt *DockerTracker) JobTemplate(jobID string) (drmaa2interface.JobTemplate, error)
JobTemplate returns the JobTemplate for the given jobID. This implements the JobTemplater interface for the DockerTracker.
func (*DockerTracker) ListArrayJobs ¶
func (dt *DockerTracker) ListArrayJobs(id string) ([]string, error)
func (*DockerTracker) ListJobCategories ¶
func (dt *DockerTracker) ListJobCategories() ([]string, error)
ListJobCategories lists all container images available to run commands on.
func (*DockerTracker) ListJobs ¶
func (dt *DockerTracker) ListJobs() ([]string, error)
func (*DockerTracker) OpenMonitoringSession ¶ added in v0.3.16
func (dt *DockerTracker) OpenMonitoringSession(name string) error
func (*DockerTracker) Wait ¶
func (dt *DockerTracker) Wait(jobid string, timeout time.Duration, states ...drmaa2interface.JobState) error
Source Files