martian: github.com/martian-lang/martian/martian/util Index | Examples | Files

package util

import "github.com/martian-lang/martian/martian/util"

Packae util includes various utility methods.

These utilities are frequently Martian-specific but do not depend on Martian runtime infrastructure.

Index

Examples

Package Files

cgroups_linux.go directory_linux.go errors.go exec_linux.go file_linux.go log.go signal.go stamp.go util.go version.go walk_linux.go zip.go

Constants

const (
    ANSI_BLACK   = 30
    ANSI_RED     = 31
    ANSI_GREEN   = 32
    ANSI_YELLOW  = 33
    ANSI_BLUE    = 34
    ANSI_MAGENTA = 35
    ANSI_CYAN    = 36
    ANSI_WHITE   = 37
)
const TIMEFMT = "2006-01-02 15:04:05"

Variables

var ENABLE_LOGGING bool = true
var HANDLED_SIGNALS = [...]os.Signal{
    os.Interrupt,
    syscall.SIGHUP,
    syscall.SIGTERM,
    syscall.SIGUSR1,
    syscall.SIGUSR2,
}

After a call to SetupSignalHandlers, these signals will be handled by waiting for all pending critical sections to complete, running all registered handlers, and then exiting with return code 1

func ArrayToString Uses

func ArrayToString(data []interface{}) []string

func Atoi Uses

func Atoi(s []byte) (int64, error)

Atoi returns parses the utf8-encoded bytes of a decimal string of native size. Use this instead of strconv.Atoi to avoid copying bytes to string.

func Colorize Uses

func Colorize(s string, c int) string

Surrounds the given string with ANSI color control characters.

func CountDirNames Uses

func CountDirNames(fd int) (int, error)

CountDirNames returns the number of files in the directory opened with the given file descriptor. Unlike the generic implementation, it avoids the string allocation overhead of actually getting the names out.

func CreateZip Uses

func CreateZip(zipPath string, filePaths []string) error

func DieIf Uses

func DieIf(err error)

End the process if err is not nil. Because this method waits up to one minute for critical sections to end, it should not be called from inside a critical section.

func EnterCriticalSection Uses

func EnterCriticalSection()

EnterCriticalSection should be called before performing a sequence of operations which should be logically atomic with respect to signals. Handled signals will not terminate the process until ExitCriticalSection is called. If the process has already been signaled, EnterCriticalSection will block.

Note that EnterCriticalSection should never be called when already in a critical section, as it may deadlock if the process is signaled between the first and second call.

func EnvRequire Uses

func EnvRequire(reqs [][]string, log bool) map[string]string

func ExitCriticalSection Uses

func ExitCriticalSection()

ExitCriticalSection should be called after EnterCriticalSection once the logically atomic sequence of operations has completed.

func FileCreateTime Uses

func FileCreateTime(info os.FileInfo) time.Time

func FindUniquePath Uses

func FindUniquePath(fname string, searchPaths []string) (string, error)

FindUniquePath searches through searchPaths for a path such that path.Join(p, fname) points to an existing file. If more than one path in searchPaths satisfies that requirement, an error is returned indicating the ambiguity. If no such path exists, an error satisfying os.IsNotExist will be returned.

func FormatEnv Uses

func FormatEnv(envs map[string]string) []string

func FormatMroPath Uses

func FormatMroPath(mroPaths []string) string

func GetCgroupMemoryLimit Uses

func GetCgroupMemoryLimit() (limit, softLimit, usage int64)

Get the lowest rss memory limit in cgroups, as well as the current usage.

func GetDirectorySize Uses

func GetDirectorySize(paths []string) (uint, uint64)

func GetFilenameWithSuffix Uses

func GetFilenameWithSuffix(dir string, fname string) string

func GetGitBranch Uses

func GetGitBranch(dir string) (string, error)

Returns the output of running 'git rev-parse --abbrev-ref HEAD' in the given directory, e.g. 'master'.

func GetGitTag Uses

func GetGitTag(dir string) (string, error)

Returns the output of running 'git describe --tags --dirty --always' in the given directory, e.g. 'v2.3.0-rc3-10-gf615588-dirty'

func GetLogger Uses

func GetLogger(component string) (*golog.Logger, bool)

Wraps the martian logger as go log.Logger object for use with, for example, net/http.HttpServer.ErrorLog

func GetMroVersion Uses

func GetMroVersion(dirs []string) (string, error)

Searches for a file named .version in the parent directory of any of the given directories and returns its content if available. If none is found, attempts to run 'git describe --tags --dirty --always' in those directories to get a version.

func GetSakeVersion Uses

func GetSakeVersion(dir string) (string, error)

Searches for a file name .version in the parent of the given directory name and returns its content.

func GetVersion Uses

func GetVersion() string

Get the build version for this binary, as embedded by the build system.

func IsRelease Uses

func IsRelease() bool

IsRelease returns true if the build system specified that this was a release version.

func Log Uses

func Log(format string, v ...interface{})

Logs the given string to the current log stream. If one has not been initialized with LogTee or LogTeeWriter, the content is buffered until one of those methods is called.

func LogError Uses

func LogError(err error, component string, format string, v ...interface{})

Logs a line in the form "<timestamp> [component] <message>" followed by a line with err.Error(). Component is intended to indicate the source of the message and should be a consistent length. By convention, this length is 7 characters.

func LogInfo Uses

func LogInfo(component string, format string, v ...interface{})

Logs a line in the form "<timestamp> [component] <message>". Component is intended to indicate the source of the message and should be a consistent length. By convention, this length is 7 characters.

func LogTee Uses

func LogTee(filename string)

Sets up the logging methods to log to the given file.

func LogTeeWriter Uses

func LogTeeWriter(writer StringWriter)

Sets up the logging methods to log to the given writer.

func MakeJSON Uses

func MakeJSON(data interface{}) string

func MakeTag Uses

func MakeTag(key string, value string) string

func MergeEnv Uses

func MergeEnv(envs map[string]string) []string

func Mkdir Uses

func Mkdir(p string) error

func MkdirAll Uses

func MkdirAll(p string) error

func ParseMroFlags Uses

func ParseMroFlags(opts map[string]interface{}, doc string, martianOptions []string, martianArguments []string)

func ParseMroPath Uses

func ParseMroPath(mroPath string) []string

func ParseTag Uses

func ParseTag(tag string) (string, string)

func ParseTagsOpt Uses

func ParseTagsOpt(opt string) []string

func Pdeathsig Uses

func Pdeathsig(attr *syscall.SysProcAttr, sig syscall.Signal) *syscall.SysProcAttr

Add pdeathsig to a SysProcAttr structure, if the operating system supports it, and return the object. On other platforms, do nothing.

func Pluralize Uses

func Pluralize(n int) string

func Print Uses

func Print(format string, v ...interface{})

Like Log, but also prints to standard output.

func PrintError Uses

func PrintError(err error, component string, format string, v ...interface{})

Like LogError but also prints to standard output.

func PrintInfo Uses

func PrintInfo(component string, format string, v ...interface{})

Like LogInfo but also prints to standard output.

func Println Uses

func Println(format string, v ...interface{})

func ReadFileAt Uses

func ReadFileAt(dirFd int, name string) (b []byte, err error)

ReadFileAt returns all of the bytes in a file opened relative to the directory with the given file descriptor.

func ReadZip Uses

func ReadZip(zipPath string, filePath string) ([]byte, error)

func ReadZipFile Uses

func ReadZipFile(zipPath, filePath string) (io.ReadCloser, error)

Opens a file within a zip archive for reading.

func Readdirnames Uses

func Readdirnames(readPath string) (names []string, err error)

func RegisterSignalHandler Uses

func RegisterSignalHandler(object HandlerObject)

Registers an object as having cleanup work which should be allowed to complete before the process terminates due to a handled signal.

func RelPath Uses

func RelPath(p string) string

func Render Uses

func Render(dir string, tname string, data interface{}) string

func SearchPaths Uses

func SearchPaths(fname string, searchPaths []string) (string, bool)

SearchPaths searches through searchPaths for the first path such that path.Join(p, fname) points to an existing file, and returns that joined path, or false if no such path is present.

func SetPrintLogger Uses

func SetPrintLogger(w StringWriter)

Sets the target for Print* logging.

func SetupSignalHandlers Uses

func SetupSignalHandlers()

Initializes the global signal handler.

func Suicide Uses

func Suicide(success bool)

Kill this process cleanly, after waiting for critical sections and handlers to complete.

func Timestamp Uses

func Timestamp() string

func UnregisterSignalHandler Uses

func UnregisterSignalHandler(object HandlerObject)

func Unzip Uses

func Unzip(zipPath string) error

func UnzipIgnoreExisting Uses

func UnzipIgnoreExisting(zipPath string) error

UnzipIgnoreExisting unzips the given archive, skipping over files which already exist.

func ValidateID Uses

func ValidateID(id string) error

func VerifyLogFile Uses

func VerifyLogFile() error

If the logger has an open file, check that the file exists at the expected location on disk, and that the file that is at that location is the same as the one that is open.

func Walk Uses

func Walk(root string, walkFn filepath.WalkFunc) error

Faster, Unix-specific implementation of filepath.Walk, which avoids the directory sort and uses openat and fstatat to avoid forcing extra dirent syncs. This can save a lot of time on NFS servers, and also provides more consistent behavior in the face of directory renames or changes to the current process working directory.

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. Walk does not follow symbolic links.

Unlike filepath.Walk, Walk makes ABSOLUTELY NO GUARANTEES about the order in which the files are traversed.

Unlike filepath.Walk, the Sys() method of the os.FileInfo object passed to walkFn will be of type golang.org/x/sys/unix.Stat_t.

Code:

if root, err := ioutil.TempDir("", "examplewalk"); err != nil {
    fmt.Println("Failed to create tempdir:", err)
    return
} else {
    defer os.RemoveAll(root)
    if err := os.MkdirAll(path.Join(root, "a", "b", "c", "d"), 0777); err != nil {
        fmt.Println(err)
    }
    if err := os.Symlink(path.Join(root, "a", "b", "c", "d"),
        path.Join(root, "a", "b", "c", "ds")); err != nil {
        fmt.Println(err)
    }
    if err := os.MkdirAll(path.Join(root, "a", "b", "e", "f"), 0777); err != nil {
        fmt.Println(err)
    }
    if err := os.MkdirAll(path.Join(root, "a", "b", "g"), 0777); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a.txt"), []byte("Test string"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "b.txt"), []byte("Test string2"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a", "c.txt"), []byte("Test string33"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a", "d.txt"), []byte("Test string444"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a", "b", "g", "e.txt"), []byte("Test string5555"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a", "b", "g", "f.txt"), []byte("Test string66666"), 0666); err != nil {
        fmt.Println(err)
    }
    if err := ioutil.WriteFile(path.Join(root, "a", "b", "e", "f", "g.txt"), []byte("Test string777777"), 0666); err != nil {
        fmt.Println(err)
    }

    if err := Walk(root, func(p string, info os.FileInfo, err error) error {
        if !strings.HasPrefix(p, root) {
            fmt.Println("Path was not rooted:", p)
        }
        if err != nil {
            fmt.Printf("Error reading %s:\n%v\n", strings.TrimPrefix(p, root), err)
        } else {
            if info.IsDir() {
                fmt.Printf("Directory '%s'\n", strings.TrimPrefix(p, root))
            } else if info.Mode()&os.ModeSymlink != 0 {
                fmt.Printf("Symlink '%s'\n", strings.TrimPrefix(p, root))
            } else {
                fmt.Printf("File '%s' had %d bytes\n", strings.TrimPrefix(p, root), info.Size())
            }
        }
        if path.Base(p) == "f" {
            fmt.Println("Skipping", path.Base(p))
            return filepath.SkipDir
        } else {
            return err
        }
    }); err != nil {
        fmt.Println(err)
    }

    if err := Walk(path.Join(root, "b"), func(p string, info os.FileInfo, err error) error {
        if err != nil {
            fmt.Printf("Correctly expected error reading %s\n",
                strings.TrimPrefix(p, root))
        } else {
            fmt.Printf("File '%s' had %d bytes\n", p, info.Size())
        }
        return err
    }); err != nil {
        fmt.Println("Correctly expected error")
    }
    // Unordered Output:
    // Directory ''
    // Directory '/a'
    // File '/a.txt' had 11 bytes
    // File '/b.txt' had 12 bytes
    // Directory '/a/b'
    // File '/a/c.txt' had 13 bytes
    // File '/a/d.txt' had 14 bytes
    // Directory '/a/b/c'
    // Directory '/a/b/e'
    // Directory '/a/b/g'
    // Directory '/a/b/c/d'
    // Symlink '/a/b/c/ds'
    // Directory '/a/b/e/f'
    // File '/a/b/g/e.txt' had 15 bytes
    // File '/a/b/g/f.txt' had 16 bytes
    // Skipping f
    // Correctly expected error reading /b
    // Correctly expected error
}

Output:

Directory ''
Directory '/a'
File '/a.txt' had 11 bytes
File '/b.txt' had 12 bytes
Directory '/a/b'
File '/a/c.txt' had 13 bytes
File '/a/d.txt' had 14 bytes
Directory '/a/b/c'
Directory '/a/b/e'
Directory '/a/b/g'
Directory '/a/b/c/d'
Symlink '/a/b/c/ds'
Directory '/a/b/e/f'
File '/a/b/g/e.txt' had 15 bytes
File '/a/b/g/f.txt' had 16 bytes
Skipping f
Correctly expected error reading /b
Correctly expected error

func WidthForInt Uses

func WidthForInt(max int) int

Gets the number of digits required to display a given integer in base 10. Optimizes for the common cases.

type HandlerObject Uses

type HandlerObject interface {
    HandleSignal(sig os.Signal)
}

Interface for objects which need to perform cleanup operations when the process is being terminated by a handled signal.

The HandleSignal method is called after all critical sections have completed. Attempting to enter a critical section from within the HandleSignal method will result in a deadlock.

type Logger Uses

type Logger struct {
    // contains filtered or unexported fields
}
var LOGGER *Logger = nil

func (*Logger) Write Uses

func (logger *Logger) Write(msg []byte) (int, error)

func (*Logger) WriteString Uses

func (logger *Logger) WriteString(msg string) (int, error)

type MartianError Uses

type MartianError struct {
    Msg string
}

MartianError

func (*MartianError) Error Uses

func (self *MartianError) Error() string

type SignalHandler Uses

type SignalHandler struct {
    // contains filtered or unexported fields
}

func (*SignalHandler) Notify Uses

func (self *SignalHandler) Notify()

Notify this handler of signals.

type StringWriter Uses

type StringWriter interface {
    io.Writer
    WriteString(string) (int, error)
}

StringWriter is the interface for writers which can write both bytes and strings.

type ZipError Uses

type ZipError struct {
    ZipPath  string
    FilePath string
}

ZipError

func (*ZipError) Error Uses

func (self *ZipError) Error() string

Package util imports 25 packages (graph) and is imported by 11 packages. Updated 2019-09-25. Refresh now. Tools for package owners.