skywalker

package module
v0.0.0-...-e262af3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 31, 2021 License: BSD-3-Clause Imports: 6 Imported by: 10

README

skywalker GoDoc Build Status Build status codecov Go Report Card

Skywalker is a package to allow one to concurrently go through a filesystem with ease.

Features

  • Concurrency
  • BlackList filtering
  • WhiteList filtering
  • Filter by Directory
  • Filter by Extension
  • Glob Filtering (provided by gobwas/glob)

For matching to work properly across platforms. Please use /. In gobwas/glob the \ is an escape character (so you can escape *, ?, etc...) making it difficult to know if you want to escape a character or go into directory.

Example

package main

import (
    "fmt"
    "sort"
    "strings"
    "sync"

    "github.com/dixonwille/skywalker"
)

type ExampleWorker struct {
    *sync.Mutex
    found []string
}

func (ew *ExampleWorker) Work(path string) {
    //This is where the necessary work should be done.
    //This will get concurrently so make sure it is thread safe if you need info across threads.
    ew.Lock()
    defer ew.Unlock()
    ew.found = append(ew.found, path)
}

func main() {
    //Following two functions are only to create and destroy data for the example
    defer teardownData()
    standupData()

    ew := new(ExampleWorker)
    ew.Mutex = new(sync.Mutex)

    //root is the root directory of the data that was stood up above
    sw := skywalker.New(root, ew)
    sw.DirListType = skywalker.LTBlacklist
    sw.DirList = []string{"sub"}
    sw.ExtListType = skywalker.LTWhitelist
    sw.ExtList = []string{".pdf"}
    err := sw.Walk()
    if err != nil {
        fmt.Println(err)
        return
    }
    sort.Sort(sort.StringSlice(ew.found))
    for _, f := range ew.found {
        fmt.Println(strings.Replace(f, sw.Root, "", 1))
    }
}

Documentation

Overview

Package skywalker walks through a filesystem concurrently.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ListType

type ListType int

ListType is used to specify how to handle the contents of a list

const (
	//LTBlacklist is used to specify that a list is to exclude the contents.
	LTBlacklist ListType = iota
	//LTWhitelist is used to specify that a list is to include the contents.
	LTWhitelist
)

type Skywalker

type Skywalker struct {
	//Root is where the file walker starts. It is converted to an absolute path before start.
	Root string

	//List and ListType should only be used for fine filtering of paths.
	//It uses https://github.com/gobwas/glob for glob checking on each patch check.
	ListType ListType
	List     []string

	//ExtList and ExtListType are used to narrow down the files by their extensions.
	//Make sure to include the preceding ".".
	ExtListType ListType
	ExtList     []string

	//DirList and DirListType are used to narrow down by directories.
	//Will skip the appropriate directories and their files/subfolders.
	DirListType ListType
	DirList     []string

	//NumWorkers are how many workers are listening to the queue to do the work.
	NumWorkers int

	//QueueSize is how many paths to queue up at a time.
	//Useful for fine control over memory usage if needed.
	QueueSize int

	//Worker is the function that is called on each file/directory.
	Worker Worker

	//FilesOnly should be set to true if you only want to queue up files.
	FilesOnly bool
	// contains filtered or unexported fields
}

Skywalker can concurrently go through files in Root and call Worker on everything. It is recommended to use DirList and ExtList as much as possible as it is more perfomant than List.

Example
package main

import (
	"fmt"
	"sort"
	"strings"
	"sync"

	"github.com/dixonwille/skywalker"
)

type ExampleWorker struct {
	*sync.Mutex
	found []string
}

func (ew *ExampleWorker) Work(path string) {
	//This is where the necessary work should be done.
	//This will get concurrently so make sure it is thread safe if you need info across threads.
	ew.Lock()
	defer ew.Unlock()
	ew.found = append(ew.found, path)
}

func main() {
	//Following two functions are only to create and destroy data for the example
	defer teardownData()
	standupData()

	ew := new(ExampleWorker)
	ew.Mutex = new(sync.Mutex)

	//root is the root directory of the data that was stood up above
	sw := skywalker.New(root, ew)
	sw.DirListType = skywalker.LTBlacklist
	sw.DirList = []string{"sub"}
	sw.ExtListType = skywalker.LTWhitelist
	sw.ExtList = []string{".pdf"}
	err := sw.Walk()
	if err != nil {
		fmt.Println(err)
		return
	}
	sort.Sort(sort.StringSlice(ew.found))
	for _, f := range ew.found {
		show := strings.Replace(f, sw.Root, "", 1)
		show = strings.Replace(show, "\\", "/", -1)
		fmt.Println(show)
	}
}
Output:

/subfolder/few.pdf
/the/few.pdf

func New

func New(root string, worker Worker) *Skywalker

New creates a new Skywalker that can walk through the specified root and calls the Worker on each file and/or directory. Defaults Skywalker to have 20 workers, a QueueSize of 100 and only queue files.

func (*Skywalker) Walk

func (sw *Skywalker) Walk() error

Walk goes through the files and folders in Root and calls the worker on each. Checks the lists specified to check whether it should ignore files or directories. It also handles the creation of workers and queues needed for walking.

type Worker

type Worker interface {
	Work(path string)
}

Worker is anything that knows what to do with a path.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL