README ¶
libzipfs
------------------- ---------------
| go executable | | zip file |
------------------- ---------------
\ /
--> libzipfs-combiner <-
|
v
----------------------------------------------------
| go executable | zip file | 256-byte footer | = a new single executable with all media inside
----------------------------------------------------
Libzipfs lets you ship a filesystem of media resources inside your golang application. This is done by attaching a Zip file containing the directories of your choice to the end of your application, and following it with a short footer to allow the go executable to locate the files and serve them via a fuse mount point. The client will typically be either be Go or C embedded in the go executable (see for example testfiles/api.go), but can be another application altogether. For the later use, see for example the libzipfs/cmd/mountzip/mountzip.go example source code.
Use cases
-
You have a bunch of images/scripts/files to be served from a webserving application, and you want to bundle those resources alongside your Go webapp. libzipfs lets you easily create a single executable that contains all your resources in one place.
-
If you are using CGO to call C code, and that C code expects to be able to read files from somewhere on the filesystem, you can package up all those files, ship them with the executable, and they can be read from the fuse mountpoint -- where the C code can find and use them. For example, my https://github.com/glycerine/rmq project embeds R inside a Go binary, and libzipfs allows R libraries to be easily shipped all together in a single binary.
status
Excellent. Works well and is very useful. I only use it on OSX and Linux. On OSX
you need to have OSX Fuse installed first. On Linux you'll need to either sudo yum install fuse
or sudo apt-get install fuse-utils
to obtain the /bin/fusermount
utility.
installation
$ go get -t -u -v github.com/glycerine/libzipfs
$ cd $GOPATH/src/github.com/glycerine/libzipfs && make
$ ## note 1: the libzipfs-combiner and mountzip utilities are now in your $GOPATH/bin
$ ## note 2: be sure that $GOPATH/bin is added to your PATH env variable
$ ## e.g. in your ~/.bashrc you have: export PATH=$GOPATH/bin:$PATH
$ go test -v # to run the test suite
$ make demo # to see the demo code run
origins
The libzipfs library is derived from Tommi Virtanen's work https://github.com/bazil/zipfs, which is fantastic and provides a fuse-mounted read-only filesystem from a Zip file. The zipfs library and https://github.com/bazil/fuse are doing the heavy lifting behind the scenes.
The project was inspired by https://github.com/bazil/zipfs and https://github.com/shurcooL/vfsgen
In particular, vfsgen is a similar approach, but I needed the ability to serve files to legacy code that expects to read from a file system.
libzipfs: a fully integrated solution
Libzipfs builds on top of zipfs to allow developers the ability to assemble a "combined" file from an executable and a Zip file containing the directories you wish to have available to your application at runtime. The structure of the combined file looks like this:
----------------------------------------------------
| executable | zip file | 256-byte footer |
----------------------------------------------------
^ ^
byte 0 byte N
The embedded Zip file (the middle part in the diagram above) can then be made available via a fuse mountpoint. The Go executable will contain Go code to accomplish this. The 256-bite footer at the end of the file describes the location of the embedded zip file. The combined file is still an executable, and can be run directly.
creating a combined executable and Zip file
the libzipfs-combiner
utility does this for you.
For example: assuming that my.go.binary
and hi.zip
already exist,
and you wish to create a new combo executable called my.go.binary.combo
,
you would do:
$ libzipfs-combiner --help
libzipfs-combiner --help
Usage of libzipfs-combiner:
-exe string
path to the executable file
-o string
path to the combined output file to be written (or split if -split given)
-split
split the output file back apart (instead of combine which is the default)
-zip string
path to the Zip file to embed
$ libzipfs-combiner --exe my.go.binary -o my.go.binary.combo -zip hi.zip
api/code code inside your my.go.binary.combo
binary:
type make demo
and see testfiles/api.go for a full demo:
Our demo zip file testfiles/hi.zip
is a simple zip file with one file hello
that resides inside two nested directories:
$ unzip -Z -z testfiles/hi.zip
Archive: testfiles/hi.zip 478 bytes 3 files
drwxr-xr-x 3.0 unx 0 bx stor 19-Dec-15 17:27 dirA/
drwxr-xr-x 3.0 unx 0 bx stor 19-Dec-15 17:27 dirA/dirB/
-rw-r--r-- 3.0 unx 12 tx stor 19-Dec-15 17:27 dirA/dirB/hello
3 files, 12 bytes uncompressed, 12 bytes compressed: 0.0%
the go code:
package main
import (
"bytes"
"fmt"
"io/ioutil"
"path"
"github.com/glycerine/libzipfs"
)
// build instructions:
//
// cd libzipfs && make
// cd testfiles; go build -o api-demo api.go
// libzipfs-combiner -exe api-demo -zip hi.zip -o api-demo-combo
// ./api-demo-combo
func main() {
z, mountpoint, err := libzipfs.MountComboZip()
if err != nil {
panic(err)
}
defer z.Stop() // if you want to stop serving files
// access the files from `hi.zip` at mountpoint
by, err := ioutil.ReadFile(path.Join(mountpoint, "dirA", "dirB", "hello"))
if err != nil {
panic(err)
}
by = bytes.TrimRight(by, "\n")
fmt.Printf("we should see our file dirA/dirB/hello from inside hi.zip, containing 'salutations'.... '%s'\n", string(by))
if string(by) != "salutations" {
panic("problem detected")
}
fmt.Printf("Excellent: all looks good.\n")
}
example number 2: mountzip
The mountzip
utility (see the source code in libzipfs/cmd/mountzip/mountzip.go
) mounts a zip file of your choice on a directory of your choice.
$ cd $GOPATH/src/github.com/glycerine/libzipfs
$ make # installs the mountzip utility into $GOPATH/bin
$ mountzip -help
Usage of mountzip:
-mnt string
directory to fuse-mount the Zip file on
-zip string
path to the Zip file to mount
$
$ mkdir /tmp/hi
$ mountzip -zip testfiles/hi.zip -mnt /tmp/hi
Zip file 'hi.zip' mounted at directory '/tmp/hi'. [press ctrl-c to exit and unmount]
license
MIT license. See enclosed LICENSE file.
Documentation ¶
Overview ¶
combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination. libzipfs will look for this footer and use it to determine where the internalized zipfile filesystem starts.
combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination. libzipfs will look for this footer and use it to determine where the internalized zipfile filesystem starts.
Index ¶
- Constants
- Variables
- func Blake2HashFile(path string) (hash []byte, length int64, err error)
- func DirExists(name string) bool
- func DisplayVersionAndExitIfRequested()
- func DoCombineExeAndZip(cfg *CombinerConfig) error
- func FileExists(name string) bool
- func FindMount() error
- func FindMountUmount() error
- func FindUmount() error
- func ShouldRetry(err error) bool
- func TSPrintf(format string, a ...interface{})
- func TrimTrailingSlashes(mountpoint string) string
- func VPrintf(format string, a ...interface{})
- func VersionString() string
- func WaitUntilMounted(mountPoint string) error
- func WaitUntilUnmounted(mountPoint string) error
- type CombinerConfig
- type Dir
- type FS
- type File
- type FileHandle
- type Footer
- type FooterArray
- type FuseZipFs
Constants ¶
const BLAKE2_HASH_LEN = 64
const LIBZIPFS_FOOTER_LEN = 256
const MAGIC_NUM_LEN = 16
Variables ¶
var GITLASTCOMMIT string // git rev-parse HEAD
var GITLASTTAG string // git describe --abbrev=0 --tags
track git version of this lib
var MAGIC1 = []byte("\nLibZipFs00\n")
var MAGIC2 = []byte("\nLibZipFsEnd\n")
var Verbose bool
Functions ¶
func DisplayVersionAndExitIfRequested ¶
func DisplayVersionAndExitIfRequested()
func DoCombineExeAndZip ¶
func DoCombineExeAndZip(cfg *CombinerConfig) error
func FileExists ¶
func FindMountUmount ¶
func FindMountUmount() error
func FindUmount ¶
func FindUmount() error
func ShouldRetry ¶
helper for reading in a loop. will panic on unknown error.
func TrimTrailingSlashes ¶
must trim any trailing slash from the mountpoint, or else mount can fail
func VersionString ¶
func VersionString() string
func WaitUntilMounted ¶
func WaitUntilUnmounted ¶
Types ¶
type CombinerConfig ¶
type CombinerConfig struct { ExecutablePath string ZipfilePath string OutputPath string Split bool }
func (*CombinerConfig) DefineFlags ¶
func (c *CombinerConfig) DefineFlags(fs *flag.FlagSet)
call DefineFlags before myflags.Parse()
func (*CombinerConfig) ValidateConfig ¶
func (c *CombinerConfig) ValidateConfig() error
call c.ValidateConfig() after myflags.Parse()
type Dir ¶
type Dir struct {
// contains filtered or unexported fields
}
type FileHandle ¶
type FileHandle struct {
// contains filtered or unexported fields
}
func (*FileHandle) Read ¶
func (fh *FileHandle) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error
func (*FileHandle) Release ¶
func (fh *FileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) error
type Footer ¶
type Footer struct {}
func DoSplitOutExeAndZip ¶
func DoSplitOutExeAndZip(cfg *CombinerConfig) (*Footer, error)
func ReadFooter ¶
func ReadFooter(combinedPath string) (footerStartOffset int64, ft *Footer, comb *os.File, err error)
client take responsibility for closing combFd when done with it; it is the open file handled (if err == nil) for reading from the file at combinedPath.
func ReifyFooterAndDoInexpensiveChecks ¶
func ReifyFooterAndDoInexpensiveChecks(by []byte, combinedPath string, footerStartOffset int64) (*Footer, error)
must return err if foot is bad
func (*Footer) FillHashes ¶
func (foot *Footer) FillHashes(cfg *CombinerConfig) error
func (*Footer) GetFooterChecksum ¶
func (*Footer) VerifyExeZipChecksums ¶
func (foot *Footer) VerifyExeZipChecksums(cfg *CombinerConfig) (err error)
type FooterArray ¶
type FooterArray [LIBZIPFS_FOOTER_LEN]byte
type FuseZipFs ¶
type FuseZipFs struct { ZipfilePath string MountPoint string Ready chan bool ReqStop chan bool Done chan bool // contains filtered or unexported fields }
func MountComboZip ¶
The Main API entry point for mounting a combo file vis FUSE to make the embedded Zip file directory available. Users should call fzfs.Stop() when/if they wish to stop serving files at mountpoint.
func NewFuseZipFs ¶
func NewFuseZipFs(zipFilePath, mountpoint string, byteOffsetToZipFileStart int64, bytesAvail int64, footerBytes int64) *FuseZipFs
Mount a possibly combined/zipfile at mountpiont. Call Start() to start servicing fuse reads.
If the file has a libzipfs footer on it, set footerBytes == LIBZIPFS_FOOTER_LEN. The bytesAvail value should describe how long the zipfile is in bytes, and byteOffsetToZipFileStart should describe how far into the (possibly combined) zipFilePath the actual zipfile starts.
func NewFuzeZipFsFromCombo ¶
mount the comboFilePath file in a temp directory mountpoint created just for this purpose, and return the mountpoint and a handle to the fuse fileserver in fzfs.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
cmd
|
|
libzipfs-combiner
combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination.
|
combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination. |