HotPotatoFS

HotPotatoFS

Groupcache + FUSE 
= Read-Only, Distribute, in-Memory-Caching Filesystem in Go
= Playing catch in a group with a FUSE
= HotPotato

HotPotato is a simple, read-only, in-memory-Cacheing, FUSE filesystem used to put groupcache in front of a slow disk or network mount (nfs, s3, smb etc) on one or many machines to reduce the time needed to repeatedly read a file.

It was developed primairly for use in parralel data analysis and simulation. You can use it to speed up a parralel R, python, matlab or whatever job running via golem, qsub, gnu parralel etc without needing to rewrite the analysis code.

It is expermental software in an early state of development and may break.

It is written in go (golang) using groupcache and the bazil.org fuse library:

https://github.com/golang/groupcache

http://bazil.org/fuse/

QuickStart

With go and go path set up:

Instalation

go get github.com/ryanbressler/HotPotatoFS
go install github.com/ryanbressler/HotPotatoFS/hotpot

Single Machine

hotpot -mountpoint /hotpotato -target /nfsmount

Multi Machine

peerfile.txt:

http://host1:8080
http://host2:8080

host1:

hotpot -mountpoint /hotpotato -target /nfsmount -me http://host1:8080 -peers peerfile.txt

host2:

hotpot -mountpoint /hotpotato -target /nfsmount -me http://host2:8080 -peers peerfile.txt

Credit

HotPotatofFS was developed by members of the Shumelevich lab at the Institute for Systems Biology to support distributed computing in cancer and biomedical research as part of our work on The Cancer Genome Atlas and other projecs.

Code is under a 3 clause BSD.

Potato image copyleft David Gibbs.