README
¶
marionette
This is a Go port of the marionette programmable networy proxy.
WebBrowser Demonstration
Please install Marionette as described below, and then go to the web browser demonstration page here
Development
Marionette requires several dependencies to be installed first. Two of them
are in the third_party directory and the third one can be downloaded from
the web.
You can use the ./build_third_party.sh script in the root of this repository
to build the third party libraries or follow the instructions below to manually
build them or install them system wide.
Installing on CentOS
Ensure you have a C/C++ compiler installed:
$ yum group install -y "Development Tools"
Installing OpenFST
You must use the included third_party/openfst implementation. Also note that
static builds must be enabled via the --enable-static flag.
$ cd third_party/openfst
$ ./configure --enable-static=yes
$ make
$ sudo make install
Installing re2
You must use the included third_party/re2 implementation:
$ cd third_party/re2
$ make
$ sudo make install
GMP
Download the latest version of GMP, unpack the archive and run:
$ wget https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2
$ tar -xvjf gmp-6.1.2.tar.bz2
$ cd gmp-6.1.2
$ ./configure --enable-cxx
$ make
$ sudo make install
$ make check
Building the Marionette Binary
First, make sure you have installed Go from https://golang.org/. Next,
install dep using these instructions.
Finally, retrieve the source, update project dependencies, and install the
marionette binary:
$ go get github.com/redjack/marionette
$ cd $GOPATH/src/github.com/redjack/marionette
$ dep ensure
$ go install ./cmd/marionette
The marionette binary is now installed in your $GOPATH/bin folder.
Installing new build-in formats
When adding new formats, you'll need to first install go-bindata:
$ go get -u github.com/jteeuwen/go-bindata/...
Then you can use go generate to convert the asset files to Go files:
$ go generate ./...
To install the original marionette library for comparing tests, download the latest version, unpack the archive and run:
Testing
Use the built-in go testing command to run the unit tests:
$ go test ./...
If you have the original Python marionette installed then you can run tests
of the ports using the python tag:
$ go test -tags python ./regex2dfa
$ go test -tags python ./fte
Demo
HTTP-over-FTP
In this example, we'll mask our HTTP traffic as FTP packets.
First, follow the installation instructions above on your client & server machines.
Start the server proxy on your server machine and forward traffic to a server
such as google.com.
$ marionette server -format ftp_simple_blocking -proxy google.com:80
listening on [::]:2121, proxying to google.com:80
Start the client proxy on your client machine and connect to your server proxy.
Replace $SERVER_IP with the IP address of your server.
$ marionette client -format ftp_simple_blocking -server $SERVER_IP
listening on 127.0.0.1:8079, connected to <SERVER_IP>
Finally, send a curl to 127.0.0.1:8079 and you should see a response from
google.com:
$ curl 127.0.0.1:8079
Documentation
¶
Index ¶
- Constants
- Variables
- func RegisterPlugin(module, method string, fn PluginFunc)
- type BufferedConn
- type Cell
- type Cipher
- type ClientProxy
- type DFA
- type Dialer
- type FSM
- type Listener
- type NetDialer
- type PluginFunc
- type ServerProxy
- type Stream
- func (s *Stream) Close() error
- func (s *Stream) CloseRead() error
- func (s *Stream) CloseWrite() error
- func (s *Stream) Closed() bool
- func (s *Stream) Dequeue(n int) *Cell
- func (s *Stream) Enqueue(cell *Cell) error
- func (s *Stream) ID() int
- func (c *Stream) LocalAddr() net.Addr
- func (s *Stream) ModTime() time.Time
- func (s *Stream) Read(b []byte) (n int, err error)
- func (s *Stream) ReadBufferLen() int
- func (s *Stream) ReadCloseNotify() <-chan struct{}
- func (s *Stream) ReadClosed() bool
- func (s *Stream) ReadNotify() <-chan struct{}
- func (s *Stream) ReadWriteCloseNotified() bool
- func (c *Stream) RemoteAddr() net.Addr
- func (c *Stream) SetDeadline(t time.Time) error
- func (c *Stream) SetReadDeadline(t time.Time) error
- func (c *Stream) SetWriteDeadline(t time.Time) error
- func (s *Stream) Write(b []byte) (n int, err error)
- func (s *Stream) WriteBufferLen() int
- func (s *Stream) WriteCloseNotified() bool
- func (s *Stream) WriteCloseNotifiedNotify() <-chan struct{}
- func (s *Stream) WriteCloseNotify() <-chan struct{}
- func (s *Stream) WriteClosed() bool
- func (s *Stream) WriteNotify() <-chan struct{}
- type StreamSet
- func (ss *StreamSet) Close() (err error)
- func (ss *StreamSet) Create() *Stream
- func (ss *StreamSet) Dequeue(n int) *Cell
- func (ss *StreamSet) Enqueue(cell *Cell) error
- func (ss *StreamSet) Stream(id int) *Stream
- func (ss *StreamSet) Streams() []*Stream
- func (ss *StreamSet) WriteNotify() <-chan struct{}
Constants ¶
const ( // CellHeaderSize is the number of non-payload bytes used by a cell. CellHeaderSize = 25 // MaxCellLength is the maximum allowed size of a serialized cell. MaxCellLength = 32768 )
const ( // Normal cells carry zero or more bytes in a payload. CellTypeNormal = 0x1 // EOS (end-of-stream) cells mark the end of streams and carry no payload. CellTypeEOS = 0x2 )
const ( PartyClient = "client" PartyServer = "server" )
Party constants.
const ( // StreamCloseTimeout is the amount of time before an idle read-closed or // write-closed stream is reaped by a monitoring goroutine. StreamCloseTimeout = 5 * time.Second )
Variables ¶
var ( // ErrNoTransitions is returned from FSM.Next() when no transitions can be found. ErrNoTransitions = errors.New("no transitions available") // ErrRetryTransition is returned from FSM.Next() when a transition should be reattempted. ErrRetryTransition = errors.New("retry transition") // ErrUUIDMismatch is returned when a cell is received from a different UUID. // This can occur when communicating with a peer using a different MAR document. ErrUUIDMismatch = errors.New("uuid mismatch") )
var ( // ErrStreamClosed is returned enqueuing cells or writing data to a closed stream. // Dequeuing cells and reading data will be available until pending data is exhausted. ErrStreamClosed = errors.New("marionette: stream closed") // ErrWriteTooLarge is returned when a Write() is larger than the buffer. ErrWriteTooLarge = errors.New("marionette: write too large") )
var ( // ErrDialerClosed is returned when trying to operate on a closed dialer. ErrDialerClosed = errors.New("marionette: dialer closed") )
var ( // ErrListenerClosed is returned when trying to operate on a closed listener. ErrListenerClosed = errors.New("marionette: listener closed") )
var Logger = zap.NewNop()
Logger is the global marionette logger.
Rand returns a new PRNG seeded from the current time. This function can be overridden by the tests to provide a repeatable PRNG.
Functions ¶
func RegisterPlugin ¶
func RegisterPlugin(module, method string, fn PluginFunc)
RegisterPlugin adds a plugin to the plugin registry. Panic on duplicate registration.
Types ¶
type BufferedConn ¶
BufferedConn wraps a net.Conn and continually reads from it into a buffer.
The buffer is inspectable and seekable by the caller. This provides buffering until a complete cell can be decoded from the connection. The buffer is sized based on the max cell size and does not support cells that exceed that size.
func NewBufferedConn ¶
func NewBufferedConn(conn net.Conn, bufferSize int) *BufferedConn
NewBufferedConn returns a new BufferConn wrapping conn, sized to bufferSize.
func (*BufferedConn) Append ¶
func (conn *BufferedConn) Append(b []byte)
Append adds b to the end of the buffer, under lock.
func (*BufferedConn) Peek ¶
func (conn *BufferedConn) Peek(n int, blocking bool) ([]byte, error)
Peek returns the first n bytes of the read buffer. If n is -1 then returns any available data after attempting a read.
type Cell ¶
type Cell struct {
Type int // Record type (normal, end-of-stream)
Payload []byte // Data
Length int // Size of marshaled data, if specified.
StreamID int // Associated stream
SequenceID int // Record number within stream
UUID int // MAR format identifier
InstanceID int // MAR instance identifier
}
Cell represents a single unit of data sent between the client & server.
This cell is associated with a specific stream and the encoder/decoders handle ordering based on sequence id.
func (*Cell) Compare ¶
Compare returns -1 if c has a lower sequence than other, 1 if c has a higher sequence than other, and 0 if both cells have the same sequence.
func (*Cell) Equal ¶
Equal returns true if the payload, stream id, sequence, uuid, and instance id are the same.
func (*Cell) MarshalBinary ¶
MarshalBinary returns a byte slice with a serialized cell.
func (*Cell) UnmarshalBinary ¶
UnmarshalBinary decodes a serialized cell.
type Cipher ¶
type Cipher interface {
Capacity() int
Encrypt(plaintext []byte) (ciphertext []byte, err error)
Decrypt(ciphertext []byte) (plaintext, remainder []byte, err error)
}
Cipher represents the interface to the FTE Cipher.
type ClientProxy ¶
type ClientProxy struct {
// contains filtered or unexported fields
}
ClientProxy represents a proxy between incoming connections and a marionette dialer.
func NewClientProxy ¶
func NewClientProxy(ln net.Listener, dialer *Dialer) *ClientProxy
NewClientProxy returns a new instance of ClientProxy.
func (*ClientProxy) Open ¶
func (p *ClientProxy) Open() error
Open starts the proxy listeners and waits for connections.
type DFA ¶
type DFA interface {
Capacity() int
Rank(s string) (rank *big.Int, err error)
Unrank(rank *big.Int) (ret string, err error)
NumWordsInSlice(n int) (numWords *big.Int, err error)
}
DFA represents the interface to the DFA ranker.
type Dialer ¶
type Dialer struct {
// Underlying NetDialer used for net connection.
Dialer NetDialer
// contains filtered or unexported fields
}
Dialer represents a client-side dialer that communicates over the marionette protocol.
type FSM ¶
type FSM interface {
io.Closer
// Document & FSM identifiers.
UUID() int
SetInstanceID(int)
InstanceID() int
// Party & networking.
Party() string
Host() string
Port() int
// The current state in the FSM.
State() string
// Returns true if State() == 'dead'
Dead() bool
// Moves to the next available state.
// Returns ErrNoTransition if there is no state to move to.
Next(ctx context.Context) error
// Moves through the entire state machine until it reaches 'dead' state.
Execute(ctx context.Context) error
// Restarts the FSM so it can be reused.
Reset()
// Returns an FTE cipher or DFA from the cache or creates a new one.
Cipher(regex string, n int) (Cipher, error)
DFA(regex string, msgLen int) (DFA, error)
// Returns the network connection attached to the FSM.
Conn() *BufferedConn
// Listen opens a new listener to accept data and drains into the buffer.
Listen() (int, error)
// Returns the stream set attached to the FSM.
StreamSet() *StreamSet
// Sets and retrieves key/values from the FSM.
SetVar(key string, value interface{})
Var(key string) interface{}
// Returns a copy of the FSM with a different format.
Clone(doc *mar.Document) FSM
Logger() *zap.Logger
}
FSM represents an interface for the Marionette state machine.
type Listener ¶
type Listener struct {
// Specifies directory for dumping stream traces. Passed to StreamSet.TracePath.
TracePath string
// contains filtered or unexported fields
}
Listener listens on a port and communicates over the marionette protocol.
type NetDialer ¶
type NetDialer interface {
Dial(network, address string) (net.Conn, error)
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
NetDialer is an abstract dialer. net.Dialer implements the NetDialer interface.
type PluginFunc ¶
PluginFunc represents a plugin in the MAR language.
func FindPlugin ¶
func FindPlugin(module, method string) PluginFunc
FindPlugin returns a plugin function by module & name.
type ServerProxy ¶
type ServerProxy struct {
// Host and port to proxy requests to.
// Ignored if a socks5 server is enabled.
Addr string
// Server used for proxying requests.
Socks5Server *socks5.Server
// contains filtered or unexported fields
}
ServerProxy represents a proxy between a marionette listener and another server.
func NewServerProxy ¶
func NewServerProxy(ln *Listener) *ServerProxy
NewServerProxy returns a new instance of ServerProxy.
func (*ServerProxy) Close ¶
func (p *ServerProxy) Close() error
func (*ServerProxy) Open ¶
func (p *ServerProxy) Open() error
type Stream ¶
type Stream struct {
// Stream verbosely logs to trace writer when set.
TraceWriter io.Writer
// contains filtered or unexported fields
}
Stream represents a readable and writable connection for plaintext data. Data is injected into the stream using cells which provide ordering and payload data. Implements the net.Conn interface.
func (*Stream) Close ¶
Close marks the stream as closed for writes. The server will close the read side.
func (*Stream) CloseWrite ¶
CloseWrite marks the stream as closed for writes.
func (*Stream) Enqueue ¶
Enqueue pushes a cell's payload on to the stream if it is the next sequence. Out of sequence cells are added to the queue and are read after earlier cells.
func (*Stream) ReadBufferLen ¶
ReadBufferLen returns the number of bytes in the read buffer.
func (*Stream) ReadCloseNotify ¶
func (s *Stream) ReadCloseNotify() <-chan struct{}
ReadCloseNotify returns a channel that sends when the stream has been closed for writing.
func (*Stream) ReadClosed ¶
ReadClosed returns true if the stream has been closed for reads.
func (*Stream) ReadNotify ¶
func (s *Stream) ReadNotify() <-chan struct{}
ReadNotify returns a channel that receives a notification when a new read is available.
func (*Stream) ReadWriteCloseNotified ¶
ReadWriteCloseNotified returns true if the stream is closed for read and write and has been notified.
func (*Stream) RemoteAddr ¶
RemoteAddr returns the remote address. Implements net.Conn.
func (*Stream) SetDeadline ¶
SetDeadline is a no-op. Implements net.Conn.
func (*Stream) SetReadDeadline ¶
SetReadDeadline is a no-op. Implements net.Conn.
func (*Stream) SetWriteDeadline ¶
SetWriteDeadline is a no-op. Implements net.Conn.
func (*Stream) Write ¶
Write appends b to the write buffer. This method will continue to try until the entire byte slice is written atomically to the buffer.
func (*Stream) WriteBufferLen ¶
WriteBufferLen returns the number of bytes in the write buffer.
func (*Stream) WriteCloseNotified ¶
WriteCloseNotified returns true if the stream has notified the peer connection of the end of stream.
func (*Stream) WriteCloseNotifiedNotify ¶
func (s *Stream) WriteCloseNotifiedNotify() <-chan struct{}
func (*Stream) WriteCloseNotify ¶
func (s *Stream) WriteCloseNotify() <-chan struct{}
WriteCloseNotify returns a channel that sends when the stream has been closed for writing.
func (*Stream) WriteClosed ¶
WriteClosed returns true if the stream has been requested to be closed for writes.
func (*Stream) WriteNotify ¶
func (s *Stream) WriteNotify() <-chan struct{}
WriteNotify returns a channel that receives a notification when a new write is available.
type StreamSet ¶
type StreamSet struct {
// Callback executed when a new stream is created.
OnNewStream func(*Stream)
// Directory for storing stream traces.
TracePath string
// contains filtered or unexported fields
}
StreamSet represents a multiplexer for a set of streams for a connection.
func NewStreamSet ¶
func NewStreamSet() *StreamSet
NewStreamSet returns a new instance of StreamSet.
func (*StreamSet) Dequeue ¶
Dequeue returns a cell containing data for a random stream's write buffer.
func (*StreamSet) Enqueue ¶
Enqueue pushes a cell onto a stream's read queue. If the stream doesn't exist then it is created.
func (*StreamSet) WriteNotify ¶
func (ss *StreamSet) WriteNotify() <-chan struct{}
WriteNotify returns a channel that receives a notification when a new write is available.
Source Files
Directories