README
¶
Haystack
A minimal clickstream service written in Go. Utilises AWS Kinesis as the cloud Kafka equivalent.
Package includes the producer service, consumer, and front-end examples to get you started.
Pull requests welcome!
AWS Authentication
In order to authenticate with your Kinesis instance, create a new key, download as a csv (rootkey.csv), and place in application root.
Sample Producer
curl -X POST -d '{"cid": "12345", "type": "pageview", "ref": "http://localhost/test.html", "context": ""}' http://localhost:5004/log
Dependencies
- Using Sendgridlab's kinesis library: https://github.com/sendgridlabs/go-kinesis
- Martini for producer endpoint: https://github.com/codegangsta/martini
Releases
- 0.0.1 initialisation
Documentation
¶
Index ¶
- Constants
- func CheckStream(k *kinesis.Kinesis, streamname string) bool
- func CreateStream(k *kinesis.Kinesis, tempStreamName string, shards int) bool
- func DeleteStream(k *kinesis.Kinesis, streamName string) bool
- func GenDict(data []string) map[string]string
- func GetMessages(k *kinesis.Kinesis, streamName string, shardID string) (out [][]byte, ok bool)
- func GetShardIterator(k *kinesis.Kinesis, streamName string, shardId string) string
- func InitKinesis() (*kinesis.Kinesis, bool)
- func LogFile(message string)
- func PassQuery(stream string, message string) (ok bool)
- func PrintRecords(records [][]byte)
- func SendMessage(k *kinesis.Kinesis, streamName string, msg *Message, comms chan bool)
- func ServerInit(quit chan bool)
- func StoreRecords(records []byte) bool
- type Keys
- type Message
- type Producer
Constants ¶
const (
MAX_MSG_SIZE uint32 = (50 << (10 * 1))
)
Variables ¶
This section is empty.
Functions ¶
func CheckStream ¶
CheckStream verifies whether a stream already exists
func CreateStream ¶
CreateStream creates a new stream
func DeleteStream ¶
DeleteStream deletes an existing stream
func GetMessages ¶
func GetShardIterator ¶
func InitKinesis ¶
InitKinesis initialises a connection to the Kinesis service
func SendMessage ¶
SendMessage generates a new JSON blob and sends to a stream
func ServerInit ¶
func ServerInit(quit chan bool)
ServerInit launches the Producer endpoint ...need to figure out how to do graceful shutdown.
func StoreRecords ¶
StoreRecords is an example of a local store
Types ¶
type Message ¶
type Message struct {
Uid string `json:"cid"` // user ID
Ref string `json:"ref"` // referral site
EventType string `json:"type"` // type of event
Data string `json:"context"` // data blob
Timestamp string `json:"timestamp"` // timestamp
}
Message represents the clickstream data to be passed to a stream.
Source Files