README
¶
spigo and simianviz
SIMulate Interactive Actor Network VIsualiZation - simianviz - also visualize the simian army in action. Follow @simianviz on twitter to get update notifications.
Originally called Simulate Protocol Interactions in Go - spigo - the name spigo is taken, however simianviz wasn't, so domains have been registered etc. and the name will transition over the coming months.
Using simianviz to view architecture diagrams
Launch the dependency graph visualization in your browser
For a local installation of the UI, with no network dependencies, you can start the service and browse localhost:8000 using:
$ cd ui
$ npm install
$ npm run dev
With a local install of the Neo4j graph database, you can write the graph into Neo4j with -n, and additionally write the flow data with -n -c. Subsections of the graph and flows can be viewed using Neo4j queries. The default local Neo4j instance is expected to be at localhost:7474, this can be overridden by setting NEO4JURL in the environment. The password must be set in the environment as well.
$ ./spigo -n
2016/05/21 08:50:51 Neo4j requires environment variable NEO4JPASSWORD is set
Running the simulation to generate new visualizations and metrics
$ ./spigo -h
Usage of ./spigo:
-a string
Architecture to create or read, fsm, migration, or read from json_arch/<arch>_arch.json (default "netflixoss")
-c Collect metrics and flows to json_metrics csv_metrics neo4j and via http: extvars
-cpuprofile string
Write cpu profile to file
-cpus int
Number of CPUs for Go runtime (default 4)
-d int
Simulation duration in seconds (default 10)
-f Filter output names to simplify graph by collapsing instances to services
-g Enable GraphML logging of nodes and edges to gml/<arch>.graphml
-j Enable GraphJSON logging of nodes and edges to json/<arch>.json
-kv string
Configuration key:value - chat:10ms sets default message insert rate
-m Enable console logging of every message
-n Enable Neo4j logging of nodes and edges
-p int
Pirate population for fsm or scale factor % for other architectures (default 100)
-r Reload graph from json/<arch>.json to setup architecture
-s int
Sequence number to create multiple runs for ui to step through in json/<arch><s>.json
-u string
Polling interval for Eureka name service, increase for large populations (default "1s")
-w int
Wide area regions to replicate architecture into, defaults based on 6 AWS region names (default 1)
Developing new architectures and simulation features
There were too many top level packages so a more hierachical directory structure was setup.
Directory structure
- spigo # binary built for MacOS
- spigo.go # main program
- actors # go code for packaged behaviors
- tooling # go support code
- ui # visualization code using d3 and js
- misc # scripts to run all tests and regenerate output
- json_arch # architecture definition files
- json # json dependency graph output
- json_metrics # flow, metrics and guesstimate output
- csv_metrics # histograms saved as tables
- png # images for readme
- archived # old files and packages
- gml # old graphml dependency graphs
The format of architecture definition files is described in json_arch/README
Docker compose (v1 and default to version2) yaml files can be converted to architecture json by extracting the dependency tree. Some editing to set the right relative counts and actor packages will be needed.
$ cd compose2arch; go install
$ compose2arch -file myarch.yaml > json_arch/myarch.json
Contributing and forking Spigo/SimianViz
Here's a useful guide to managing forked go programs on github. Thanks to Kurt, Priya and Henri for their initial contributions and advice.
Ideas for future enhancements and interfaces have been accumulated as Issues, please take a look there and feel free to add your own comments and ideas, or go ahead and implement something.
Next steps, papers and presentations
See this paper for some Occam code and results for the original version of this idea circa 2007.
The public launch was during a short keynote presentation at the March 2015 O'Reilly Software Architecture Conference: Monitoring Microservices - A Challenge - Video of the 10 minute talk
Most of the microservices presentations on Adrian's slideshare account discuss the current state of work.
The basic framework is in place, but more interesting behaviors, automonous running, and user input to control or stop the simulation haven't been added yet.
Next steps include connecting the output directly to the browser over a websocket so the dynamic behavior of the graph can be seen in real time. A lot of refactoring has cleaned up the code and structure in preparation for more interesting features.
Jason Brown's list of interesting Gossip papers might contain something interesting to try and implement...
Documentation
¶
Overview ¶
Package main for spigo - simulate protocol interactions in go. Terminology is a mix of NetflixOSS, promise theory and flying spaghetti monster lore
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
actors
|
|
|
denominator
Package denominator simulates a global DNS service Takes incoming traffic and spreads it over elb's in multiple regions
|
Package denominator simulates a global DNS service Takes incoming traffic and spreads it over elb's in multiple regions |
|
edda
Package edda Logs the architecture configuration (nodes and links) as it evolves
|
Package edda Logs the architecture configuration (nodes and links) as it evolves |
|
elb
Package elb simulates an Elastic Load Balancer Takes incoming traffic and spreads it over microservices in three availability zones
|
Package elb simulates an Elastic Load Balancer Takes incoming traffic and spreads it over microservices in three availability zones |
|
eureka
Package eureka is a service registry for the architecture configuration (nodes) as it evolves and passes data to edda for logging nodes and edges
|
Package eureka is a service registry for the architecture configuration (nodes) as it evolves and passes data to edda for logging nodes and edges |
|
karyon
Package karyon simulates a generic business logic microservice Takes incoming traffic and calls into dependent microservices in a single zone
|
Package karyon simulates a generic business logic microservice Takes incoming traffic and calls into dependent microservices in a single zone |
|
monolith
Package monolith simulates a monolithic business logic microservice Takes incoming traffic and calls into dependent microservices across all zones
|
Package monolith simulates a monolithic business logic microservice Takes incoming traffic and calls into dependent microservices across all zones |
|
packagenames
Package packagenames has name definitions for packages that simulate behaviors
|
Package packagenames has name definitions for packages that simulate behaviors |
|
pirate
Package pirate is a participant in the social network, listens to the FSM and to other pirates chats at a variable rate by giving gold and namedropping friends
|
Package pirate is a participant in the social network, listens to the FSM and to other pirates chats at a variable rate by giving gold and namedropping friends |
|
priamCassandra
Package priamCassandra simulates a cassandra cluster with NetflixOSS Priam Takes incoming traffic and calls into cross zone and cross region nodes
|
Package priamCassandra simulates a cassandra cluster with NetflixOSS Priam Takes incoming traffic and calls into cross zone and cross region nodes |
|
staash
Package staash simulates a generic data access layer microservice Takes incoming traffic and calls into dependent microservices in a single zone
|
Package staash simulates a generic data access layer microservice Takes incoming traffic and calls into dependent microservices in a single zone |
|
store
Package store simulates a generic business logic microservice Takes incoming traffic and calls into dependent microservices in a single zone
|
Package store simulates a generic business logic microservice Takes incoming traffic and calls into dependent microservices in a single zone |
|
zuul
Package zuul simulates a api proxy microservice Takes incoming traffic and calls into dependent microservices in a single zone
|
Package zuul simulates a api proxy microservice Takes incoming traffic and calls into dependent microservices in a single zone |
|
Package compose reads docker-compose yaml files and generates architecture json output Would use https://github.com/docker/libcompose if it wasn't so mind-numbingly complicated
|
Package compose reads docker-compose yaml files and generates architecture json output Would use https://github.com/docker/libcompose if it wasn't so mind-numbingly complicated |
|
utility to read a docker compose yaml file and write out an arch_json
|
utility to read a docker compose yaml file and write out an arch_json |
|
tooling
|
|
|
archaius
Package archaius holds all configuration information, named after the netflixoss project
|
Package archaius holds all configuration information, named after the netflixoss project |
|
architecture
Package architecture reads a microservice architecture definition from a file It creates and controls a collection of aws and netflix application microservices
|
Package architecture reads a microservice architecture definition from a file It creates and controls a collection of aws and netflix application microservices |
|
asgard
Package asgard contains shared code that is used to create an aws/netflixoss style architecture (including lamp and migration)
|
Package asgard contains shared code that is used to create an aws/netflixoss style architecture (including lamp and migration) |
|
chaosmonkey
Package chaosmonkey deletes nodes
|
Package chaosmonkey deletes nodes |
|
collect
Package collect throughput and response times using Go-Kit Metrics
|
Package collect throughput and response times using Go-Kit Metrics |
|
dhcp
Package dhcp provides individual IP addresses by name, simulating dhcp
|
Package dhcp provides individual IP addresses by name, simulating dhcp |
|
flow
Package flow processes gotocol context information to collect and export request flows across the system
|
Package flow processes gotocol context information to collect and export request flows across the system |
|
fsm
Package fsm implements a Flexible State Manager (Flying Spaghetti Monster) It creates and controls a large social network of pirates via channels (the noodly touch) or reads in a network from a json file.
|
Package fsm implements a Flexible State Manager (Flying Spaghetti Monster) It creates and controls a large social network of pirates via channels (the noodly touch) or reads in a network from a json file. |
|
gotocol
Package gotocol provides protocol support to send a variety of commands listener channels and types over a single channel type
|
Package gotocol provides protocol support to send a variety of commands listener channels and types over a single channel type |
|
graphgexf
Package graphgexf writes nodes and edges to spigo.graphml in the XML based GEXFformat used by the freely available Gephi tool
|
Package graphgexf writes nodes and edges to spigo.graphml in the XML based GEXFformat used by the freely available Gephi tool |
|
graphjson
Package graphjson saves and loads architectures to and from graph json files
|
Package graphjson saves and loads architectures to and from graph json files |
|
graphml
Package graphml writes nodes and edges to spigo.graphml in the XML based GraphML format used by the freely available yEdit tool from yfiles.com
|
Package graphml writes nodes and edges to spigo.graphml in the XML based GraphML format used by the freely available yEdit tool from yfiles.com |
|
graphneo4j
Package graphneo4j saves architectures to neo4j
|
Package graphneo4j saves architectures to neo4j |
|
handlers
Package handlers contains common code used for message handling
|
Package handlers contains common code used for message handling |
|
migration
Package migration implements a simulation of migration to a global large scale microservice architecture It creates and controls a collection of aws, lamp, netflixoss and netflix application microservices or reads in a network from a json file.
|
Package migration implements a simulation of migration to a global large scale microservice architecture It creates and controls a collection of aws, lamp, netflixoss and netflix application microservices or reads in a network from a json file. |
|
names
Package names creates and accesses the nanoservice naming hierarchy
|
Package names creates and accesses the nanoservice naming hierarchy |
|
ribbon
Package ribbon is named after the NetflixOSS routing and load balancing project, functions for routing traffic
|
Package ribbon is named after the NetflixOSS routing and load balancing project, functions for routing traffic |
|
usl
Package usl implements universal scalability law functions as described in https://github.com/VividCortex/ebooks/blob/master/scalability.pdf
|
Package usl implements universal scalability law functions as described in https://github.com/VividCortex/ebooks/blob/master/scalability.pdf |