README
¶
This doc discusses about the common mechanisms shared by all catalog services.
Monitoring
FireCamp integrates with Telegraf to monitor the stateful services. You could create one telegraf service for one stateful service. For more details, please refer toFireCamp Telegraf.
Max Memory and CPU
In general, you could set the reserved and max memory for the stateful service container. By default, the reserved memory is 256MB, and the max memory is unlimited. You could set the them when creating the service, "reserve-memory" and "max-memory". For the JVM based service such as Kafka, the reserved memory will be the max size of JVM heap size and reserved memory, max(JVM heap size, reserved memory).
You could also set the reserved and max cpu for the stateful service container, "reserve-cpuunits" and "max-cpuunits". AWS ECS assigns 1,024 cpu units for every CPU core. By default, the reserved cpu units is 256, and no limit for the max cpu units.
Note: AWS ECS does not support to limit the max cpu units. You could only set the reserved cpu units on ECS.
Logging
FireCamp integrates with cloud logs. On AWS, the service logs are sent to AWS CloudWatch.
AWS ECS and Docker Swarm have the same log group and stream names.
- Every stateful service will have its own log group: clustername-servicename-serviceUUID. Every stateful service member will get the stream name: membername/hostname/containerID.
- The stateful service task, such as the init task, will use the same log group with the stateful service. The log stream has the task type as prefix.
- The stateless service, such as the FireCamp management service, Kafka Manager service, etc, will have the same log group name with the stateful service. The log stream has the service name as prefix.
K8s will be slightly different. K8s does not support custom log driver like AWS ECS and Docker Swarm supports. K8s simply writes the log to the local files. The Fluentd DaemonSet is used to send the logs to CloudWatch. The log group and stream are different, as K8s has namespace and init container concepts.
K8s log group: clustername-namespace-servicename-serviceUUID. K8s log stream: membername/containername/hostname/containerID. FireCamp will set the init/stop container name with the init and stop prefix in the pod. Please refer to FireCamp K8s Fluentd ConfigMap for the format.
FireCamp System Configs
Transparent Huge Pages
The THP (Transparent Huge Pages) is set to madvise. THP might impact the performance for some services such as MongoDB and Redis. While, some service such as Cassandra could benefit from THP.
File Descriptors
The file descriptors are increased 100,000, as the stateful service such as Kafka/ElasticSearch may use a large number of files.
Number of Threads
The number of threads are increased to 64,000. So the stateful service could use as many threads as needed.
Tcp backlog
Both /proc/sys/net/core/somaxconn and /proc/sys/net/ipv4/tcp_max_syn_backlog are increased to 512.
On AWS ECS, the service container directly uses the host network stack for better network performance. So the service container could also see the somaxconn to be 512. While Docker Swarm service creation does not support host network. The docker run command supports modifying the sysctl options, but Docker Swarm does not support the sysctl options yet.
For Docker Swarm, we would have to use some workarounds before Docker Swarm supports the sysctl option. The workaround is: mount host /proc inside a container and configure network stack from within. Create the service with "-v /proc:/writable-proc". Then could change somaxconn inside container, "echo 512 > /writable-proc/sys/net/core/somaxconn", "echo 512 > /writable-proc/sys/net/ipv4/tcp_max_syn_backlog".
Tcp tuning
The idle connection timeout is adjusted. The socket memory is increased to handle thousands of concurrent connections. ''' net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.core.rmem_default=16777216 net.core.wmem_default=16777216 net.core.optmem_max=40960 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 '''
Memory Configs
swapping is bad for performance and stability. For example, if the JVM heap is swapped out to disk, the JVM garbage collections may last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. FireCamp sets the vm.swappiness to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions. And the vm.overcommit_memory is disabled. If the system memory is not enough for the stateful service, it would be better to add more memory.
mmap may be used by some stateful services such as ElasticSearch, Cassandra. The vm.max_map_count is increased to 1048575.
memlock is set to unlimited. So the service such as Cassandra could lock the memory in RAM and never get moved to the swap disk.
Documentation
¶
Index ¶
- Constants
- func GenServiceMemberHosts(cluster string, service string, replicas int64) string
- func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string
- func GenServiceMemberURIs(cluster string, service string, replicas int64, port int64) string
- func GenStatelessServiceReplicaConfigs(cluster string, service string, replicas int) []*manage.ReplicaConfig
- func IsMemberConfigFile(filename string) bool
- func IsServiceConfigFile(filename string) bool
- func UpdateServiceConfigHeapAndJMX(oldContent string, heapSizeMB int64, jmxUser string, jmxPasswd string) string
- func ValidateUpdateOptions(heapSizeMB int64, jmxUser string, jmxPasswd string) error
Constants ¶
const ( // The service configurable variables and some service specific variables SERVICE_FILE_NAME = "service.conf" // The specific variables for one service member MEMBER_FILE_NAME = "member.conf" BindAllIP = "0.0.0.0" // the default password is uuid for better security // every service will define its own jmx port JmxDefaultRemoteUser = "jmxuser" JmxReadOnlyAccess = "readonly" JmxReadWriteAccess = "readwrite" )
Variables ¶
This section is empty.
Functions ¶
func GenServiceMemberHosts ¶ added in v0.9.5
GenServiceMemberHosts creates the hostname list of all service members, example: myzoo-0.t1-firecamp.com,myzoo-1.t1-firecamp.com Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.
func GenServiceMemberHostsWithPort ¶ added in v0.9.5
func GenServiceMemberHostsWithPort(cluster string, service string, replicas int64, port int64) string
GenServiceMemberHostsWithPort creates the hostname:port list of all service members, example: myzoo-0.t1-firecamp.com:2181,myzoo-1.t1-firecamp.com:2181 Note: this currently works for the service that all members have a single name format, such as ZooKeeper, Kafka, etc. For service such as ElasticSearch, MongoDB, that has different name formats, this is not suitable.
func GenServiceMemberURIs ¶ added in v0.9.5
GenServiceMemberURIs creates the list of URIs for all service members, example: http://myes-0.t1-firecamp.com:9200,http://myes-1.t1-firecamp.com:9200
func GenStatelessServiceReplicaConfigs ¶ added in v0.9.5
func GenStatelessServiceReplicaConfigs(cluster string, service string, replicas int) []*manage.ReplicaConfig
GenStatelessServiceReplicaConfigs generates the replica configs for the stateless service.
func IsMemberConfigFile ¶ added in v0.9.6
IsMemberConfigFile checks if the file is the member.conf file
func IsServiceConfigFile ¶ added in v0.9.6
IsServiceConfigFile checks if the file is the service.conf file
Types ¶
This section is empty.
Source Files
Directories