rpcx

rpcx

中文介绍

rpcx is a distributed RPC framework like Alibaba Dubbo and Weibo Motan. It is developed based on Go net/rpc and provides extra governance features.

When we talk about RPC frameworks, Dubbo is first framework we should introduced, and there is also Dubbox mantained by dangdang. Dubbo has been widely used in e-commerce companies in China, for example, Alibaba, Jingdong and Dangdang.

Though Dubbo has still used Spring 2.5.6.SEC03 and seems has not been supported by Alibaba no longer, some other companies still use it and maintained their branches.

DUBBO is a distributed service framework , provides high performance and transparent RPC remote service call. It is the core framework of Alibaba SOA service governance programs. There are 3,000,000,000 calls for 2,000+ services per day, and it has been widely used in various member sites of Alibaba Group.

Motan is open source now by Weibo. As Zhang Lei said, he is current main developer of Motan:

Motan started in 2013. There are 100 billion calls for hundreds of service callsevery day.

Those two RPC frameworks are developed by Java. There are other famous RPC frameworks such as thrift、finagle。

Goal of rpcx is implemented a RPC framework like Dubbo in Go ecosphere. It is developed by Go, and for Go.

It is a distributed、plugable RPC framework with governance (service discovery、load balancer、fault tolerance、monitor, etc.).

As you know, there are some RPC frameworks, for example, net/rpc、grpc-go、gorilla-rpc, Then why re-invent a wheel?

Although those Go RPC frameworks work well, but their function is relatively simple and only implement end-to end communications. Some product features of service management functions are lack, such as service discovery, Load balancing, fault tolerance.

So I created rpcx and expect it could become a RPC framework like Dubbo.

The similar project is go-micro.

What's RPC

From wikiPedia:

In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in another address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote.[1] This is a form of client–server interaction (caller is client, executer is server), typically implemented via a request–response message-passing system. The object-oriented programming analog is remote method invocation (RMI). The RPC model implies a level of location transparency, namely that calling procedures is largely the same whether it is local or remote, but usually they are not identical, so local calls can be distinguished from remote calls. Remote calls are usually orders of magnitude slower and less reliable than local calls, so distinguishing them is useful.

RPCs are a form of inter-process communication (IPC), in that different processes have different address spaces: if on the same host machine, they have distinct virtual address spaces, even though the physical address space is the same; while if they are on different hosts, the physical address space is different. Many different (often incompatible) technologies have been used to implement the concept.

Sequence of events during an RPC

1. The client calls the client stub. The call is a local procedure call, with parameters pushed on to the stack in the normal way.
2. The client stub packs the parameters into a message and makes a system call to send the message. Packing the parameters is called marshalling.
3. The client's local operating system sends the message from the client machine to the server machine.
4. The local operating system on the server machine passes the incoming packets to the server stub.
5. The server stub unpacks the parameters from the message. Unpacking the parameters is called unmarshalling.
6. Finally, the server stub calls the server procedure. The reply traces the same steps in the reverse direction.

There are two ways to implement RPC frameworks. One focusses on cross-language calls and the other focusses on service governance.

Dubbo、DubboX、Motan are RPC framework of service governance . Thrift、gRPC、Hessian、Hprose are RPC framework of cross-language calls.

rpcx is a RPC framework of service governance.

Features

more features

• bases on net/rpc. a Go net/prc project can be converted rpcx project whit few changes.
• Plugable. Features are implemented by Plugins such as service discovery.
• Commnuicates with TCP long connections.
• support many codec. for example, Gob、Json、MessagePack、gencode、ProtoBuf.
• Service dicovery. support ZooKeeper、Etcd.
• Fault tolerance：Failover、Failfast、Failtry.
• Load banlancer：support randomSelecter, RoundRobin, consistent hash etc.
• scalable.
• Other: metrics、log.
• Authorization.

Architecture

rpcx contains three roles : RPC Server，RPC Client and Registry.

• Server registers services on Registry
• Client queries service list and select a server from server list returned from Registry.
• When a Server is down, Registry can remove this server and then client can remove it too.

So far rpcx support zookeeper, etcd as Registry，Consul support is developing。

Benchmark

Test Environment

• CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 24 cores
• Memory: 16G
• OS: Linux Server-3 2.6.32-358.el6.x86_64, CentOS 6.4
• Go: 1.6.2

Test request is copied from protobuf and encoded to a proto message. Its size is 581 bytes. The response is same to test request but server has handled the decoding and encoding processing.

The concurrent clients are 100, 1000,2000 and 5000. Count of the total requests for all clients are 1,000,000.

Test Result

If you use too many clients, the throughput (transations per second) will be worse. It looks 1000 clients is feasible.

When you use clients, clients should be shared as possible.

you can use test code in _benchmark to test. server is used to start a server and client is used as clients via protobuf.

The above test is that client and server are running on the same mechine. If I run them on separate servers, test results are:

If they are running on cluster mode, one is for the client and two are for two servers, test results are:

Basically its throughput is greater than throughput of single node and less than 2 times of throughput of single node.

The below lists benchmarks of serialization libraries:

[root@localhost rpcx]# go test -bench . -test.benchmem
PASS
BenchmarkNetRPC_gob-16            100000             18742 ns/op             321 B/op          9 allocs/op
BenchmarkNetRPC_jsonrpc-16        100000             21360 ns/op            1170 B/op         31 allocs/op
BenchmarkNetRPC_msgp-16           100000             18617 ns/op             776 B/op         35 allocs/op
BenchmarkRPCX_gob-16              100000             18718 ns/op             320 B/op          9 allocs/op
BenchmarkRPCX_json-16             100000             21238 ns/op            1170 B/op         31 allocs/op
BenchmarkRPCX_msgp-16             100000             18635 ns/op             776 B/op         35 allocs/op
BenchmarkRPCX_gencodec-16         100000             18454 ns/op            4485 B/op         17 allocs/op
BenchmarkRPCX_protobuf-16         100000             17234 ns/op             733 B/op         13 allocs/op

Comparision with gRPC

gRPC is the RPC framework by Google. It support multiple programming lanaguage. I have compared three cases for prcx and gRPC. It shows rpcx is much better than gRPC.

Test results of rpcx has been listed on the above. Here is test results of gRPC.

one client and one server in a same machine

one client and one server in two machines

one client on a machine and three servers in three machines