Rosetta Bitcoin
ROSETTA-BITCOIN IS CONSIDERED ALPHA SOFTWARE.
USE AT YOUR OWN RISK! COINBASE ASSUMES NO RESPONSIBILITY OR LIABILITY IF THERE IS A BUG IN THIS IMPLEMENTATION.
Overview
rosetta-bitcoin
provides a reference implementation of the Rosetta API for
Bitcoin in Golang. If you haven't heard of the Rosetta API, you can find more
information here.
Features
- Rosetta API implementation (both Data API and Construction API)
- UTXO cache for all accounts (accessible using the Rosetta
/account/balance
API)
- Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address
System Requirements
rosetta-bitcoin
has been tested on an AWS c5.2xlarge instance.
This instance type has 8 vCPU and 16 GB of RAM.
Network Settings
To increase the load rosetta-bitcoin
can handle, it is recommended to tune your OS
settings to allow for more connections. On a linux-based OS, you can run the following commands (source):
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_max_syn_backlog=10000
sysctl -w net.core.somaxconn=10000
sysctl -p (when done)
We have not tested rosetta-bitcoin
with net.ipv4.tcp_tw_recycle
and do not recommend
enabling it.
You should also modify your open file settings to 100000
. This can be done on a linux-based OS
with the command: ulimit -n 100000
.
Memory-Mapped Files
rosetta-bitcoin
uses memory-mapped files to
persist data in the indexer
. As a result, you must run rosetta-bitcoin
on a 64-bit
architecture (the virtual address space easily exceeds 100s of GBs).
If you receive a kernel OOM, you may need to increase the allocated size of swap space
on your OS. There is a great tutorial for how to do this on Linux here.
Usage
As specified in the Rosetta API Principles,
all Rosetta implementations must be deployable via Docker and support running via either an
online
or offline
mode.
YOU MUST INSTALL DOCKER FOR THE FOLLOWING INSTRUCTIONS TO WORK. YOU CAN DOWNLOAD DOCKER HERE.
Install
Running the following commands will create a Docker image called rosetta-bitcoin:latest
.
From GitHub
To download the pre-built Docker image from the latest release, run:
curl -sSfL https://raw.githubusercontent.com/coinbase/rosetta-bitcoin/master/install.sh | sh -s
Do not try to install rosetta-bitcoin using GitHub Packages!
From Source
After cloning this repository, run:
make build-local
Run
Running the following commands will start a Docker container in
detached mode with
a data directory at <working directory>/bitcoin-data
and the Rosetta API accessible at port 8080
.
Mainnet:Online
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 8333:8333 rosetta-bitcoin:latest
If you cloned the repository, you can run make run-mainnet-online
.
Mainnet:Offline
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 rosetta-bitcoin:latest
If you cloned the repository, you can run make run-mainnet-offline
.
Testnet:Online
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 18333:18333 rosetta-bitcoin:latest
If you cloned the repository, you can run make run-testnet-online
.
Testnet:Offline
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 rosetta-bitcoin:latest
If you cloned the repository, you can run make run-testnet-offline
.
Architecture
rosetta-bitcoin
uses the syncer
, storage
, parser
, and server
package
from rosetta-sdk-go
instead
of a new Bitcoin-specific implementation of packages of similar functionality. Below
you can find a high-level overview of how everything fits together:
Optimizations
- Automatically prune bitcoind while indexing blocks
- Reduce sync time with concurrent block indexing
- Use Zstandard compression to reduce the size of data stored on disk without needing to write a manual byte-level encoding
Concurrent Block Syncing
To speed up indexing, rosetta-bitcoin
uses concurrent block processing with a "wait free" design (using the channels function instead of the sleep function to signal which threads are unblocked). This allows rosetta-bitcoin
to fetch multiple inputs from disk while it waits for inputs that appeared in recently processed blocks to save to disk.
Testing with rosetta-cli
To validate rosetta-bitcoin
, install rosetta-cli
and run one of the following commands:
rosetta-cli check:data --configuration-file rosetta-cli-conf/testnet/config.json
- This command validates that the Data API information in the testnet
network is correct. It also ensures that the implementation does not miss any balance-changing operations.
rosetta-cli check:construction --configuration-file rosetta-cli-conf/testnet/config.json
- This command validates the blockchain’s construction, signing, and broadcasting.
rosetta-cli check:data --configuration-file rosetta-cli-conf/mainnet/config.json
- This command validates that the Data API information in the mainnet
network is correct. It also ensures that the implementation does not miss any balance-changing operations.
Issues
Interested in helping fix issues in this repository? You can find to-dos in the Issues section with the help wanted
tag. Be sure to reach out on our community before you tackle anything on this list.
Development
make deps
to install dependencies
make test
to run tests
make lint
to lint the source code
make salus
to check for security concerns
make build-local
to build a Docker image from the local context
make coverage-local
to generate a coverage report
License
This project is available open source under the terms of the Apache 2.0 License.
© 2021 Coinbase