README
¶
filecoin-data-transfer-service
Command-line application to create and manage data migration to Filecoin
The Filecoin Data Transfer Service (FTS) creates a queue of storage tasks based on local data and leverages the Powergate API to move data to deals on the Filecoin network.
Textile connects and extends Libp2p, IPFS, and Filecoin. Three interoperable technologies makeup Textile:
Join us on our public Slack channel for news, discussions, and status updates. Check out our blog for the latest posts and announcements.
Warning This project is still pre-release and is not ready for production usage.
Table of Contents
- Prerequisites
- Installation
- Localnet mode
- Production setup
- Tests
- Benchmark
- Contributing
- Changelog
- License
Prerequisites
To build from source, you need to have Go 1.14 or newer installed.
Installation
To build and install the FTS command-line tool, run:
make install
Usage
▶ fts run --help
Run a storage task pipeline to migrate large collections of data to Filecoin
Usage:
fts run [flags]
Flags:
-d, --dry-run run through steps without pushing data to Powergate API.
--folder string path to folder containing directories or files to process. (default "f")
-h, --help help for run
-i, --include-all include hidden files & folders from the top-level folder.
--ipfsrevproxy string the reverse proxy multiaddr of IPFS node. (default "127.0.0.1:6002")
-m, --mainnet sets staging limits based on localnet or mainnet.
-o, --output string the location to store intermediate and final results. (default "results.json")
-r, --resume resume tasks stored in the local results output file.
-e, --retry-errors retry tasks with error status.
Global Flags:
-t, --token string the user auth token
Examples
dry-run
fts run --folder ~/archives/tasks --dry-run
production run
fts run --folder ~/archives/tasks --mainnet
resume
fts run --resume
Output
Default output is stored in results.json. Each task is an entry in a results array that is continuously updated while the job is running.
{
"name": "high-res-pdfs",
"path": "/archives/tasks/n-q/high-res-pdfs",
"bytes": 9073000000,
"cid": "QmPWjS6YizvHTiP5oVAycvSPJBAhvSHnKsDHdr6ex54VzP",
"jobID": "441c9c6e-f5c9-4933-913f-5a739ccb0b38",
"stage": "Complete",
"deals": [
{
"id": "441c9c6e-f5c9-4933-913f-5a739ccb0b38",
"api_id": "f92ce374-dd51-42d9-9eb2-10a0ef24db66",
"cid": "QmPWjS6YizvHTiP5oVAycvSPJBAhvSHnKsDHdr6ex54VzP",
"status": 5,
"deal_info": [
{
"proposal_cid": "bafyreihgf3ezv3ameqthzurd2sfbt5to36iroazztlsbosiqjyphkjbjj4",
"state_id": 7,
"state_name": "StorageDealActive",
"miner": "f01000",
"piece_cid": "baga6ea4seaqch4hlqplqpcnjfl2rbzo7hgrbvahpwugkgzyfq24sgbkhntnggay",
"size": 16256,
"price_per_epoch": 7629,
"start_epoch": 6096,
"duration": 521601,
"deal_id": 4,
"activation_epoch": 579
}
],
"created_at": 1608080158
}
]
}
Design
Contributing
This project is a work in progress. As such, there's a few things you can do right now to help out:
- Ask questions! We'll try to help. Be sure to drop a note (on the above issue) if there is anything you'd like to work on and we'll update the issue to let others know. Also get in touch on Slack.
- Open issues, file issues, submit pull requests!
- Perform code reviews. More eyes will help a) speed the project along b) ensure quality and c) reduce possible future bugs.
- Take a look at the code. Contributions here that would be most helpful are top-level comments about how it should look based on your understanding. Again, the more eyes the better.
- Add tests. There can never be enough tests.
Changelog
Changelog is published to Releases.
Directories