python-cli

Flywheel CLI

flywheel-cli is a library and command line interface for interacting with a Flywheel site.

One part of the CLI implemented in Python, the other part implemented in Go. In the future new features should be implemented in Python. The commands that implemented in Go will be migrated to Python later. The goal is to use the Go part only as a wrapper to easily create cross-platform binaries.

Running

poetry install
poetry shell
fw <command>

Setup development environment

poetry install
poetry shell
pre-commit install

Building

Building cross-platform binaries

Requirements

In order to build the binaries, the following things are needed

• python 3.8 with virtualenv and pip
• golang 1.12.5
• gox: go get github.com/mitchellh/gox
• go-bindata: go get github.com/jteeuwen/go-bindata/...
• glide: go get github.com/Masterminds/glide
• musl-gcc (for make archive)
• upx (for make archive, compress)

Build binaries

make

The binary will be compiled to bin/{OS}_{ARCH}/fw.

Fresh build

make clean
make

Publishing

Publishing uses an MR / Pipeline based solution to make it easy to create a new release.

The workflow is the following:

• Running a pipeline with the RELEASE variable set to the desired version
• The pipeline will create a new merge request (MR)
• Accepting the merge request will create a new tag, build the required components and update Umbrella

Creating the release MR (running the pipelin)

• Navigate to the project's CI/CD - Pipelines page
• Click the blue Run pipeline button (top right)
• Select the target branch from the drop-down:
  • main/master for normal releases (default)
  • hotfix-X.Y for hotfixes
• Enter a variable named RELEASE with the desired version (eg. 1.2.3-rc.4)
• Click the blue Run pipeline button

Relase merge request

• Navigate to the project's Merge requests page
• A new MR should have been created with the title Release <version specified above>
• The MR's description should have a section MR Changelog with the changelog for verification
• The Infra release update section specifies which branch (RELEASE_BRANCH=) and commit message (RELEASE_COMMIT=) will be used for the Umbrella update. If any of the variable is empty there will be no Umbrella update.
• To merge the MR you have to Approve (blue button) and Merge (green button).

See qa-ci repo for more detailes.

Updating

poetry update

Tests

Testing principles

• New features should be covered with tests
• Tests should use the minimum amount of mocking

Running tests

poetry run pytest tests
OR
poetry run pre-commit run -a test:pre-commit:pytest

Running linting, formatting, etc

poetry run pre-commit run -a test:flywheel-lint
poetry run pre-commit run -a test:pre-commit:isort
poetry run pre-commit run -a test:pre-commit:pylint
poetry run pre-commit run -a test:pre-commit:pylint-tests

Running all pre-commit hooks

poetry run pre-commit run -a

Dependencies

Packages

• flywheel-bids
• flywheel-migration
• flywheel-sdk
• fw-storage
• fw-utils

Dev

• fw-http-testserver

Components/softwate

• Core

To be able to communicate with CORE, one has to use fw login with an API key. The key can be found on the profile page.

• SQLite / PostgreSQL

Ingest uses a database to store information. It uses SQLite for local ingests, and PostgreSQL for cluster ingest

Entrypoint

flywheel_cli.main:main

Software components depend on this

• Ingest

Ingest Tech Overview

Branches

• master: the new release is tagged from this branch, MRs are merged into master
• hotfix-<version>: used for backporting features from master

Application data

User data

The logged in user info is stored in ~/.config/flywheel/user.json

Cache

The binary contains the python executable and dependencies, which are extracted upon the first start.

The extracted data is stored in ~/.cache/flywheel. Logs are also stored in the same directory.

Commands

Served from Python

• login
• logout
• ls
• cp
• import
• export
• download
• upload
• sync
• ingest
• deid
• status
• version
• admin

Served from Go

• batch
• bids
• gear
• job

Code

Directory structure

• flywheel_cli Python code

  • commands All command definition available with fw <command> is defined here. The __init__:add_commands adds all the subcommands.

  • exchange Helpers/utils for the gear commands

  • importers Functions for the fw import commands. This functionality is now deprecated in favour of fw ingest

  • ingest Functions for the fw ingest command

    • client Functions that provide Database actions (db.py, db_transactions.py) and provide API actions for following a cluster ingest (api.py)

    • scanners Functions for scanning (parsing) for different types of strategies (dicom, template, folder, project)

    • strategies

    • tasks Task definitions. Each ingest has multiple stages, each stage might run several jobs in parallel. The default flow is: Configuring->Scanning->Resolving->Preparing->Uploading->Finalizing

  • sync Functions for the fw sync command.

  • walker Functions that are providing a unified way for listing/iterating over files from different sources, like S3, GCS, Azure and local filesístem.

• go Go code

• tests Python tests and test data