gozilla

Gozilla

In development.....

About

Gozilla is an open source package repository software.

Backed by web servers and storage, one can manage personal and group repositories to store packages.

Could be compared to github (for source code) for generated assets, or bintray.

Assets are created in packages and match a package version.

Package access can be public or private, access is managed via repository access rights and publication status (published = public read access). Debian/Rpm generated repositories and Docker have same access management than related package.

Supported packages:

• raw: put any kind of file ;-)
• maven: upload/download maven releases and snapshots via mvn
• deb: creates debian repository to host .deb packages for apt support
• rpm: creates fedora/centos/etc. repository to host .rpm packages for yum/dnf support
• docker: optionally support a Docker registry
• git lfs: store git large files on gozilla

Supported storages:

• local (dev/test only)
• openstack swift
• add your own storage backend...

Sub components

• gozilla-lib: libraries
• gozilla-deb: deb repository handling
• gozilla-rpm: rpm repository handling
• gozilla-ui: web interface
• gozilla-cli: command-line tool
• gozilla-git-lfs: git lfs custom agent
• gozilla-operate: admin/operator cli and server

Setup

For elasticsearch, vm.max_map_count must be set:

sysctl -w vm.max_map_count=262144

Create first user and load elasticsearch schema with goz-init docker/process

• GOZ_ES_SCHEMA: path_to_schema.json file (default = ./schema.json)

You can create first admin user setting env variables:

Mandatory:

• GOZ_USER: admin user identifier to create
• GOZ_USER_PASSWORD: admin password

Optional:

• GOZ_USER_EMAIL: admin email
• GOZ_USER_CUSTOMER: user customer, default = "admins", will be created if not exists
• GOZ_CUSTOMER_PLAN: customer plan to use, default = "admin", will be create if not exists

Testing

With docker-compose, update gozilla-docker.yml file for configuration instead of gozilla.yml.

Create a .env file to set docker-compose needed env variables

• GOZ_DIR=/home/ubuntu/gozilla # path to gozilla install dir (to mount config, etc.)
• GOZ_GPG=/home/ubuntu/gnupg # path to gpg key (no password)

Optionally create a gozilla.env file to set env variables to overload with docker-compose the gozilla-docker.yml configuration values if needed

Then start components:

docker-compose up -d

Web interface is accessible on exposed nginx port. Now admin user can log via web interface or use gozilla-client (for self service operation) or gozilla-operate client to manage system.

At creation, an APIKey will be generated for all users (included admin). To get admin apikey:

curl --user *adminuser*:*adminpassword* http://localhost:8070/api/v1/user

Default will launch localstorage server and a nginx server (for dev/test only) To use Openstack Swift, update gozilla.yml file or override variables in gozilla.env. gozilla will redirect requests to nginx for upload/download. Nginx is configured to proxy back auth requests to gozilla.

If needed, create other subjects, customers, plans, users with goz-operate CLI (using admin user credentials)

To push a file using curl (replacing user id and apikey):

curl -L -H "X-GOZ-USER: test" -H "X-GOZ-APIKEY: 123" -X PUT -v -T "README.md" http://localhost:8080/api/v1/content/test/testmvn/pack1/1.0/README.md

gozilla-client helps user to manage repos and files (with upload/download)

API

authentication supports:

• basic auth with credentials user_id/apikey or user_id/password couples
• token based in Authorization header
• header based using X-GOZ-USER and X-GOZ-APIKEY headers

For token base, one can get a token with POST endpoint /api/{apiversion}/user/auth (sending {user: xxxx, password: yyyy})

Then place token in next requests:

Authorization: Bearer XXXXXX

It is more than advised to limit authentication containing password to strictly required transactions (recover lost apikey, etc.). User should use userid/apikey combinations and/or tokens

supported packages

debian

debian repo is for all package versions
debian repo fully regenerated when a package version is (re)published, replacing older versions repo
all debs must be for same distrib/component/archs

# upload deb
curl -L -H "X-GOZ-USER: test" -H "X-GOZ-APIKEY: 123" -X PUT -v -F file=@python3-biomaj3-core_3.0.19-1_all.deb http://localhost:8080/api/v1/debian/subject/repo/package/version/python3-biomaj3-core_3.0.19-1_all.deb?distribution=stretch&component=main&architectures=amd64

# access debian repo (repo url http://localhost:8080/api/v1/debian/subject/repo/package/public)
curl -v  -L -H "X-GOZ-USER: test" -H "X-GOZ-APIKEY: 123"  http://localhost:8080/api/v1/debian/test/testmvn/biomaj3/public/dists/stretch/InRelease

rpm

same as debian with rpm path, one directory per distrib/arch, repo config should point to distrib dir

docker

need dedicated fqdn and https proxy (forward to registry or goz)

image names must be formatted: subject-repo/package:version

conan

conan repository, versioning not supported

git-lfs

raw data repository can be used for git lfs storage

need extra custom git lfs agent

TODO

backend: manage personal vs org quota size? docker: set grafana admin password ui: manage expired token on page load backend[done]/ui[todo]: add to repo homepage/issues_page backend: on member add, do not add immediatly but send mail to user with token. On approval user is added.... backend: email config and sending backend/ui: edit user profile (update, apikey regeneration, email update, ..) backend/ui: manage products (contains multiple packages) backend/ui: authentication: add openid auth backend: deletion subject/repo/package/packageversion/file via API (can be done via CLI) backend/ui: password reset (forgot or password modification) backend: support relnotes/relnotes_link at package and package version level (if none on version, use package)

DONE backend: api regeneration backend: password update backend: add init process backend/cli: manage customer/plan (by operators), on ui display only, management via CLI => goz-operate backend/ui: quota management: set quota on buckets, get current usage (progress: added swift methods) backend: update subject/repo/package/packageversion info/members ui: create repo/package/version ui: manage members of subject/repo ui: manage visibility of subject/repo ui: store with user info, user plan stats: prometheus backend: disable customer update/... no operator operations (operators will use goz-operate) other: continuous scan to get subjects quota usage to store as stat in influxdb backend: add download/upload stats per package/version backend: fix subject creation ui: add download/upload instructions ui: show user profile add conan support handle anonymous read access on read endpoints authentication: CAS customer/plan management by admin via CLI (gozadmin) gozadmin: cli tool for admin maven repo generation debian repo generation rpm repo generation update subject/repo/... info docker registry proxy CAS authentication authentication checks user deletion not supported, can only be deactivated

curl

both uploads are accepted. For multipart, must use file parameter

curl -L -H "X-GOZ-USER: test" -H "X-GOZ-APIKEY: 123" -X PUT -v -F file=@/home/osallou/Development/NOSAVE/debian/debianmed/build-area/python3-biomaj3-core_3.0.19-1_all.deb "http://localhost:8080/api/v1/debian/test/testmvn/biomaj3/3/python3-biomaj3-core_3.0.19-1_all.deb?distribution=stretch&component=main&architectures=amd64"

curl -L -H "X-GOZ-USER: test" -H "X-GOZ-APIKEY: 123" -X PUT -v -T "python3-biomaj3-core_3.0.19-1_all.deb" "http://localhost:8080/api/v1/debian/test/testmvn/biomaj3/3/python3-biomaj3-core_3.0.19-1_all.deb?distribution=stretch&component=main&architectures=amd64"