iquota

package module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 5, 2020 License: BSD-3-Clause Imports: 11 Imported by: 0

README 

===============================================================================
Linux CLI tool for Isilon OneFS SmartQuota reporting
===============================================================================

------------------------------------------------------------------------
What is iquota?
------------------------------------------------------------------------

iquota is a command line tool and associated server application for reporting
quotas from Isilon nfs mounts using the OneFS API. Isilon is a scale out NAS
platform by EMC and when used in a Linux environment lacks adequate CLI tools
for reporting quotas from client mounts. For example, in an HPC environment
where Isilon is mounted via nfs on a front end machine running Linux, users
need a way to check and report on their quotas. This project aims to provide a
tool very similar to the native quota command in Linux but for Isilon nfs
mounts. This `post <https://community.emc.com/message/762183#762183>`_ sums up
essentially what this project aims to provide.  

The following diagram shows the basic architecture of iquota:

.. image:: docs/iquota-diagram.png

Linux clients mount Isilon /ifs over nfs. Users obtain kerberos credentials via
knit and run the iquota client command which connects to the iquota-server
(proxy) over HTTPS (using GSSAPI/SPNEGO for auth). The iquota-proxy server
validates the users kerberos credentials and requests quota information using
the OneFS API. The iquota-server connects to the OneFS API using a system
account which has read-only access to quota information using OneFS RBAC (role
based access control).

------------------------------------------------------------------------
Features
------------------------------------------------------------------------

- User/Group quota reporting from command line
- Kerberos based authentication
- Caching via redis

------------------------------------------------------------------------
Requirements
------------------------------------------------------------------------

- Isilon OneFS API (v7.2.1)
- Linux
- Kerberos
- sssd-ifp (SSSD InfoPipe responder)

------------------------------------------------------------------------
Install and configure iquota-server proxy
------------------------------------------------------------------------

*Note these docs are for CentOS 7.x. May need adjusting depending on your
flavor of Linux*

Download the RPM release `here <https://github.com/ubccr/iquota/releases>`_::

  $ rpm -Uvh iquota-server-0.x.x-x.el7.centos.x86_64.rpm

Create user account for accessing OneFS API
============================================

Create a role in OneFS that allows read-only access to quota information. For
example::

    # isi auth roles create --name=roQuotaUser --description='Readonly quota access'
    # isi auth roles modify --add-priv-ro=ISI_PRIV_QUOTA --role=roQuotaUser

Create a system user/pass account and add this user to the role. This will be
the user account the iquota-server will use to connect to OneFS API.

Setup Kerberos HTTP keytab
===========================

The iquota-server uses Kerberos authentication. You'll need to create a HTTP
service keytab file. For example::

    kadmin: addprinc -randkey HTTP/host.domain.com@YOUR-REALM.COM

If using FreeIPA you can run::

    $ ipa service-add HTTP/host.domain.com
    $ ipa-getkeytab -s master.domain.com -p HTTP/host.domain.com -k http.keytab

Configure sssd
===============

iquota-server uses sssd-ifp (SSSD InfoPipe responder) over DBUS to fetch the
unix groups for a given user. For more information on sssd-ifp see `here
<https://jhrozek.fedorapeople.org/sssd/1.12.0/man/sssd-ifp.5.html>`_.

Ensure the sssd-dbus package is installed::

    # yum install sssd-dbus

Configure sssd-ifp. Add the following lines to /etc/sssd/sssd.conf::

    [sssd]
    services = nss, sudo, pam, ssh, ifp

    [ifp]
    allowed_uids = iquota, root

Restart sssd to make the changes take effect::

    # systemctl restart sssd

You can test to ensure sssd-ifp is configured properly by running the following
command. The array of unix groups for the user should be displayed::

    # dbus-send --print-reply --system \
      --dest=org.freedesktop.sssd.infopipe \
      /org/freedesktop/sssd/infopipe \
      org.freedesktop.sssd.infopipe.GetUserGroups \
      string:username

       array [
          string "physics"
          string "compsci"
          string "users"
       ]

Configure iquota.yaml
=====================

Edit iquota configuration file. Add host, port, user/pass for OneFS API, path to
http keytab::

    $ vim /etc/iquota/iquota.yaml 
    onefs_host: "localhost"
    onefs_port: 8080
    onefs_user: "user"
    onefs_pass: "pass"
    keytab: "/path/to/http.keytab"
    [ edit to taste ]

It's highly recommended to run iquota-server using HTTPS. You'll need an SSL
cert/private_key either using FreeIPA's PKI, self-signed, or from a commercial
certificate authority. Creating SSL certs is outside the scope of this
document. You can also run iquota-server behind haproxy or Apache/Nginx.

Copy your SSL cert/private_key to the following directories and set correct
paths in /etc/iquota/iquota.yaml. The iquota-server binary will run as non-root
user (iquota) so need to ensure file perms are set correctly::

    $ mkdir /etc/iquota/{cert,private}
    $ cp my.crt /etc/iquota/cert/my.crt
    $ cp my.key /etc/iquota/private/my.key
    $ chmod 640 /etc/iquota/private/my.key
    $ chgrp iquota /etc/iquota/private/my.key

Start iquota-server service
============================

Start iquota-server service::

    $ systemctl restart iquota-server
    $ systemctl enable iquota-server

To view iquota-server system logs run::

    $ journalctl -u iquota-server

------------------------------------------------------------------------
Install iquota on all client machines mounting /ifs over nfs
------------------------------------------------------------------------

On all client machines mounting Isilon /ifs over nfs install the iquota client.
Download the RPM release `here <https://github.com/ubccr/iquota/releases>`_::

  $ rpm -Uvh iquota-0.x.x-x.el7.centos.x86_64.rpm

Edit iquota configuration file. Add URL for iquota-server::

    $ vim /etc/iquota/iquota.yaml 
    iquota_url: "http://host.domain.com"
    [ edit to taste ]

------------------------------------------------------------------------
Usage
------------------------------------------------------------------------

Check user/group quotas::

    $ kinit walterwhite
    Password for walterwhite@REALM:
    $ iquota -u -g
    User quotas:
    Filesystem  user               files      used     limit    grace 
    /ifs/user
                (default)                             2.0 GB   1 week 
                walterwhite           34    370 kB    2.0 GB   1 week 

    Group quotas:
    Filesystem  group              files      used     limit    grace 
    /ifs/projects
                (default)                             520 GB   1 week 
                hermanos               4    699 MB    520 GB   1 week

------------------------------------------------------------------------
Configure caching
------------------------------------------------------------------------

iquota-server can optionally be configured to cache results for a given time
period. This helps reduce the load on the OneFS API and provide better iquota
performance. To enable caching first install redis then update
/etc/iquota/iquota.yaml.

Install Redis (install from EPEL)::

    $ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    $ yum install redis
    $ systemctl restart redis
    $ systecmtl enable redis

Edit /etc/iquota/iquota.yaml and restart::

    $ vi /etc/iquota/iquota.yaml
    enable_caching: true

    $ systecmtl restart iquota-server

------------------------------------------------------------------------
References
------------------------------------------------------------------------

1. OneFS API Docs - https://community.emc.com/docs/DOC-48273

------------------------------------------------------------------------
License
------------------------------------------------------------------------

iquota is released under a BSD style license. See the LICENSE file.

Documentation

Overview

Package iquota provides a Go client for the Isilon OneFS API and command line tools for reporting SmartQuotas

Index

Constants

View Source 
const (
	SERVICE_PLATFORM  = "platform"
	SERVICE_NAMESPACE = "namespace"
	RESOURCE_SESSION  = "/session/1/session"
)
View Source 
const (
	RESOURCE_QUOTAS = "/platform/1/quota/quotas"

	QUOTA_TYPE_DIRECTORY     = "directory"
	QUOTA_TYPE_USER          = "user"
	QUOTA_TYPE_GROUP         = "group"
	QUOTA_TYPE_DEFAULT_USER  = "default-user"
	QUOTA_TYPE_DEFAULT_GROUP = "default-group"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

OneFS Client

func NewClient 

func NewClient(host string, port int, user, passwd, cacert string) *Client

Create new OneFS api client

func (*Client) FetchAllOverQuota added in v0.0.3 

func (c *Client) FetchAllOverQuota(path string) (*QuotaResponse, error)

Fetch all quotas that have exceeded one or more of their thresholds

func (*Client) FetchGroupQuota 

func (c *Client) FetchGroupQuota(path, group string) (*QuotaResponse, error)

Fetch Group Quota

func (*Client) FetchQuota 

func (c *Client) FetchQuota(path, qtype, persona string, resolveNames, exceededOnly bool) (*QuotaResponse, error)

Fetch Quota

func (*Client) FetchQuotaResume added in v0.0.3 

func (c *Client) FetchQuotaResume(resume string) (*QuotaResponse, error)

Continue fetching quotas from the previous request (

func (*Client) FetchUserQuota 

func (c *Client) FetchUserQuota(path, user string) (*QuotaResponse, error)

Fetch User Quota

func (*Client) NewSession 

func (c *Client) NewSession() (string, error)

Authenticate to OneFS API and create a session for multiple requests over a period of time.

func (*Client) Url 

func (c *Client) Url(resource string) string

Return URL for OneFS api given a resource

type IsiError 

type IsiError struct {
	Code    string `json:"code"`
	Message string `json:"message"`
}

OneFS API error

func (*IsiError) Error 

func (e *IsiError) Error() string

type Persona 

type Persona struct {
	// "Serialized form (e.g. 'UID:0', 'USER:name', 'GID:0', 'GROUP:wheel', 'SID:S-1-1').
	Id string `json:"id"`

	// Persona name, must be combined with type
	Name string `json:"name"`

	// Type of persona when using name "user", "group", "wellknown"
	Type string `json:"type"`
}

A persona consists of either a 'type' and 'name' or a 'ID'.

type Quota 

type Quota struct {
	// If true, SMB shares using the quota directory see the quota thresholds
	// as share size
	Container bool `json:"container"`

	// True if the quota provides enforcement, otherwise a accounting quota
	Enforced bool `json:"enforced"`

	// The system ID given to the quota
	Id string `json:"id"`

	// If true, quota governs snapshot data as well as head data
	IncludeSnapshots bool `json:"include_snapshots"`

	// Summary of notifications: 'custom' indicates one or more notification
	// rules available from the notifications sub-resource; 'default' indicates
	// system default rules are used; 'disabled' indicates that no
	// notifications will be used for this quota.
	Notifications string `json:"notifications"`

	// For user and group quotas, true if the quota is linked and controlled by
	// a parent default-* quota. Linked quotas cannot be modified until they
	// are unlinked
	Linked bool `json:"linked"`

	// The /ifs path governed
	Path string `json:"path"`

	// True if the accounting is accurate on the quota.  If false, this quota
	// is waiting on completion of a QuotaScan job
	Ready bool `json:"ready"`

	// If true, thresholds apply to data plus filesystem overhead required to
	// store the data (i.e. 'physical' usage)
	ThresholdsIncludeOverhead bool `json:"thresholds_include_overhead"`

	// The type of quota "directory", "user", "group", "default-user",
	// "default-group"
	Type string `json:"type"`

	// Nested types
	Persona   *Persona   `json:"persona"`
	Threshold *Threshold `json:"thresholds"`
	Usage     *Usage     `json:"usage"`
}

Isilon SmartQuota

type QuotaParams 

type QuotaParams struct {
	User  string `schema:"user"`
	Group string `schema:"group"`
	Type  string `schema:"type"`
	Path  string `schema:"path"`
}

type QuotaResponse 

type QuotaResponse struct {
	// List of errors
	Errors []*IsiError `json:"errors"`

	// List of Quotas
	Quotas []*Quota `json:"quotas"`

	// Continue returning results from previous call using this token (token
	// should come from the previous call, resume cannot be used with other
	// options).
	Resume string `json:"resume"`
}

Response returned from Quota endpoint

type QuotaRestResponse 

type QuotaRestResponse struct {
	Default *Quota   `json:"default"`
	Quotas  []*Quota `json:"quotas"`
}

type Threshold 

type Threshold struct {
	// Usage bytes at which notifications will be sent but writes will not be denied
	Advisory int `json:"advisory"`

	// True if the advisory threshold has been hit
	AdvisoryExceeded bool `json:"advisory_exceeded"`

	// Time at which advisory threshold was hit
	AdvisoryLast_exceeded int `json:"advisory_last_exceeded"`

	// Usage bytes at which further writes will be denied
	Hard int `json:"hard"`

	// True if the hard threshold has been hit
	HardExceeded bool `json:"hard_exceeded"`

	// Time at which hard threshold was hit
	HardLastExceeded int `json:"hard_last_exceeded"`

	// Usage bytes at which notifications will be sent and soft grace time will be started
	Soft int `json:"soft"`

	// True if the soft threshold has been hit
	SoftExceeded bool `json:"soft_exceeded"`

	// Time in seconds after which the soft threshold has been hit before writes will be denied
	SoftGrace int `json:"soft_grace"`

	// Time at which soft threshold was hit
	SoftLastExceeded int `json:"soft_last_exceeded"`
}

Quota thresholds

type Usage 

type Usage struct {
	// Number of inodes (filesystem entities) used by governed data
	Inodes int `json:"inodes"`

	// Apparent bytes used by governed data
	Logical int `json:"logical"`

	// Bytes used for governed data and filesystem overhead
	Physical int `json:"physical"`
}

Quota usage

Source Files

View all Source files

Directories

Path Synopsis
cmd
ipanfs
cache panfs user and group quotas
 cache panfs user and group quotas
iquota
display quota usage and limits
 display quota usage and limits
iquota-server
Proxy server to handle requests to Isilon OneFS API for smart quota reporting
 Proxy server to handle requests to Isilon OneFS API for smart quota reporting

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.