hmsclient

package
v0.0.0-...-d9513b6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2020 License: Apache-2.0 Imports: 7 Imported by: 0

README

GO Hive Metastore Client

This is the Hive metastore client library for Golang

Installation

Standard go get:

$ go get github.com/akolb1/gometastore/hmsbench

Usage & Example

For API usage and examples, see GoDoc

Example usage:

import	(
    "log"
    "github.com/akolb1/gometastore/hmsclient"
)

func printDatabases() {
    client, err := hmsclient.Open("localhost", 9083)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()
    databases, err := client.GetAllDatabases()
    if err != nil {
        log.Fatal(err)
    }
    for _, d := range databases {
        fmt.Println(d)
    }
}

Documentation

Overview

Package hmsclient provides methods for accessing Hive Metastore over Thrift protocol. Only unsecured Thrift connections are supported.

Example usage: 

  import(
    "log"
    "github.com/terry-sm/gometastore/hmsclient"
  )

  func printDatabases() {
    client, err := hmsclient.Open("localhost", 9083)
    if err != nil {
      log.Fatal(err)
    }
    defer client.Close()

    databases, err := client.GetAllDatabases()
    if err != nil {
      log.Fatal(err)
	}
    for _, d := range databases {
      fmt.Println(d)
    }
  }
Example 
package main

import (
	"fmt"
	"log"

	"github.com/terry-sm/gometastore/hmsclient"
)

func main() {
	client, err := hmsclient.Open("localhost", 9083)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(client.GetAllDatabases())
}

Output:

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func MakePartition 

func MakePartition(table *hive_metastore.Table,
	values []string, parameters map[string]string,
	location string) (*hive_metastore.Partition, error)

MakePartition creates Partition object from ordered list of partition values. Only string values are currently supported. Parameters: 

table  - Hive table for which partition is added
values - List of partition values which should match partition schema

Types

type Database 

type Database struct {
	Name        string                       `json:"name"`
	Description string                       `json:"description,omitempty"`
	Owner       string                       `json:"owner,omitempty"`
	OwnerType   hive_metastore.PrincipalType `json:"ownerType,omitempty"`
	Location    string                       `json:"location"`
	Parameters  map[string]string            `json:"parameters,omitempty"`
}

Database is a container of other objects in Hive.

type MetastoreClient 

type MetastoreClient struct {
	// contains filtered or unexported fields
}

MetastoreClient represents client handle.

func Open 

func Open(host string, port int) (*MetastoreClient, error)

Open connection to metastore and return client handle.

Example 
package main

import (
	"fmt"
	"log"

	"github.com/terry-sm/gometastore/hmsclient"
)

func main() {
	client, err := hmsclient.Open("localhost", 9083)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(client.GetAllDatabases())
}

Output:

func (*MetastoreClient) AddPartition 

func (c *MetastoreClient) AddPartition(partition *hive_metastore.Partition) (*hive_metastore.Partition, error)

AddPartition adds partition to Hive table.

func (*MetastoreClient) AddPartitions 

func (c *MetastoreClient) AddPartitions(newParts []*hive_metastore.Partition) error

AddPartitions adds multipe partitions in a single call.

func (*MetastoreClient) AlterTable 

func (c *MetastoreClient) AlterTable(dbName string, tableName string,
	table *hive_metastore.Table) error

AlterTable modifies existing table with data from the new table

func (*MetastoreClient) Clone 

func (c *MetastoreClient) Clone() (client *MetastoreClient, err error)

Clone metastore client and return a new client with its own connection to metastore.

func (*MetastoreClient) Close 

func (c *MetastoreClient) Close()

Close connection to metastore. Handle can't be used once it is closed.

func (*MetastoreClient) CreateDatabase 

func (c *MetastoreClient) CreateDatabase(db *Database) error

CreateDatabase creates database with the specified name, description, parameters and owner.

func (*MetastoreClient) CreateTable 

func (c *MetastoreClient) CreateTable(table *hive_metastore.Table) error

CreateTable Creates HMS table

func (*MetastoreClient) DropDatabase 

func (c *MetastoreClient) DropDatabase(dbName string, deleteData bool, cascade bool) error

DropDatabases removes the database specified by name Parameters: 

dbName     - database name
deleteData - if true, delete data as well
cascade    - delete everything under the db if true

func (*MetastoreClient) DropPartition 

func (c *MetastoreClient) DropPartition(dbName string,
	tableName string, values []string, dropData bool) (bool, error)

DropPartition drops partition specified by values.

func (*MetastoreClient) DropPartitionByName 

func (c *MetastoreClient) DropPartitionByName(dbName string,
	tableName string, partName string, dropData bool) (bool, error)

DropPartitionByName drops partition specified by name.

func (*MetastoreClient) DropPartitions 

func (c *MetastoreClient) DropPartitions(dbName string,
	tableName string, partNames []string) error

DropPartitions drops multiple partitions within a single table. Partitions are specified by names.

func (*MetastoreClient) DropTable 

func (c *MetastoreClient) DropTable(dbName string, tableName string, deleteData bool) error

DropTable drops table. Parameters 

dbName     - Database name
tableName  - Table name
deleteData - if True, delete data as well

func (*MetastoreClient) GetAllDatabases 

func (c *MetastoreClient) GetAllDatabases() ([]string, error)

GetAllDatabases returns list of all Hive databases.

Example 
package main

import (
	"fmt"
	"log"

	"github.com/terry-sm/gometastore/hmsclient"
)

func main() {
	client, err := hmsclient.Open("localhost", 9083)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()
	fmt.Println(client.GetAllDatabases())
}

Output:

func (*MetastoreClient) GetAllTables 

func (c *MetastoreClient) GetAllTables(dbName string) ([]string, error)

GetAllTables returns list of all table names for a given database

func (*MetastoreClient) GetCurrentNotificationId 

func (c *MetastoreClient) GetCurrentNotificationId() (int64, error)

GetCurrentNotificationId returns value of last notification ID

func (*MetastoreClient) GetDatabase 

func (c *MetastoreClient) GetDatabase(dbName string) (*Database, error)

GetDatabase returns detailed information about specified Hive database.

func (*MetastoreClient) GetDatabases 

func (c *MetastoreClient) GetDatabases(pattern string) ([]string, error)

GetDatabases returns list of all databases matching pattern. The pattern is interpreted by HMS.

func (*MetastoreClient) GetNextNotification 

func (c *MetastoreClient) GetNextNotification(lastEvent int64,
	maxEvents int32) ([]*hive_metastore.NotificationEvent, error)

GetNextNotification returns next available notification.

func (*MetastoreClient) GetPartitionByName 

func (c *MetastoreClient) GetPartitionByName(dbName string, tableName string,
	partName string) (*hive_metastore.Partition, error)

GetPartitionByName returns Partition for the given partition name.

func (*MetastoreClient) GetPartitionNames 

func (c *MetastoreClient) GetPartitionNames(dbName string, tableName string, max int) ([]string, error)

GetPartitionNames returns list of partition names for a table.

func (*MetastoreClient) GetPartitions 

func (c *MetastoreClient) GetPartitions(dbName string, tableName string,
	maxCunt int) ([]*hive_metastore.Partition, error)

GetPartitions returns all (or up to maxCount partitions of a table.

func (*MetastoreClient) GetPartitionsByNames 

func (c *MetastoreClient) GetPartitionsByNames(dbName string, tableName string,
	partNames []string) ([]*hive_metastore.Partition, error)

GetPartitionsByNames returns multiple partitions specified by names.

func (*MetastoreClient) GetTable 

func (c *MetastoreClient) GetTable(dbName string, tableName string) (*hive_metastore.Table, error)

GetTable returns detailed information about the specified table

func (*MetastoreClient) GetTableMeta 

func (c *MetastoreClient) GetTableMeta(db string,
	table string, tableTypes []string) ([]*hive_metastore.TableMeta, error)

GetTableMeta returns list of tables matching specified search criteria. Parameters: 

db - database name pattern
table - table name pattern
tableTypes - list of Table types - should be either TABLE or VIEW

func (*MetastoreClient) GetTableObjects 

func (c *MetastoreClient) GetTableObjects(dbName string, tableNames []string) ([]*hive_metastore.Table, error)

GetTableObjects returns list of Table objects for the given database and list of table names.

func (*MetastoreClient) GetTables 

func (c *MetastoreClient) GetTables(dbName string, pattern string) ([]string, error)

GetTables returns list of tables matching given pattern for the given database. Matching is performed on the server side.

func (*MetastoreClient) GetTablesByType 

func (c *MetastoreClient) GetTablesByType(dbName string,
	table string, tableType string) ([]string, error)

GetTablesByType returns list of tables matching specified search criteria. Parameters: 

dbName - database name
table - table name pattern
tableType - Table type - should be either TABLE or VIEW

type PartitionBuilder 

type PartitionBuilder struct {
	Table      *hive_metastore.Table
	Parameters map[string]string
	Values     []string
	Location   string
}

func NewPartitionBuilder 

func NewPartitionBuilder(table *hive_metastore.Table, values []string) (*PartitionBuilder, error)

func (*PartitionBuilder) Build 

func (p *PartitionBuilder) Build() *hive_metastore.Partition

func (*PartitionBuilder) WithLocation 

func (pb *PartitionBuilder) WithLocation(location string) *PartitionBuilder

func (*PartitionBuilder) WithParameter 

func (pb *PartitionBuilder) WithParameter(key string, value string) *PartitionBuilder

func (*PartitionBuilder) WithParameters 

func (pb *PartitionBuilder) WithParameters(parameters map[string]string) *PartitionBuilder

type TableBuilder 

type TableBuilder struct {
	Db            string
	Name          string
	Type          TableType
	Serde         string
	Owner         string
	InputFormat   string
	OutputFormat  string
	Location      string
	Columns       []hive_metastore.FieldSchema
	PartitionKeys []hive_metastore.FieldSchema
	Parameters    map[string]string
}

TableBuilder provides builder pattern for table objects

func NewTableBuilder 

func NewTableBuilder(db string, tableName string) *TableBuilder

func (*TableBuilder) AsExternal 

func (tb *TableBuilder) AsExternal() *TableBuilder

Mark table as external

func (*TableBuilder) Build 

func (tb *TableBuilder) Build() *hive_metastore.Table

Build HMS Table object.

func (*TableBuilder) WithColumns 

func (tb *TableBuilder) WithColumns(columns []hive_metastore.FieldSchema) *TableBuilder

WithColumns specifies table columns

func (*TableBuilder) WithInputFormat 

func (tb *TableBuilder) WithInputFormat(format string) *TableBuilder

WithInputFormat specifies table input format

func (*TableBuilder) WithLocation 

func (tb *TableBuilder) WithLocation(location string) *TableBuilder

WithLocation specifies table location

func (*TableBuilder) WithOutputFormat 

func (tb *TableBuilder) WithOutputFormat(format string) *TableBuilder

WithOutputFormat specifies table output format

func (*TableBuilder) WithOwner 

func (tb *TableBuilder) WithOwner(owner string) *TableBuilder

WithOwner specifies table owner

func (*TableBuilder) WithParameter 

func (tb *TableBuilder) WithParameter(name string, value string) *TableBuilder

WithParameter adds table parameter

func (*TableBuilder) WithParameters 

func (tb *TableBuilder) WithParameters(parameters map[string]string) *TableBuilder

WithParameters specifies table parameters

func (*TableBuilder) WithPartitionKeys 

func (tb *TableBuilder) WithPartitionKeys(partKeys []hive_metastore.FieldSchema) *TableBuilder

WithPartitionKeys specifies table partition keys

func (*TableBuilder) WithSerde 

func (tb *TableBuilder) WithSerde(serde string) *TableBuilder

WithSerde specifies table serde

func (*TableBuilder) WithType 

func (tb *TableBuilder) WithType(t TableType) *TableBuilder

WithType specifies table type

type TableType 

type TableType int
const (
	TableTypeManaged TableType = iota
	TableTypeExternal
	TableTypeView
	TableTypeIndex
)

func (TableType) String 

func (val TableType) String() string

Source Files

View all Source files

Directories

Path Synopsis
thrift
gen-go/hive_metastore
Package hive_metastore provides raw access to HMS Thrift API.
 Package hive_metastore provides raw access to HMS Thrift API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.