Documentation
¶
Index ¶
- type BatchController
- func (ctl BatchController) BatchList() (map[string]string, error)
- func (ctl BatchController) BatchResourcesList(batchID string) ([]string, error)
- func (ctl BatchController) BatchResourcesSet(batchID string, resourceIDs []string) error
- func (ctl BatchController) BatchSet(batchID string, timestamp string) error
- func (ctl BatchController) Close()
- func (ctl BatchController) Destroy(batchID string) error
- func (ctl BatchController) ListenAndDestroy()
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BatchController ¶
type BatchController struct {
Topics topics.Controller
// contains filtered or unexported fields
}
func NewBatchController ¶
func NewBatchController() BatchController
func (BatchController) BatchList ¶
func (ctl BatchController) BatchList() (map[string]string, error)
BatchList returns a list of the current batches
func (BatchController) BatchResourcesList ¶
func (ctl BatchController) BatchResourcesList(batchID string) ([]string, error)
BatchResourcesList returns the list of resource ids of a batch
func (BatchController) BatchResourcesSet ¶
func (ctl BatchController) BatchResourcesSet(batchID string, resourceIDs []string) error
BatchResourcesSet records the resource ids of a batch in Redis
func (BatchController) BatchSet ¶
func (ctl BatchController) BatchSet(batchID string, timestamp string) error
BatchSet records a batch in Redis
func (BatchController) Close ¶
func (ctl BatchController) Close()
func (BatchController) Destroy ¶
func (ctl BatchController) Destroy(batchID string) error
Destroy deletes a batch by removing its Redis keys and Kafka topics. The Redis counter is not deleted right away for testing purposes but we set an expiration duration.
func (BatchController) ListenAndDestroy ¶
func (ctl BatchController) ListenAndDestroy()
ListenAndDestroy consumes load.* topics and destroy a batch when all resources have been loaded. Each time a new resource instance is loaded, the batch_id counter of the resource type resource_id in Redis is incremented. A batch counter is a Redis hash of key "batch:{batch_id}:counter" containing elements of keys "resource:{resource_id}:extracted" and "resource:{resource_id}:loaded". "resource:{resource_id}:extracted" refers to the number of resources of type {resource_id} extracted. "resource:{resource_id}:loaded" refers to the number of loaded resources of type {resource_id}. The list of resource types of a batch is in a Redis set "batch:{batch_id}:resources"
Source Files