luci: go.chromium.org/luci/appengine/mapper/internal/tasks Index | Files

package tasks

import "go.chromium.org/luci/appengine/mapper/internal/tasks"

Package tasks contains definition of task queue tasks used by the mapper.

Index

Package Files

gen.go tasks.pb.go

type FanOutShards Uses

type FanOutShards struct {
    JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

FanOutShards enqueues a bunch of ProcessShard named tasks (one per shard).

Enqueued transactionally by SplitAndLaunch after it has constructed shards.

func (*FanOutShards) Descriptor Uses

func (*FanOutShards) Descriptor() ([]byte, []int)

func (*FanOutShards) GetJobId Uses

func (m *FanOutShards) GetJobId() int64

func (*FanOutShards) ProtoMessage Uses

func (*FanOutShards) ProtoMessage()

func (*FanOutShards) Reset Uses

func (m *FanOutShards) Reset()

func (*FanOutShards) String Uses

func (m *FanOutShards) String() string

func (*FanOutShards) XXX_DiscardUnknown Uses

func (m *FanOutShards) XXX_DiscardUnknown()

func (*FanOutShards) XXX_Marshal Uses

func (m *FanOutShards) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*FanOutShards) XXX_Merge Uses

func (m *FanOutShards) XXX_Merge(src proto.Message)

func (*FanOutShards) XXX_Size Uses

func (m *FanOutShards) XXX_Size() int

func (*FanOutShards) XXX_Unmarshal Uses

func (m *FanOutShards) XXX_Unmarshal(b []byte) error

type ProcessShard Uses

type ProcessShard struct {
    JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
    ShardId              int64    `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
    TaskNum              int64    `protobuf:"varint,3,opt,name=task_num,json=taskNum,proto3" json:"task_num,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

ProcessShard sequentially reads the entities belonging to a key range assigned to a shard and applies the mapper to them.

Upon reaching 1 min mark, relaunches itself, increasing task_num. Thus ProcessShard is actually a chain of tasks that runs as long as needed to completely process the shard.

func (*ProcessShard) Descriptor Uses

func (*ProcessShard) Descriptor() ([]byte, []int)

func (*ProcessShard) GetJobId Uses

func (m *ProcessShard) GetJobId() int64

func (*ProcessShard) GetShardId Uses

func (m *ProcessShard) GetShardId() int64

func (*ProcessShard) GetTaskNum Uses

func (m *ProcessShard) GetTaskNum() int64

func (*ProcessShard) ProtoMessage Uses

func (*ProcessShard) ProtoMessage()

func (*ProcessShard) Reset Uses

func (m *ProcessShard) Reset()

func (*ProcessShard) String Uses

func (m *ProcessShard) String() string

func (*ProcessShard) XXX_DiscardUnknown Uses

func (m *ProcessShard) XXX_DiscardUnknown()

func (*ProcessShard) XXX_Marshal Uses

func (m *ProcessShard) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ProcessShard) XXX_Merge Uses

func (m *ProcessShard) XXX_Merge(src proto.Message)

func (*ProcessShard) XXX_Size Uses

func (m *ProcessShard) XXX_Size() int

func (*ProcessShard) XXX_Unmarshal Uses

func (m *ProcessShard) XXX_Unmarshal(b []byte) error

type RequestJobStateUpdate Uses

type RequestJobStateUpdate struct {
    JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
    ShardId              int64    `protobuf:"varint,2,opt,name=shard_id,json=shardId,proto3" json:"shard_id,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

RequestJobStateUpdate is transactionally emitted by ProcessShard when shard's state changes.

It eventually (with some throttling) causes UpdateJobState to be emitted, which updates the job state based on states of the shards.

func (*RequestJobStateUpdate) Descriptor Uses

func (*RequestJobStateUpdate) Descriptor() ([]byte, []int)

func (*RequestJobStateUpdate) GetJobId Uses

func (m *RequestJobStateUpdate) GetJobId() int64

func (*RequestJobStateUpdate) GetShardId Uses

func (m *RequestJobStateUpdate) GetShardId() int64

func (*RequestJobStateUpdate) ProtoMessage Uses

func (*RequestJobStateUpdate) ProtoMessage()

func (*RequestJobStateUpdate) Reset Uses

func (m *RequestJobStateUpdate) Reset()

func (*RequestJobStateUpdate) String Uses

func (m *RequestJobStateUpdate) String() string

func (*RequestJobStateUpdate) XXX_DiscardUnknown Uses

func (m *RequestJobStateUpdate) XXX_DiscardUnknown()

func (*RequestJobStateUpdate) XXX_Marshal Uses

func (m *RequestJobStateUpdate) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*RequestJobStateUpdate) XXX_Merge Uses

func (m *RequestJobStateUpdate) XXX_Merge(src proto.Message)

func (*RequestJobStateUpdate) XXX_Size Uses

func (m *RequestJobStateUpdate) XXX_Size() int

func (*RequestJobStateUpdate) XXX_Unmarshal Uses

func (m *RequestJobStateUpdate) XXX_Unmarshal(b []byte) error

type SplitAndLaunch Uses

type SplitAndLaunch struct {
    JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

SplitAndLaunch task splits the key range into shards and kicks off processing of each individual shard.

Enqueued transactionally when creating a new mapping job.

func (*SplitAndLaunch) Descriptor Uses

func (*SplitAndLaunch) Descriptor() ([]byte, []int)

func (*SplitAndLaunch) GetJobId Uses

func (m *SplitAndLaunch) GetJobId() int64

func (*SplitAndLaunch) ProtoMessage Uses

func (*SplitAndLaunch) ProtoMessage()

func (*SplitAndLaunch) Reset Uses

func (m *SplitAndLaunch) Reset()

func (*SplitAndLaunch) String Uses

func (m *SplitAndLaunch) String() string

func (*SplitAndLaunch) XXX_DiscardUnknown Uses

func (m *SplitAndLaunch) XXX_DiscardUnknown()

func (*SplitAndLaunch) XXX_Marshal Uses

func (m *SplitAndLaunch) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*SplitAndLaunch) XXX_Merge Uses

func (m *SplitAndLaunch) XXX_Merge(src proto.Message)

func (*SplitAndLaunch) XXX_Size Uses

func (m *SplitAndLaunch) XXX_Size() int

func (*SplitAndLaunch) XXX_Unmarshal Uses

func (m *SplitAndLaunch) XXX_Unmarshal(b []byte) error

type UpdateJobState Uses

type UpdateJobState struct {
    JobId                int64    `protobuf:"varint,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
    XXX_NoUnkeyedLiteral struct{} `json:"-"`
    XXX_unrecognized     []byte   `json:"-"`
    XXX_sizecache        int32    `json:"-"`
}

UpdateJobState is emitted after one or more shards have changed their state.

It recalculates the job's state based on state of all its shards. Throttled to 0.5 QPS.

func (*UpdateJobState) Descriptor Uses

func (*UpdateJobState) Descriptor() ([]byte, []int)

func (*UpdateJobState) GetJobId Uses

func (m *UpdateJobState) GetJobId() int64

func (*UpdateJobState) ProtoMessage Uses

func (*UpdateJobState) ProtoMessage()

func (*UpdateJobState) Reset Uses

func (m *UpdateJobState) Reset()

func (*UpdateJobState) String Uses

func (m *UpdateJobState) String() string

func (*UpdateJobState) XXX_DiscardUnknown Uses

func (m *UpdateJobState) XXX_DiscardUnknown()

func (*UpdateJobState) XXX_Marshal Uses

func (m *UpdateJobState) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*UpdateJobState) XXX_Merge Uses

func (m *UpdateJobState) XXX_Merge(src proto.Message)

func (*UpdateJobState) XXX_Size Uses

func (m *UpdateJobState) XXX_Size() int

func (*UpdateJobState) XXX_Unmarshal Uses

func (m *UpdateJobState) XXX_Unmarshal(b []byte) error

Package tasks imports 3 packages (graph) and is imported by 1 packages. Updated 2018-11-15. Refresh now. Tools for package owners.