Documentation
¶
Overview ¶
Copyright 2019 Google Inc. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // NumberConcurrentListTasks is used to both determine listing memory constraints, and set the MaxOutstandingMessages for the PubSub List subscription. NumberConcurrentListTasks = flag.Int("number-concurrent-list-tasks", 4, "The maximum number of list tasks the agent will process at any given time.") )
Functions ¶
This section is empty.
Types ¶
type DepthFirstListHandler ¶
type DepthFirstListHandler struct {
// contains filtered or unexported fields
}
DepthFirstListHandler is responsible for handling depth-first list tasks. For each list task, the handler produces a single output file, the list file. In the list file, there is a list of all the files in the directories listed. Unexplored directories are written to the end of the list file.
func NewDepthFirstListHandler ¶
func NewDepthFirstListHandler(storageClient *storage.Client, st *stats.Tracker) *DepthFirstListHandler
NewDepthFirstListHandler returns a new DepthFirstListHandler.
func (*DepthFirstListHandler) Do ¶
func (h *DepthFirstListHandler) Do(ctx context.Context, taskReqMsg *taskpb.TaskReqMsg, reqStart time.Time) *taskpb.TaskRespMsg
type DirectoryInfoStore ¶
type DirectoryInfoStore struct {
// contains filtered or unexported fields
}
DirectoryInfoStore stores a sorted list of DirectoryInfos and keeps track of the approximate number of bytes used to store them.
func NewDirectoryInfoStore ¶
func NewDirectoryInfoStore() *DirectoryInfoStore
func (*DirectoryInfoStore) Add ¶
func (s *DirectoryInfoStore) Add(dirInfo listpb.DirectoryInfo) error
Add adds the given dirInfo to the DirectoryInfoStore. If the given dirInfo is already stored in the DirectoryInfoStore or an invalid dirInfo is passed (dirInfo.Path is not set), Add returns an error.
func (*DirectoryInfoStore) DirectoryInfos ¶
func (s *DirectoryInfoStore) DirectoryInfos() []listpb.DirectoryInfo
DirectoryInfos returns a sorted list of the DirectoryInfos stored in the DirectoryInfoStore.
func (*DirectoryInfoStore) Len ¶
func (s *DirectoryInfoStore) Len() int
Len returns the number of directories stored in the DirectoryInfoStore.
func (*DirectoryInfoStore) RemoveFirst ¶
func (s *DirectoryInfoStore) RemoveFirst() *listpb.DirectoryInfo
RemoveFirst removes the first DirectoryInfo from the DirectoryInfoStore as determined by case sensitive alphabetical order. If there are no DirectoryInfos in the store, nil is returned.
func (*DirectoryInfoStore) Size ¶
func (s *DirectoryInfoStore) Size() int
Size returns an approximation of the bytes currently used by the DirectoryInfoStore.
type ListFileEntry ¶
func ParseListFileLine ¶
func ParseListFileLine(line string) (*ListFileEntry, error)
func (ListFileEntry) String ¶
func (l ListFileEntry) String() string
type ListHandlerV3 ¶
type ListHandlerV3 struct {
// contains filtered or unexported fields
}
ListHandlerV3 is responsible for handling job run version 3 depth-first list tasks. When this handler processes a list task, it produces two files. The first file is a list file that contains all the contents of a listed directory (both files and child directories are listed). The second file contains a list of all the unexplored directories (directories that were discovered but not yet listed). This style of listing allows us to easily detect the deletion of directories.
func NewListHandlerV3 ¶
func NewListHandlerV3(storageClient *storage.Client, st *stats.Tracker) *ListHandlerV3
NewListHandlerV3 returns a new ListHandlerV3.
func (*ListHandlerV3) Do ¶
func (h *ListHandlerV3) Do(ctx context.Context, taskReqMsg *taskpb.TaskReqMsg) *taskpb.TaskRespMsg
Source Files