README
¶
esdump
Stream docs from Elasticsearch to stdout for ad-hoc data mangling using the Scroll API. Just like solrdump, but for elasticsearch.
Libraries can use both GET and POST requests to issue scroll requests.
- elasticsearch-py uses POST
- esapi uses GET
This tool uses HTTP GET only, and does not clear scrolls (which would probably use DELETE) so this tools work with read-only servers, that only allow GET.
Install
$ go install github.com/miku/esdump/cmd/esdump@latest
Or via a release.
Usage
esdump uses the elasticsearch scroll API to stream documents to stdout. First
written to extract samples from https://search.fatcat.wiki (a scholarly
communications preservation and discovery project).
$ esdump -s https://search.fatcat.wiki -i fatcat_release -q 'web archiving'
Usage of esdump:
-i string
index name (default "fatcat_release")
-ids string
a path to a file with one id per line to fetch
-l int
limit number of documents fetched, zero means no limit
-mq string
path to file, one lucene query per line
-q string
lucene syntax query to run, example: 'affiliation:"alberta"' (default "*")
-s string
elasticsearch server (default "https://search.fatcat.wiki")
-scroll string
context timeout (default "5m")
-size int
batch size (default 1000)
-v show version
-verbose
be verbose
Performance data points
925636 docs in 4m47.460217252s (3220 docs/s)
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BasicScroller ¶
type BasicScroller struct {
Server string // https://search.elastic.io
Index string
Query string // query_string query, will be url escaped, so ok to write: '(f:value) OR (g:"hi there")'
Scroll string // context timeout, e.g. "5m"
Size int // number of docs per request
MaxRetries int // Retry of stranger things, like "unexpected EOF"
// contains filtered or unexported fields
}
BasicScroller abstracts iteration over larger result sets via https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-scroll. Not using the official library since esapi (may) use POST (and other verbs), whereas some endpoints disallow anything but GET requests.
func (*BasicScroller) Bytes ¶
func (s *BasicScroller) Bytes() []byte
Bytes returns the current response body.
func (*BasicScroller) Elapsed ¶
func (s *BasicScroller) Elapsed() time.Duration
Elapsed returns the elasped time.
func (*BasicScroller) Next ¶
func (s *BasicScroller) Next() bool
Next fetches the next batch, which is accessible via Bytes or String methods. Returns true, if successful, false if stream ended or an error occured. The error can be accessed separately.
func (*BasicScroller) String ¶
func (s *BasicScroller) String() string
String returns current request body as string.
func (*BasicScroller) Total ¶
func (s *BasicScroller) Total() int
Total returns total documents retrieved.
type MassQuery ¶
type MassQuery struct {
Server string // https://search.elastic.io
Index string
Queries []string // query_string queries
Size int
Writer io.Writer
Err error
}
MassQuery runs many requests in parallel. Does no pagination. Useful for the moment to get the result set size for a given query. TODO: This is just a special case to request many URL in parallel and combining the results. TODO: Look into "multisearch", https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html.
type SearchResponse ¶
type SearchResponse struct {
Hits struct {
Hits []struct {
Id string `json:"_id"`
Index string `json:"_index"`
Score float64 `json:"_score"`
Source json.RawMessage `json:"_source"`
Type string `json:"_type"`
} `json:"hits"`
MaxScore float64 `json:"max_score"`
TotalValue interface{} `json:"total"`
} `json:"hits"`
ScrollID string `json:"_scroll_id"`
Shards struct {
Failed int64 `json:"failed"`
Skipped int64 `json:"skipped"`
Successful int64 `json:"successful"`
Total int64 `json:"total"`
} `json:"_shards"`
TimedOut bool `json:"timed_out"`
Took int64 `json:"took"`
}
SearchResponse is an basic search response with an unparsed source field.
func (*SearchResponse) Total ¶ added in v0.1.9
func (s *SearchResponse) Total() int64
Total handles elasticsearch v6/v7 api changes.
Source Files
Directories