README
¶
Search Engine in Go
This is experimental project baked with go-colly and blevesearch to build search engine in Go.
Prerequisite
To use package in blevex (blevesearch's extension), it is required to install C dependency as following steps.
This is verified against Debian GNU/Linux 8.11 (jessie)
$ sudo apt-get install libleveldb-dev libstemmer-dev libicu-dev build-essential
$ cd ~
$ git clone https://github.com/blevesearch/cld2.git
$ cd cld2/internal/
$ ./compile_libs.sh
$ sudo cp *.so /usr/local/lib
Minimum requirement for ICU analyzer and tokenizer, libicu-dev only is required. Thus, go ahead and simply install it by issuing this command in prompt
$ sudo apt-get install libicu-dev
Installation
Use git clone or go get to download project to your go workspace in $GOPATH then run dep ensure to initialise project.
$ go get github.com/atthakorn/search-engine
$ cd $GOPATH/src/github.com/atthakorn/search-engine
$ dep ensure
Config
Here is the list of parameter you can find in .config.yml
# sites to crawl
entryPoint:
- https://en.wikipedia.org/wiki/Main_Page
# max depth for crawler to follow
maxDepth: 2
# max worker
parallelism: 1
# random delay (second)
delay: 1
# path to store scraped data from crawler
dataPath: "data/crawl"
# path to store indexed data
indexPath: "data/index"
# http address, 0.0.0.0:8080 or :8080, it means listening all ipv4 address in local machine
httpAddress: ":8080"
Crawling & Indexing
To crawling websites, just cd to project root and run
$ go run cmd/crawl/main.go
Once, crawl complete, the scraped data will be kept at ./data/crawl/*.json
Now you can index crawler's scraped data by issuing following command
$ go run cmd/index/main.go
The data will be indexed by boltdb, the file will be located at ./data/index/*
Start Server
Search Engine comes with simple search application , you can start http server by following command
$ go run cmd/http/main.go
and you can open application via browser at http://localhost:8080
License
This project is licensed under MIT License
Directories