README
¶
In progress...
Documentation
¶
Overview ¶
Package goctftime parses and stores data from ctftime.org into a Firestore database so that it can be easily indexed and queried by an Android application that displays CTF Time data.
Handlers ¶
The various handlers listen on the server for a GET request to their respective path. This triggers the execution of the handler logic. Each handler is responsible for parsing and storing some portion of ctftime.org. Handler logic is broken up into two phases.
The first phase triggers multiple goroutines to parse and store data concurrently. By default, the maximum number of goroutines running at once is 10. To change the maximum number of goroutines running at once, modify the maxRoutines variable. This concurrent phase only requests pages that we have scraped before.
The second phase operates on a single thread and checks to see if new content exists. If new content exists, it is parsed and stored in Firestore. Finally, we update the value used in phase one to delineate the range of known content.
GetLast and UpdateLast ¶
Each scraper operates concurrently as it requests known content. The definition of "known" is based on a Firestore value for the respective piece of content (teams, events, etc.). This value is retrieved and possibly updated on each scraping iteration.
Index ¶
- func CalculateHash(data interface{}) string
- func CompareCtfHash(id int, ctf Ctf, fbc FirebaseContext) (bool, error)
- func CompareTeamHash(id int, team Team, fbc FirebaseContext) (bool, error)
- func Connect(token option.ClientOption) (*firestore.Client, error)
- func DefaultHandler(_ http.ResponseWriter, _ *http.Request)
- func Fetch(url string) (*http.Response, error)
- func GenerateToken() (option.ClientOption, error)
- func GetLastCtfId(fbc FirebaseContext) int
- func GetLastTeamId(fbc FirebaseContext) int
- func ParseAndStoreCtf(ctfId int, resp *http.Response, fbc FirebaseContext) error
- func ParseAndStoreTeam(teamId int, resp *http.Response, fbc FirebaseContext) error
- func StoreCtf(ctfId int, ctf Ctf, fbc FirebaseContext) error
- func StoreTeam(teamId int, team Team, fbc FirebaseContext) error
- func UpdateCtfsHandler(w http.ResponseWriter, r *http.Request)
- func UpdateLastCtfId(fbc FirebaseContext, newCtfId int)
- func UpdateLastTeamId(fbc FirebaseContext, newPageNumber int)
- func UpdateTeamsHandler(w http.ResponseWriter, r *http.Request)
- type Ctf
- type FirebaseContext
- type Member
- type Score
- type Team
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CalculateHash ¶
func CalculateHash(data interface{}) string
CalculateHash calculates the sha256 hash of a data structure.
func CompareCtfHash ¶
func CompareCtfHash(id int, ctf Ctf, fbc FirebaseContext) (bool, error)
func CompareTeamHash ¶
func CompareTeamHash(id int, team Team, fbc FirebaseContext) (bool, error)
func Connect ¶
func Connect(token option.ClientOption) (*firestore.Client, error)
Connect connects to Firestore and returns an authenticated client that can read to/write from the database.
func DefaultHandler ¶
func DefaultHandler(_ http.ResponseWriter, _ *http.Request)
DefaultHandler handles any request not defined by an existing handler. This function simply drops spurious requests and should not be modified.
func Fetch ¶
Fetch executes a GET request to a URL. If the response status code is not 200, an error is returned specifying which URL failed and the status code of the request.
func GenerateToken ¶
func GenerateToken() (option.ClientOption, error)
GenerateToken reads an API key and returns an option to be used by Connect.
func GetLastCtfId ¶
func GetLastCtfId(fbc FirebaseContext) int
func GetLastTeamId ¶
func GetLastTeamId(fbc FirebaseContext) int
func ParseAndStoreCtf ¶
func ParseAndStoreCtf(ctfId int, resp *http.Response, fbc FirebaseContext) error
func ParseAndStoreTeam ¶
func ParseAndStoreTeam(teamId int, resp *http.Response, fbc FirebaseContext) error
func UpdateCtfsHandler ¶
func UpdateCtfsHandler(w http.ResponseWriter, r *http.Request)
func UpdateLastCtfId ¶
func UpdateLastCtfId(fbc FirebaseContext, newCtfId int)
func UpdateLastTeamId ¶
func UpdateLastTeamId(fbc FirebaseContext, newPageNumber int)
func UpdateTeamsHandler ¶
func UpdateTeamsHandler(w http.ResponseWriter, r *http.Request)
Types ¶
type FirebaseContext ¶
type FirebaseContext struct {
Ctx context.Context // context used in connection to Firestore
Fb firestore.Client // client used in connection to Firestore
}
FirebaseContext contains the necessary variables to get/set Firestore data
func NewFirebaseContext ¶
func NewFirebaseContext(ctx context.Context, token option.ClientOption) (FirebaseContext, error)
NewFirebaseContext creates a new FirebaseContext object for a Firestore request.
type Team ¶
type Team struct {
Hash string
// General
Aliases []string
Academic string
CountryCode string
Description string
Logo string // relative URL
Members []Member
Name string
NameCaseInsensitive string
Scores map[string]Score
// Social
Email string
ICQ string
Jabber string
LinkedIn string
OtherLinks []string
Skype string
Telegram string
Twitter string
Website string
}
Source Files