scraper

package
v0.0.0-...-a54d9e5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 22, 2017 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

    handlers just turn the logic of slicing up HTML into little bits™

    Package scraper is a utility for extracting data from the course data html page

		Structure - <tr> come in pairs as sibling elements
		<table class="datadisplaytable">
			<tbody>
				<tr></tr> // Metadata like CRN, Course name, and Section
				<tr></tr>	// Instructor restrictions, Prereqs, Attributes etc...
				<tr></tr> // Metadata again
				<tr></tr> // All that other data
				...
			</tbody>
		</table>

    Models for rethinkDB

Index

Constants

This section is empty.

Variables

View Source
var DB r.Term
View Source
var Session *r.Session

Functions

func Commit

func Commit(reg *CourseData)

func Drop

func Drop() error

CLI drop function

func Index

func Index() (int, error)

Go ahead and index the registry of data

func InitDB

func InitDB() error

not running this in init to prevent slow startup

func WriteCourses

func WriteCourses(courses map[string]*Course) error

writes courses to Rethink

func WriteDepartments

func WriteDepartments(departments map[string]*Department) error

writes Departments to Rethink

func WriteInstructors

func WriteInstructors(instructors map[string]*Instructor) error

writes instructors to Rethink

func WriteSections

func WriteSections(ci map[string]*Section) error

writes sections to Rethink

Types

type CatalogData

type CatalogData struct {
	Course      string //CSC1052
	Credits     string //3.00
	Description string //Great class
}

type Course

type Course struct {
	ID          string   `gorethink:"id,omitempty"` //Department+Class (should be unique)
	Department  string   `gorethink:"department"`
	Name        string   `gorethink:"name"`
	Credits     string   `gorethink:"credits"`
	Description string   `gorethink:"description"`
	Class       string   `gorethink:"class"`
	Sections    []string `gorethink:"sections"`
}

1052 is taught by plenty of teachers

type CourseData

type CourseData struct {
	Instructors map[string]*Instructor
	Courses     map[string]*Course
	Departments map[string]*Department
	CI          map[string]*Section
}
var (
	Registrar *CourseData
)

type Department

type Department struct {
	Name    string   `gorethink:"name"`
	Courses []string `gorethink:"courses"`
}

Departments can have many teachers, but teachers don't always belong to a department

type Instructor

type Instructor struct {
	Name     string   `gorethink:"name"`
	Id       string   `gorethink:"id"`
	Sections []string `gorethink:"sections"`
}

instructor teaches many sections

type Schedule

type Schedule struct {
	Days      string `gorethink:"days"`      //MWF
	StartTime string `gorethink:"startTime"` //15:15
	EndTime   string `gorethink:"endTime"`   //16:30
	Location  string `gorethink:"location"`  //Mendel G86
}

days, times, and location of a class

type Section

type Section struct {
	CRN          string        `gorethink:"crn"`          //33919
	Instructors  []*Instructor `gorethink:"-"`            // -- Used for indexing
	Instructor   []string      `gorethink:"instructors"`  //[Daniel Joyce]
	Attributes   []string      `gorethink:"attributes"`   //Core Science
	Restrictions string        `gorethink:"restrictions"` //University Alliance
	Prereqs      []string      `gorethink:"prereqs"`      //CSC-1051
	Name         string        `gorethink:"name"`         //Algorithms & Data Structures
	Comment      string        `gorethink:"comment"`      //Learn all about Linked Lists!
	Section      string        `gorethink:"section"`      //001
	Class        string        `gorethink:"class"`        //1052
	Credits      string        `gorethink:"credits"`      //3.00
	Enrolled     int           `gorethink:"enrolled"`     //07
	Size         int           `gorethink:"size"`         //30
	Department   string        `gorethink:"department"`   //CSC
	Description  string        `gorethink:"description"`  //Redundant description data, but we're only dealing w 4MB
	Schedule     []*Schedule   `gorethink:"schedule"`     //[{10:30, 11:45, MWF, Mendel G86}]
}

might as well model out all these different tables

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL