score

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2021 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package score provides support for scoring tests.

It is intended to be used in concert with the QuickFeed web service, which automates execution and scoring of student implemented assignments aimed to pass a given set of tests.

QuickFeed computes the score according to the formulas below, providing a percentage score for a test or a group of tests. The Weight parameter can be used to give more or less weight to some Score objects (representing different test sets). For example, if TestA has Weight 2 and TestB has Weight 1, then passing TestA gives twice the score of passing TestB.

QuickFeed computes the final score as follows: 

TotalWeight     = sum(Weight)
TaskScore[i]    = Score[i] / MaxScore[i], gives {0 < TaskScore < 1}
TaskWeight[i]   = Weight[i] / TotalWeight
TotalScore      = sum(TaskScore[i]*TaskWeight[i]), gives {0 < TotalScore < 1}

QuickFeed expects that tests are initialized in the init() method before test execution. This is done via the score.Add() method or the score.AddSub() method as shown below. Add() is used for regular tests, and AddSub() is used for subtests with individual scores. 

func init() {
    max, weight := len(fibonacciTests), 20
    score.Add(TestFibonacciMax, max, weight)
    score.Add(TestFibonacciMin, max, weight)
    for _, ft := range fibonacciTests {
        score.AddSub(TestFibonacciSubTest, subTestName("Max", ft.in), 1, 1)
    }
    for _, ft := range fibonacciTests {
        score.AddSub(TestFibonacciSubTest, subTestName("Min", ft.in), 1, 1)
    }
}

In addition, TestMain() should call score.PrintTestInfo() before running the tests to ensure that all tests are registered and will be picked up by QuickFeed. 

func TestMain(m *testing.M) {
    score.PrintTestInfo()
    os.Exit(m.Run())
}

To implement a test with scoring, you may use score.Max() to obtain a score object with Score equals to MaxScore, which may be decremented for each test failure. Note that sc.Print(t) should be called with a defer to ensure that it gets executed even if the test panics. 

func TestFibonacciMax(t *testing.T) {
    sc := score.Max()
    defer sc.Print(t)
    for _, ft := range fibonacciTests {
        out := fibonacci(ft.in)
        if out != ft.want {
            sc.Dec()
        }
    }
}

Similarly, it is also possible to use score.Min() to obtain a score object with Score equals to zero, which may be incremented for each test success. 

func TestFibonacciMin(t *testing.T) {
    sc := score.Min()
    defer sc.Print(t)
    for _, ft := range fibonacciTests {
        out := fibonacci(ft.in)
        if out == ft.want {
            sc.Inc()
        }
    }
}

Please see package score/testdata/sequence for other usage examples.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrScoreNotFound is returned if the parsed string did not contain a JSON score string.
	ErrScoreNotFound    = errors.New("Score not found in string")
	ErrScoreInterval    = errors.New("Score must be in the interval [0, MaxScore]")
	ErrMaxScore         = errors.New("MaxScore must be greater than 0")
	ErrWeight           = errors.New("Weight must be greater than 0")
	ErrEmptyTestName    = errors.New("TestName must be specified")
	ErrSecret           = errors.New("Secret field must match expected secret")
	ErrSuppressedSecret = errors.New("Error suppressed to avoid revealing secret")
)
View Source 
var File_kit_score_score_proto protoreflect.FileDescriptor

Functions

func HasPrefix 

func HasPrefix(s string) bool

HasPrefix returns true if the provided string s has a parsable prefix string.

func NewRegistry 

func NewRegistry() *registry

func NewResults 

func NewResults() *results

func Total deprecated 

func Total(scores []*Score) uint32

Total returns the total score computed over the set of scores provided. The total is a grade in the range 0-100.

Deprecated: Do not use this function; it will be removed in the future. Use Sum() instead.

Types

type BuildInfo 

type BuildInfo struct {
	ID           uint64 `protobuf:"varint,1,opt,name=ID,proto3" json:"ID,omitempty"`
	SubmissionID uint64 `protobuf:"varint,2,opt,name=SubmissionID,proto3" json:"SubmissionID,omitempty" gorm:"foreignKey:ID"`
	BuildDate    string `protobuf:"bytes,3,opt,name=BuildDate,proto3" json:"BuildDate,omitempty"`
	BuildLog     string `protobuf:"bytes,4,opt,name=BuildLog,proto3" json:"BuildLog,omitempty"`
	ExecTime     int64  `protobuf:"varint,5,opt,name=ExecTime,proto3" json:"ExecTime,omitempty"`
	// contains filtered or unexported fields
}

BuildInfo holds build data for an assignment's test execution.

func (*BuildInfo) Descriptor deprecated 

func (*BuildInfo) Descriptor() ([]byte, []int)

Deprecated: Use BuildInfo.ProtoReflect.Descriptor instead.

func (*BuildInfo) GetBuildDate 

func (x *BuildInfo) GetBuildDate() string

func (*BuildInfo) GetBuildLog 

func (x *BuildInfo) GetBuildLog() string

func (*BuildInfo) GetExecTime 

func (x *BuildInfo) GetExecTime() int64

func (*BuildInfo) GetID 

func (x *BuildInfo) GetID() uint64

func (*BuildInfo) GetSubmissionID 

func (x *BuildInfo) GetSubmissionID() uint64

func (*BuildInfo) ProtoMessage 

func (*BuildInfo) ProtoMessage()

func (*BuildInfo) ProtoReflect 

func (x *BuildInfo) ProtoReflect() protoreflect.Message

func (*BuildInfo) Reset 

func (x *BuildInfo) Reset()

func (*BuildInfo) String 

func (x *BuildInfo) String() string

type GradingScheme 

type GradingScheme struct {
	ID uint64 `json:"id"`

	Name        string
	GradePoints []uint32
	GradeNames  []string
}

GradingScheme for an assignment

func (*GradingScheme) Grade 

func (g *GradingScheme) Grade(points uint32) string

Grade computes the grade for the given points. The points must be in the range [0,100].

type Results 

type Results struct {
	BuildInfo *BuildInfo // build info for tests
	Scores    []*Score   // list of scores for different tests
	Errors    []error    // errors encountered during test execution
}

Results contains the score objects, build info, and errors.

func ExtractResults 

func ExtractResults(out, secret string, execTime time.Duration) *Results

ExtractResults returns the results from a test execution extracted from the given out string.

func (*Results) Sum 

func (r *Results) Sum() uint32

Sum returns the total score computed over the set of recorded scores. The total is a grade in the range 0-100. This method must only be called after Validate has returned nil.

func (*Results) Validate 

func (r *Results) Validate(secret string) error

Validate returns an error if one of the recorded score objects are invalid. Otherwise, nil is returned.

type Score 

type Score struct {
	ID           uint64 `protobuf:"varint,1,opt,name=ID,proto3" json:"ID,omitempty"`
	SubmissionID uint64 `protobuf:"varint,2,opt,name=SubmissionID,proto3" json:"SubmissionID,omitempty" gorm:"foreignKey:ID"`
	Secret       string `protobuf:"bytes,3,opt,name=Secret,proto3" json:"Secret,omitempty" gorm:"-"`  // the unique identifier for a scoring session
	TestName     string `protobuf:"bytes,4,opt,name=TestName,proto3" json:"TestName,omitempty"`       // name of the test
	Score        int32  `protobuf:"varint,5,opt,name=Score,proto3" json:"Score,omitempty"`            // the score obtained
	MaxScore     int32  `protobuf:"varint,6,opt,name=MaxScore,proto3" json:"MaxScore,omitempty"`      // max score possible to get on this specific test
	Weight       int32  `protobuf:"varint,7,opt,name=Weight,proto3" json:"Weight,omitempty"`          // the weight of this test; used to compute final grade
	TestDetails  string `protobuf:"bytes,8,opt,name=TestDetails,proto3" json:"TestDetails,omitempty"` // if populated, the frontend may display additional details (TODO(meling) adapt to output from go test -json)
	// contains filtered or unexported fields
}

Score give the score for a single test named TestName.

func NewScore deprecated 

func NewScore(t *testing.T, max, weight int) *Score

NewScore returns a new Score object with the given max and weight. The Score is initially 0, and Inc() and IncBy() can be called on the returned Score object, for each test that passes. The TestName is initialized to the name of the provided t.Name(). This function also prints a JSON representation of the Score object to ensure that the test is recorded by Quickfeed.

Deprecated: Do not use this function; it will be removed in the future. Use Add() in conjunction with score.Min() instead for regular tests, and AddSub() in conjunction with score.MinByName() instead for subtests.

func NewScoreMax deprecated 

func NewScoreMax(t *testing.T, max, weight int) *Score

NewScoreMax returns a new Score object with the given max and weight. The Score is initially set to max, and Dec() and DecBy() can be called on the returned Score object, for each test that fails. The TestName is initialized as the name of the provided t.Name(). This function also prints a JSON representation of the Score object to ensure that the test is recorded by Quickfeed.

Deprecated: Do not use this function; it will be removed in the future. Use Add() in conjunction with score.Max() instead for regular tests, and AddSub() in conjunction with score.MaxByName() instead for subtests.

func Parse 

func Parse(s, secret string) (*Score, error)

Parse returns a score object for the provided JSON string s which contains secret.

func (*Score) Dec 

func (s *Score) Dec()

Dec decrements score if score is greater than zero.

func (*Score) DecBy 

func (s *Score) DecBy(n int)

DecBy decrements score n times or until Score equals zero.

func (*Score) Descriptor deprecated 

func (*Score) Descriptor() ([]byte, []int)

Deprecated: Use Score.ProtoReflect.Descriptor instead.

func (*Score) Equal 

func (s *Score) Equal(other *Score) bool

Equal returns true if s equals other. Ignores the Secret field.

func (*Score) Fail 

func (s *Score) Fail()

Fail sets Score to zero.

func (*Score) GetID 

func (x *Score) GetID() uint64

func (*Score) GetMaxScore 

func (x *Score) GetMaxScore() int32

func (*Score) GetScore 

func (x *Score) GetScore() int32

func (*Score) GetSecret 

func (x *Score) GetSecret() string

func (*Score) GetSubmissionID 

func (x *Score) GetSubmissionID() uint64

func (*Score) GetTestDetails 

func (x *Score) GetTestDetails() string

func (*Score) GetTestName 

func (x *Score) GetTestName() string

func (*Score) GetWeight 

func (x *Score) GetWeight() int32

func (*Score) Inc 

func (s *Score) Inc()

Inc increments score if score is less than MaxScore.

func (*Score) IncBy 

func (s *Score) IncBy(n int)

IncBy increments score n times or until score equals MaxScore.

func (*Score) IsValid 

func (sc *Score) IsValid(secret string) error

IsValid returns an error if the score object is invalid. Otherwise, nil is returned. If the given secret matches the score's secret value, the Secret field is redacted with the empty string "".

func (*Score) Normalize 

func (s *Score) Normalize(maxScore int)

Normalize the score to the given maxScore.

func (*Score) PanicHandler 

func (s *Score) PanicHandler(t *testing.T, msg ...string)

PanicHandler recovers from a panicking test, resets the score to zero and emits an error message. This is only needed when using a single score object for multiple subtests each of which may panic, which would prevent the deferred Print() call from executing its recovery handler.

This must be called as a deferred function from within a subtest, that is within a t.Run() function: 

defer s.PanicHandler(t)

The msg parameter is optional, and will be printed in case of a panic.

func (*Score) Print 

func (s *Score) Print(t *testing.T, msg ...string)

Print prints a JSON representation of the score that can be picked up by QuickFeed. To ensure that panic message and stack trace is printed, this method must be called via defer. If a test panics, the score will be set to zero, and a panic message will be emitted. Note that, if subtests are used, each subtest must defer call the PanicHandler method to ensure that panics are caught and handled appropriately. The msg parameter is optional, and will be printed in case of a panic.

func (*Score) ProtoMessage 

func (*Score) ProtoMessage()

func (*Score) ProtoReflect 

func (x *Score) ProtoReflect() protoreflect.Message

func (*Score) RelativeScore 

func (s *Score) RelativeScore() string

RelativeScore returns a string with the following format: "TestName: score = x/y = s".

func (*Score) Reset 

func (x *Score) Reset()

func (*Score) String 

func (x *Score) String() string

func (*Score) WriteJSON deprecated 

func (s *Score) WriteJSON(w io.Writer)

WriteJSON writes the JSON representation of s to w.

Deprecated: Do not use this function; it will be removed in the future. Use Print() instead to replace both WriteString() and WriteJSON().

func (*Score) WriteString deprecated 

func (s *Score) WriteString(w io.Writer)

WriteString writes the string representation of s to w.

Deprecated: Do not use this function; it will be removed in the future. Use Print() instead to replace both WriteString() and WriteJSON().

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.