score

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 25, 2024 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package score provides support for scoring tests.

It is intended to be used in concert with the QuickFeed web service, which automates execution and scoring of student implemented assignments aimed to pass a given set of tests.

QuickFeed computes the score according to the formulas below, providing a percentage score for a test or a group of tests. The Weight parameter can be used to give more or less weight to some Score objects (representing different test sets). For example, if TestA has Weight 2 and TestB has Weight 1, then passing TestA gives twice the score of passing TestB.

QuickFeed computes the final score as follows: 

TotalWeight     = sum(Weight)
TaskScore[i]    = Score[i] / MaxScore[i], gives {0 < TaskScore < 1}
TaskWeight[i]   = Weight[i] / TotalWeight
TotalScore      = sum(TaskScore[i]*TaskWeight[i]), gives {0 < TotalScore < 1}

QuickFeed expects that tests are initialized in the init() method before test execution. This is done via the score.Add() method or the score.AddSub() method as shown below. Add() is used for regular tests, and AddSub() is used for subtests with individual scores. 

func init() {
    max, weight := len(fibonacciTests), 20
    score.Add(TestFibonacciMax, max, weight)
    score.Add(TestFibonacciMin, max, weight)
    for _, ft := range fibonacciTests {
        score.AddSub(TestFibonacciSubTest, subTestName("Max", ft.in), 1, 1)
    }
    for _, ft := range fibonacciTests {
        score.AddSub(TestFibonacciSubTest, subTestName("Min", ft.in), 1, 1)
    }
}

Tests can also be added with the score.AddWithTask() and score.AddSubWithTask() methods. Adding a task this way lets you associate any given test with a task name, as shown below. 

  func init() {
     score.AddWithTask(TestHelloWorld, "hello_world", 10, 5)
     score.AddWithTask(TestFunctions, "functions", 10, 5)
		for _, ft := range functionTests {
     	score.AddSubWithTask(TestFunctionSubTest, subTestName("functions", ft.in), "functions" 1, 1)
     }
  }

In addition, TestMain() should call score.PrintTestInfo() before running the tests to ensure that all tests are registered and will be picked up by QuickFeed. 

func TestMain(m *testing.M) {
    score.PrintTestInfo()
    os.Exit(m.Run())
}

To implement a test with scoring, you may use score.Max() to obtain a score object with Score equals to MaxScore, which may be decremented for each test failure. Note that sc.Print(t) should be called with a defer to ensure that it gets executed even if the test panics. 

func TestFibonacciMax(t *testing.T) {
    sc := score.Max()
    defer sc.Print(t)
    for _, ft := range fibonacciTests {
        out := fibonacci(ft.in)
        if out != ft.want {
            sc.Dec()
        }
    }
}

Similarly, it is also possible to use score.Min() to obtain a score object with Score equals to zero, which may be incremented for each test success. 

func TestFibonacciMin(t *testing.T) {
    sc := score.Min()
    defer sc.Print(t)
    for _, ft := range fibonacciTests {
        out := fibonacci(ft.in)
        if out == ft.want {
            sc.Inc()
        }
    }
}

Please see package score/testdata/sequence for other usage examples.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrScoreNotFound is returned if the parsed string did not contain a JSON score string.
	ErrScoreNotFound    = errors.New("score not found in string")
	ErrScoreInterval    = errors.New("score must be in the interval [0, MaxScore]")
	ErrMaxScore         = errors.New("max score must be greater than 0")
	ErrWeight           = errors.New("weight must be greater than 0")
	ErrEmptyTestName    = errors.New("test name must be specified")
	ErrSecret           = errors.New("secret field must match expected secret")
	ErrSuppressedSecret = errors.New("error suppressed to avoid revealing secret")
)
View Source 
var (
	ErrDuplicateScoreTest = errors.New("duplicate score test")
	ErrUnauthorizedLookup = errors.New("unauthorized lookup")
	ErrUnknownScoreTest   = errors.New("unknown score test")
)
View Source 
var File_kit_score_score_proto protoreflect.FileDescriptor

Functions

func HasPrefix 

func HasPrefix(s string) bool

HasPrefix returns true if the provided string s has a parsable prefix string.

func NewRegistry 

func NewRegistry() *registry

Types

type BuildInfo 

type BuildInfo struct {
	ID             uint64                 `protobuf:"varint,1,opt,name=ID,proto3" json:"ID,omitempty"`
	SubmissionID   uint64                 `protobuf:"varint,2,opt,name=SubmissionID,proto3" json:"SubmissionID,omitempty" gorm:"foreignKey:ID"`
	BuildLog       string                 `protobuf:"bytes,3,opt,name=BuildLog,proto3" json:"BuildLog,omitempty"`
	ExecTime       int64                  `protobuf:"varint,4,opt,name=ExecTime,proto3" json:"ExecTime,omitempty"`
	BuildDate      *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=BuildDate,proto3" json:"BuildDate,omitempty" gorm:"serializer:timestamp;type:datetime"`
	SubmissionDate *timestamppb.Timestamp `protobuf:"bytes,6,opt,name=SubmissionDate,proto3" json:"SubmissionDate,omitempty" gorm:"serializer:timestamp;type:datetime"`
	// contains filtered or unexported fields
}

BuildInfo holds build data for an assignment's test execution.

func (*BuildInfo) Descriptor deprecated 

func (*BuildInfo) Descriptor() ([]byte, []int)

Deprecated: Use BuildInfo.ProtoReflect.Descriptor instead.

func (*BuildInfo) GetBuildDate 

func (x *BuildInfo) GetBuildDate() *timestamppb.Timestamp

func (*BuildInfo) GetBuildLog 

func (x *BuildInfo) GetBuildLog() string

func (*BuildInfo) GetExecTime 

func (x *BuildInfo) GetExecTime() int64

func (*BuildInfo) GetID 

func (x *BuildInfo) GetID() uint64

func (*BuildInfo) GetSubmissionDate added in v0.9.0 

func (x *BuildInfo) GetSubmissionDate() *timestamppb.Timestamp

func (*BuildInfo) GetSubmissionID 

func (x *BuildInfo) GetSubmissionID() uint64

func (*BuildInfo) ProtoMessage 

func (*BuildInfo) ProtoMessage()

func (*BuildInfo) ProtoReflect 

func (x *BuildInfo) ProtoReflect() protoreflect.Message

func (*BuildInfo) Reset 

func (x *BuildInfo) Reset()

func (*BuildInfo) String 

func (x *BuildInfo) String() string

type GradingScheme 

type GradingScheme struct {
	ID uint64 `json:"id"`

	Name        string
	GradePoints []uint32
	GradeNames  []string
}

GradingScheme for an assignment

func (*GradingScheme) Grade 

func (g *GradingScheme) Grade(points uint32) string

Grade computes the grade for the given points. The points must be in the range [0,100].

type Results 

type Results struct {
	BuildInfo *BuildInfo // build info for tests
	Scores    []*Score   // list of scores for different tests
	// contains filtered or unexported fields
}

Results contains the score objects, build info, and errors.

func ExtractResults 

func ExtractResults(out, secret string, execTime time.Duration) (*Results, error)

ExtractResults returns the results from a test execution extracted from the given out string.

func (*Results) MarkdownComment 

func (r *Results) MarkdownComment(taskLocalName string, scoreLimit uint32) string

MarkdownComment returns a markdown formatted feedback comment of the test results. Only the test scores associated with the supplied task are included in the table. An example table is shown below. 

 ## Test results from latest push

	| Test Name | Score | Weight | % of Total |
	| :-------- | ----: | -----: | ---------: |
 | Test 1    |   2/4 |      1 |       6.3% |
 | Test 2    |   1/4 |      2 |       6.3% |
 | Test 3    |   3/4 |      5 |      46.9% |
 | Total     |       |        |      59.5% |

	Reviewers are assigned once the total score reaches 80%.

func (*Results) Sum 

func (r *Results) Sum() uint32

Sum returns the total score the of recorded scores. The total is a grade in the range 0-100. This method must only be called after Validate has returned nil.

func (*Results) TaskSum 

func (r *Results) TaskSum(taskName string) uint32

TaskSum returns the total score the recorded scores for the given task. The total is a grade in the range 0-100. This method must only be called after Validate has returned nil.

type Score 

type Score struct {
	ID           uint64 `protobuf:"varint,1,opt,name=ID,proto3" json:"ID,omitempty"`
	SubmissionID uint64 `protobuf:"varint,2,opt,name=SubmissionID,proto3" json:"SubmissionID,omitempty" gorm:"foreignKey:ID"`
	Secret       string `protobuf:"bytes,3,opt,name=Secret,proto3" json:"Secret,omitempty" gorm:"-"`  // the unique identifier for a scoring session
	TestName     string `protobuf:"bytes,4,opt,name=TestName,proto3" json:"TestName,omitempty"`       // name of the test
	TaskName     string `protobuf:"bytes,5,opt,name=TaskName,proto3" json:"TaskName,omitempty"`       // name of task this score belongs to
	Score        int32  `protobuf:"varint,6,opt,name=Score,proto3" json:"Score,omitempty"`            // the score obtained
	MaxScore     int32  `protobuf:"varint,7,opt,name=MaxScore,proto3" json:"MaxScore,omitempty"`      // max score possible to get on this specific test
	Weight       int32  `protobuf:"varint,8,opt,name=Weight,proto3" json:"Weight,omitempty"`          // the weight of this test; used to compute final grade
	TestDetails  string `protobuf:"bytes,9,opt,name=TestDetails,proto3" json:"TestDetails,omitempty"` // if populated, the frontend may display these details
	// contains filtered or unexported fields
}

Score give the score for a single test named TestName.

func (*Score) Dec 

func (s *Score) Dec()

Dec decrements score if score is greater than zero.

func (*Score) DecBy 

func (s *Score) DecBy(n int)

DecBy decrements score n times or until Score equals zero.

func (*Score) Descriptor deprecated 

func (*Score) Descriptor() ([]byte, []int)

Deprecated: Use Score.ProtoReflect.Descriptor instead.

func (*Score) Equal 

func (s *Score) Equal(other *Score) bool

Equal returns true if s equals other. Ignores the Secret field.

func (*Score) Fail 

func (s *Score) Fail()

Fail sets Score to zero.

func (*Score) GetID 

func (x *Score) GetID() uint64

func (*Score) GetMaxScore 

func (x *Score) GetMaxScore() int32

func (*Score) GetScore 

func (x *Score) GetScore() int32

func (*Score) GetSecret 

func (x *Score) GetSecret() string

func (*Score) GetSubmissionID 

func (x *Score) GetSubmissionID() uint64

func (*Score) GetTaskName 

func (x *Score) GetTaskName() string

func (*Score) GetTestDetails 

func (x *Score) GetTestDetails() string

func (*Score) GetTestName 

func (x *Score) GetTestName() string

func (*Score) GetWeight 

func (x *Score) GetWeight() int32

func (*Score) Inc 

func (s *Score) Inc()

Inc increments score if score is less than MaxScore.

func (*Score) IncBy 

func (s *Score) IncBy(n int)

IncBy increments score n times or until score equals MaxScore.

func (*Score) Normalize 

func (s *Score) Normalize(maxScore int)

Normalize the score to the given maxScore.

func (*Score) PanicHandler 

func (s *Score) PanicHandler(t *testing.T, msg ...string)

PanicHandler recovers from a panicking test, resets the score to zero and emits an error message. This is only needed when using a single score object for multiple subtests each of which may panic, which would prevent the deferred Print() call from executing its recovery handler.

This must be called as a deferred function from within a subtest, that is within a t.Run() function: 

defer s.PanicHandler(t)

The msg parameter is optional, and will be printed in case of a panic.

func (*Score) Print 

func (s *Score) Print(t *testing.T, msg ...string)

Print prints a JSON representation of the score that can be picked up by QuickFeed. To ensure that panic message and stack trace is printed, this method must be called via defer. If a test panics, the score will be set to zero, and a panic message will be emitted. Note that, if subtests are used, each subtest must defer call the PanicHandler method to ensure that panics are caught and handled appropriately. The msg parameter is optional, and will be printed in case of a panic.

func (*Score) ProtoMessage 

func (*Score) ProtoMessage()

func (*Score) ProtoReflect 

func (x *Score) ProtoReflect() protoreflect.Message

func (*Score) RelativeScore 

func (s *Score) RelativeScore() string

RelativeScore returns a string with the following format: "TestName: score = x/y = s".

func (*Score) Reset 

func (x *Score) Reset()

func (*Score) String 

func (x *Score) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.