templates

package
v3.2.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 25, 2024 License: MIT Imports: 71 Imported by: 3

Documentation

Overview

Package templates contains the parser for a template for the engine.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. DO NOT EDIT: this file is automatically generated by docgen

Package templates nolint //do not lint as examples with no usage

Index

Constants

View Source
const (
	Unsigned   = "unsigned"
	PDVerifier = "projectdiscovery/nuclei-templates"
)
View Source
const (
	SyntaxWarningStats           = "syntax-warnings"
	SyntaxErrorStats             = "syntax-errors"
	RuntimeWarningsStats         = "runtime-warnings"
	SkippedCodeTmplTamperedStats = "unsigned-warnings"
	ExcludedHeadlessTmplStats    = "headless-flag-missing-warnings"
	TemplatesExcludedStats       = "templates-executed"
	ExcludedCodeTmplStats        = "code-flag-missing-warnings"
	ExludedDastTmplStats         = "fuzz-flag-missing-warnings"
	SkippedUnsignedStats         = "skipped-unsigned-stats" // tracks loading of unsigned templates
	SkippedRequestSignatureStats = "skipped-request-signature-stats"
)

Variables

View Source
var (
	ErrCreateTemplateExecutor          = errors.New("cannot create template executer")
	ErrIncompatibleWithOfflineMatching = errors.New("template can't be used for offline matching")
	// track how many templates are verfied and by which signer
	SignatureStats = map[string]*atomic.Uint64{}
)
View Source
var (
	Colorizer         aurora.Aurora
	SeverityColorizer func(severity.Severity) string
)
View Source
var (
	ErrMandatoryFieldMissingFmt = errorutil.NewWithFmt("mandatory '%s' field is missing")
	ErrInvalidField             = errorutil.NewWithFmt("invalid field format for '%s' (allowed format is %s)")
	ErrWarningFieldMissing      = errorutil.NewWithFmt("field '%s' is missing")
	ErrCouldNotLoadTemplate     = errorutil.NewWithFmt("Could not load template %s: %s")
	ErrLoadedWithWarnings       = errorutil.NewWithFmt("Loaded template %s: with syntax warning : %s")
)
View Source
var (
	TemplateDoc                   encoder.Doc
	MODELInfoDoc                  encoder.Doc
	STRINGSLICEStringSliceDoc     encoder.Doc
	STRINGSLICERawStringSliceDoc  encoder.Doc
	SEVERITYHolderDoc             encoder.Doc
	MODELClassificationDoc        encoder.Doc
	HTTPRequestDoc                encoder.Doc
	GENERATORSAttackTypeHolderDoc encoder.Doc
	HTTPMethodTypeHolderDoc       encoder.Doc
	FUZZRuleDoc                   encoder.Doc
	SliceOrMapSliceDoc            encoder.Doc
	SignatureTypeHolderDoc        encoder.Doc
	MATCHERSMatcherDoc            encoder.Doc
	MatcherTypeHolderDoc          encoder.Doc
	DNSRequestDoc                 encoder.Doc
	DNSRequestTypeHolderDoc       encoder.Doc
	FILERequestDoc                encoder.Doc
	NETWORKRequestDoc             encoder.Doc
	NETWORKInputDoc               encoder.Doc
	NetworkInputTypeHolderDoc     encoder.Doc
	HEADLESSRequestDoc            encoder.Doc
	ENGINEActionDoc               encoder.Doc
	ActionTypeHolderDoc           encoder.Doc
	USERAGENTUserAgentHolderDoc   encoder.Doc
	SSLRequestDoc                 encoder.Doc
	WEBSOCKETRequestDoc           encoder.Doc
	WEBSOCKETInputDoc             encoder.Doc
	WHOISRequestDoc               encoder.Doc
	CODERequestDoc                encoder.Doc
	JAVASCRIPTRequestDoc          encoder.Doc
	HTTPSignatureTypeHolderDoc    encoder.Doc
	VARIABLESVariableDoc          encoder.Doc
)
View Source
var ErrExcluded = errors.New("the template was excluded")

ErrExcluded is returned for excluded templates

View Source
var (
	ErrNotATemplate = errorutil.NewWithTag("signer", "given filePath is not a template")
)
View Source
var (
	ReTemplateID = regexp.MustCompile(`^([a-zA-Z0-9]+[-_])*[a-zA-Z0-9]+$`)
)

Functions

func Cluster

func Cluster(list []*Template) [][]*Template

Cluster clusters a list of templates into a lesser number if possible based on the similarity between the sent requests.

If the attributes match, multiple requests can be clustered into a single request which saves time and network resources during execution.

The clusterer goes through all the templates, looking for templates with a single HTTP/DNS/TLS request to an endpoint (multiple requests aren't clustered as of now).

All the templates are iterated and any templates with request that is identical to the first individual request is compared for equality. The equality check is performed as described below -

Cases where clustering is not performed (request is considered different)

  • If request contains payloads,raw,body,unsafe,req-condition,name attributes
  • If request methods,max-redirects,disable-cookie,redirects are not equal
  • If request paths aren't identical.
  • If request headers aren't identical
  • Similarly for DNS, only identical DNS requests are clustered to a target.
  • Similarly for TLS, only identical TLS requests are clustered to a target.

If multiple requests are identified as identical, they are appended to a slice. Finally, the engine creates a single executer with a clusteredexecuter for all templates in a cluster.

func ClusterID

func ClusterID(templates []*Template) string

ClusterID transforms clusterization into a mathematical hash repeatable across executions with the same templates

func GetJsCompiler

func GetJsCompiler() *compiler.Compiler

func GetTemplateDoc

func GetTemplateDoc() *encoder.FileDoc

GetTemplateDoc returns documentation for the file templates_doc.go.

func PrintDeprecatedProtocolNameMsgIfApplicable

func PrintDeprecatedProtocolNameMsgIfApplicable(isSilent bool, verbose bool)

PrintDeprecatedProtocolNameMsgIfApplicable prints a message if deprecated protocol names are used Unless mode is silent we print a message for deprecated protocol name

func SignTemplate

func SignTemplate(templateSigner *signer.TemplateSigner, templatePath string) error

SignTemplate signs the tempalate using custom signer

func TemplateLogMessage

func TemplateLogMessage(id, name string, authors []string, templateSeverity severity.Severity) string

TemplateLogMessage returns a beautified log string for a template

func TemplateSignerLFA

func TemplateSignerLFA()

New Signer/Verification logic requires it to load content of file references and this is done respecting sandbox restrictions to avoid any security issues AllowLocalFileAccess is a function that allows local file access by disabling sandbox restrictions and **MUST** be called before signing / verifying any templates for intialization

func UseOptionsForSigner

func UseOptionsForSigner(opts *types.Options)

UseOptionsForSigner sets the options to use for signing templates instead of default options

func VerifyTemplateSignature

func VerifyTemplateSignature(templatePath string) (bool, error)

VerifyTemplateSignature verifies the signature of the template using default signers

Types

type Cache added in v3.2.1

type Cache struct {
	// contains filtered or unexported fields
}

Templates is a cache for caching and storing templates for reuse.

func NewCache added in v3.2.1

func NewCache() *Cache

New returns a new templates cache

func (*Cache) Has added in v3.2.1

func (t *Cache) Has(template string) (*Template, []byte, error)

Has returns true if the cache has a template. The template is returned along with any errors if found.

func (*Cache) Purge added in v3.2.1

func (t *Cache) Purge()

Purge the cache

func (*Cache) Store added in v3.2.1

func (t *Cache) Store(id string, tpl *Template, raw []byte, err error)

Store stores a template with data and error

type ClusterExecuter

type ClusterExecuter struct {
	// contains filtered or unexported fields
}

ClusterExecuter executes a group of requests for a protocol for a clustered request. It is different from normal executers since the original operators are all combined and post processed after making the request.

func NewClusterExecuter

func NewClusterExecuter(requests []*Template, options *protocols.ExecutorOptions) *ClusterExecuter

NewClusterExecuter creates a new request executer for list of requests

func (*ClusterExecuter) Compile

func (e *ClusterExecuter) Compile() error

Compile compiles the execution generators preparing any requests possible.

func (*ClusterExecuter) Execute

func (e *ClusterExecuter) Execute(ctx *scan.ScanContext) (bool, error)

Execute executes the protocol group and returns true or false if results were found.

func (*ClusterExecuter) ExecuteWithResults

func (e *ClusterExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error)

ExecuteWithResults executes the protocol requests and returns results instead of writing them.

func (*ClusterExecuter) Requests

func (e *ClusterExecuter) Requests() int

Requests returns the total number of requests the rule will perform

type Parser added in v3.2.1

type Parser struct {
	ShouldValidate bool
	NoStrictSyntax bool
	// contains filtered or unexported fields
}

func NewParser added in v3.2.1

func NewParser() *Parser

func (*Parser) LoadTemplate added in v3.2.1

func (p *Parser) LoadTemplate(templatePath string, t any, extraTags []string, catalog catalog.Catalog) (bool, error)

LoadTemplate returns true if the template is valid and matches the filtering criteria.

func (*Parser) LoadWorkflow added in v3.2.1

func (p *Parser) LoadWorkflow(templatePath string, catalog catalog.Catalog) (bool, error)

LoadWorkflow returns true if the workflow is valid and matches the filtering criteria.

func (*Parser) ParseTemplate added in v3.2.1

func (p *Parser) ParseTemplate(templatePath string, catalog catalog.Catalog) (any, error)

ParseTemplate parses a template and returns a *templates.Template structure

type Preprocessor

type Preprocessor interface {
	// Process processes the data and returns the processed data.
	ProcessNReturnData(data []byte) ([]byte, map[string]interface{})
	// Exists check if the preprocessor exists in the data.
	Exists(data []byte) bool
}

type TagFilter added in v3.2.1

type TagFilter struct {
	// contains filtered or unexported fields
}

TagFilter is used to filter nuclei templates for tag based execution

func NewTagFilter added in v3.2.1

func NewTagFilter(config *TagFilterConfig) (*TagFilter, error)

New returns a tag filter for nuclei tag based execution

It takes into account Tags, Severities, ExcludeSeverities, Authors, IncludeTags, ExcludeTags, Conditions.

func (*TagFilter) Match added in v3.2.1

func (tagFilter *TagFilter) Match(template *Template, extraTags []string) (bool, error)

Match filters templates based on user provided tags, authors, extraTags and severity. If the template contains tags specified in the deny-list, it will not be matched unless it is explicitly specified by user using the includeTags (matchAllows field). Matching rule: (tag1 OR tag2...) AND (author1 OR author2...) AND (severity1 OR severity2...) AND (extraTags1 OR extraTags2...) Returns true if the template matches the filter criteria, false otherwise.

type TagFilterConfig added in v3.2.1

type TagFilterConfig struct {
	Tags              []string
	ExcludeTags       []string
	Authors           []string
	Severities        severity.Severities
	ExcludeSeverities severity.Severities
	IncludeTags       []string
	IncludeIds        []string
	ExcludeIds        []string
	Protocols         types.ProtocolTypes
	ExcludeProtocols  types.ProtocolTypes
	IncludeConditions []string
}

type Template

type Template struct {
	// description: |
	//   ID is the unique id for the template.
	//
	//   #### Good IDs
	//
	//   A good ID uniquely identifies what the requests in the template
	//   are doing. Let's say you have a template that identifies a git-config
	//   file on the webservers, a good name would be `git-config-exposure`. Another
	//   example name is `azure-apps-nxdomain-takeover`.
	// examples:
	//   - name: ID Example
	//     value: "\"CVE-2021-19520\""
	ID string `` /* 174-byte string literal not displayed */
	// description: |
	//   Info contains metadata information about the template.
	// examples:
	//   - value: exampleInfoStructure
	Info model.Info `` /* 137-byte string literal not displayed */
	// description: |
	//   Flow contains the execution flow for the template.
	// examples:
	//   - flow: |
	// 		for region in regions {
	//		    http(0)
	//		 }
	//		 for vpc in vpcs {
	//		    http(1)
	//		 }
	//
	Flow string `` /* 223-byte string literal not displayed */
	// description: |
	//   Requests contains the http request to make in the template.
	//   WARNING: 'requests' will be deprecated and will be removed in a future release. Please use 'http' instead.
	// examples:
	//   - value: exampleNormalHTTPRequest
	RequestsHTTP []*http.Request `` /* 159-byte string literal not displayed */
	// description: |
	//   HTTP contains the http request to make in the template.
	// examples:
	//   - value: exampleNormalHTTPRequest
	// RequestsWithHTTP is placeholder(internal) only, and should not be used instead use RequestsHTTP
	// Deprecated: Use RequestsHTTP instead.
	RequestsWithHTTP []*http.Request `` /* 135-byte string literal not displayed */
	// description: |
	//   DNS contains the dns request to make in the template
	// examples:
	//   - value: exampleNormalDNSRequest
	RequestsDNS []*dns.Request `` /* 131-byte string literal not displayed */
	// description: |
	//   File contains the file request to make in the template
	// examples:
	//   - value: exampleNormalFileRequest
	RequestsFile []*file.Request `` /* 135-byte string literal not displayed */
	// description: |
	//   Network contains the network request to make in the template
	//   WARNING: 'network' will be deprecated and will be removed in a future release. Please use 'tcp' instead.
	// examples:
	//   - value: exampleNormalNetworkRequest
	RequestsNetwork []*network.Request `` /* 163-byte string literal not displayed */
	// description: |
	//   TCP contains the network request to make in the template
	// examples:
	//   - value: exampleNormalNetworkRequest
	// RequestsWithTCP is placeholder(internal) only, and should not be used instead use RequestsNetwork
	// Deprecated: Use RequestsNetwork instead.
	RequestsWithTCP []*network.Request `` /* 144-byte string literal not displayed */
	// description: |
	//   Headless contains the headless request to make in the template.
	RequestsHeadless []*headless.Request `` /* 151-byte string literal not displayed */
	// description: |
	//   SSL contains the SSL request to make in the template.
	RequestsSSL []*ssl.Request `` /* 131-byte string literal not displayed */
	// description: |
	//   Websocket contains the Websocket request to make in the template.
	RequestsWebsocket []*websocket.Request `` /* 155-byte string literal not displayed */
	// description: |
	//   WHOIS contains the WHOIS request to make in the template.
	RequestsWHOIS []*whois.Request `` /* 139-byte string literal not displayed */
	// description: |
	//   Code contains code snippets.
	RequestsCode []*code.Request `yaml:"code,omitempty" json:"code,omitempty" jsonschema:"title=code snippets to make,description=Code snippets"`
	// description: |
	//   Javascript contains the javascript request to make in the template.
	RequestsJavascript []*javascript.Request `` /* 159-byte string literal not displayed */

	// description: |
	//   Workflows is a yaml based workflow declaration code.
	workflows.Workflow `yaml:",inline,omitempty" jsonschema:"title=workflows to run,description=Workflows to run for the template"`
	CompiledWorkflow   *workflows.Workflow `yaml:"-" json:"-" jsonschema:"-"`

	// description: |
	//   Self Contained marks Requests for the template as self-contained
	SelfContained bool `` /* 175-byte string literal not displayed */
	// description: |
	//  Stop execution once first match is found
	StopAtFirstMatch bool `` /* 161-byte string literal not displayed */

	// description: |
	//   Signature is the request signature method
	//   WARNING: 'signature' will be deprecated and will be removed in a future release. Prefer using 'code' protocol for writing cloud checks
	// values:
	//   - "AWS"
	Signature http.SignatureTypeHolder `` /* 203-byte string literal not displayed */

	// description: |
	//   Variables contains any variables for the current request.
	Variables variables.Variable `` /* 184-byte string literal not displayed */

	// description: |
	//   Constants contains any scalar constant for the current template
	Constants map[string]interface{} `` /* 171-byte string literal not displayed */

	// TotalRequests is the total number of requests for the template.
	TotalRequests int `yaml:"-" json:"-"`
	// Executer is the actual template executor for running template requests
	Executer protocols.Executer `yaml:"-" json:"-"`

	Path string `yaml:"-" json:"-"`

	// Verified defines if the template signature is digitally verified
	Verified bool `yaml:"-" json:"-"`
	// TemplateVerifier is identifier verifier used to verify the template (default nuclei-templates have projectdiscovery/nuclei-templates)
	TemplateVerifier string `yaml:"-" json:"-"`
	// RequestsQueue contains all template requests in order (both protocol & request order)
	RequestsQueue []protocols.Request `yaml:"-" json:"-"`

	// ImportedFiles contains list of files whose contents are imported after template was compiled
	ImportedFiles []string `yaml:"-" json:"-"`
}

Template is a YAML input file which defines all the requests and other metadata for a template.

func ClusterTemplates

func ClusterTemplates(templatesList []*Template, options protocols.ExecutorOptions) ([]*Template, int)

func Parse

func Parse(filePath string, preprocessor Preprocessor, options protocols.ExecutorOptions) (*Template, error)

Parse parses a yaml request template file TODO make sure reading from the disk the template parsing happens once: see parsers.ParseTemplate vs templates.Parse

func ParseTemplateFromReader

func ParseTemplateFromReader(reader io.Reader, preprocessor Preprocessor, options protocols.ExecutorOptions) (*Template, error)

ParseTemplateFromReader reads the template from reader returns the parsed template

func (*Template) GetFileImports

func (template *Template) GetFileImports() []string

GetFileImports returns a list of files that are imported by the template

func (*Template) HasCodeProtocol

func (template *Template) HasCodeProtocol() bool

HasCodeProtocol returns true if the template has a code protocol section

func (*Template) ImportFileRefs

func (template *Template) ImportFileRefs(options *protocols.ExecutorOptions) error

ImportFileRefs checks if sensitive fields like `flow` , `source` in code protocol are referencing files instead of actual javascript / engine code if so it loads the file contents and replaces the reference

func (*Template) IsFuzzing added in v3.2.0

func (template *Template) IsFuzzing() bool

IsFuzzing returns true if the template is a fuzzing template

func (*Template) MarshalJSON

func (template *Template) MarshalJSON() ([]byte, error)

MarshalJSON forces recursive struct validation during marshal operation

func (*Template) MarshalYAML

func (template *Template) MarshalYAML() ([]byte, error)

MarshalYAML forces recursive struct validation during marshal operation

func (*Template) Requests

func (template *Template) Requests() int

Requests returns the total request count for the template

func (*Template) Type

func (template *Template) Type() types.ProtocolType

Type returns the type of the template

func (*Template) UnmarshalJSON

func (template *Template) UnmarshalJSON(data []byte) error

UnmarshalJSON forces recursive struct validation after unmarshal operation

func (*Template) UnmarshalYAML

func (template *Template) UnmarshalYAML(unmarshal func(interface{}) error) error

MarshalYAML forces recursive struct validation after unmarshal operation

func (*Template) UsesRequestSignature added in v3.2.3

func (template *Template) UsesRequestSignature() bool

UsesRequestSignature returns true if the template uses a request signature like AWS

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL