sculptor

package module
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 14, 2023 License: MIT Imports: 8 Imported by: 0

README

Sculptor

FlexFormer Logo

Go Report Card

Sculptor is a flexible and powerful Go library for transforming data from various formats (CSV, JSON, etc.) into desired Go struct types.

It is designed to significantly simplify and accelerate the process of data ingestion and formatting.

Features

  • Ease of use: Simply define your struct with struct field tag and let sculptor handle the rest.
  • Support for multiple data file formats: CSV, JSON, and more.
  • Customized your processing: setup fallbackfuncs and customfuncs to handle your data.
  • More Options: Set your own delimiter, data parser func and time latency to control the speed of data processing.
  • Efficient handling of large data sets: Optimize memory usage and performance.

Installation

go get github.com/esonhugh/sculptor

Quick Start

package main

import (
        "github.com/esonhugh/sculptor"
	"log"
)

type TestStruct struct {
	Name string `select:"name"`
	Pass string `select:"pass"`
}

func main() {
	Doc := sculptor.NewDataSculptor("test.json").
		SetDocType(sculptor.JSON_DOCUMENT).
		SetQuery("name", "user").
		SetQuery("pass", "pass").
		SetTargetStruct(&TestStruct{})
	Doc.Do()  // not block.
	go func() {
		Doc.Wait()  // let process complete and close chan 
		// Doc.Wg.Wait()  // is real Doc.Wait() do.
		Doc.Close()  // Same as close(Doc.ConstructedOutput)
    }()
	for i := range Doc.ConstructedOutput {
		log.Print(i)
	}
}

Demo

demo of merging oneforall.csv and subfinder.json

More Example

Checkout the test folder to see more examples.

Doc

godoc: https://pkg.go.dev/github.com/esonhugh/sculptor

Contributing

welcome contributions! Free for fork and Pull Request.

License

sculptor is licensed under the MIT License. See LICENSE for more details.

Made with ❤ by Esonhugh

Inspired by AkaAny/gormja_core2

Inspired by Trickest

Documentation

Overview

Package sculptor is flexible and powerful Go library for transforming data from various formats (CSV, JSON, etc.) into desired Go struct types. Getting Start with 

package main

import (
	"fmt"
	"github.com/esonhugh/sculptor"
)

type TestStruct struct {
	Name string `select:"name"`
	Pass string `select:"pass"`
}

// json like `{"user": "username","pass": "114514","content": "123444", "id":2}`
// csv like `user,pass,content,id\nusername,114514,123444,2`

Doc := sculptor.NewDataSculptor("your_test.json").
	SetDocType(sculptor.JSON_DOCUMENT).
	SetQuery("name", "user").
	SetQuery("pass", "pass").
	SetTargetStruct(&TestStruct{})
go Doc.Do()
for i := range Doc.ConstructedOutput {
	fmt.Println(i)
}

Index

Constants

View Source 
const (
	JSON_DOCUMENT = DocumentType("json")
	CSV_DOCUMENT  = DocumentType("csv")
)

DataExtrator types supported

Variables

View Source 
var DefaultOptions = Options{
	Latency:    1 * time.Millisecond,
	BufSize:    10,
	TagKey:     "select",
	DebugLevel: log.WarnLevel,
}

DefaultOptions is the default options used by DataSculptor

Functions

func Merge added in v0.1.2 

func Merge(sculptors ...*DataSculptor) chan any

Merge merges multiple DataSculptor output in one channel

Types

type DataSculptor 

type DataSculptor struct {

	// DocType is the type of document to be extracted
	DocType DocumentType

	// Filename is processed filename or path
	Filename string

	// Wg set for goruntime if thread process.
	Wg *sync.WaitGroup

	// ConstructedOutput is the channel to send the extracted data out
	ConstructedOutput chan any
	// contains filtered or unexported fields
}

DocumentType is core struct in package. it contains all the information required to extract data from a document during runtime.

func NewDataSculptor 

func NewDataSculptor(file string) *DataSculptor

NewDataSculptor returns a new DataSculptor with initialized values

func NewDataSculptorWithOptionsAndWg added in v0.1.3 

func NewDataSculptorWithOptionsAndWg(file string, o Options, wg *sync.WaitGroup) *DataSculptor

NewDataSculptorWithOptionsAndWg returns a new DataSculptor with initialized values

func NewDataSculptorWithWg added in v0.1.3 

func NewDataSculptorWithWg(file string, wg *sync.WaitGroup) *DataSculptor

NewDataSculptorWithWg returns a new DataSculptor with initialized values

func (*DataSculptor) Close added in v0.1.3 

func (d *DataSculptor) Close()

func (*DataSculptor) CurrentTarget 

func (d *DataSculptor) CurrentTarget() any

CurrentTarget func returns the current target struct during process. Helpful in SetFallbackFunc and SetCustomFunc.

func (*DataSculptor) Do 

func (d *DataSculptor) Do() *DataSculptor

Do func extract the data from file and put it into the ConstructedOutput chan Do() is no not blocked.

func (*DataSculptor) Error 

func (d *DataSculptor) Error() error

Error() func returns the last error occurred while processing the record. Helpful in SetFallbackFunc and SetCustomFunc.

func (*DataSculptor) SetBufSize added in v0.1.2 

func (d *DataSculptor) SetBufSize(size int)

SetBufSize will reset the buffer size of the channel

func (*DataSculptor) SetCSVDelimiter 

func (d *DataSculptor) SetCSVDelimiter(r rune) *DataSculptor

SetCSVDelimiter sets the delimiter for the CSV document if you set the document type to CSV. Else it will make error.

func (*DataSculptor) SetCustomFunc 

func (d *DataSculptor) SetCustomFunc(f ...Func) *DataSculptor

SetCustomFunc sets the customFunc which will be called between constructing targetStruct complete and sending the extracted data to the channel.

func (*DataSculptor) SetDocType 

func (d *DataSculptor) SetDocType(docType DocumentType) *DataSculptor

SetDocType sets the document type for the given filename. If Supported, It will automatically set the scanner for the given document type. If not supported, it will panic. If you want Set you own scanners please use SetScanner() and follow the interface.

func (*DataSculptor) SetFallbackFunc 

func (d *DataSculptor) SetFallbackFunc(f ...Func) *DataSculptor

SetFallbackFunc sets the fallbackFunc which will be called when framework can't handle the record.

func (*DataSculptor) SetOption 

func (d *DataSculptor) SetOption(o Options) *DataSculptor

SetOption set options to replace default options

func (*DataSculptor) SetQuery 

func (d *DataSculptor) SetQuery(tagName string, Query string) *DataSculptor

SetQuery sets the query for the given tag name

func (*DataSculptor) SetScanner 

func (d *DataSculptor) SetScanner(dataParser parser.RawDataParser) *DataSculptor

SetScanner sets the scanner (parser.RawDataParser) which used to extract data from the document. SetScanner is helpful if you want to use your own scanner to process your file.

func (*DataSculptor) SetTargetStruct 

func (d *DataSculptor) SetTargetStruct(targetStruct any) *DataSculptor

SetTargetStruct sets the target struct with the given struct pointer. Helpful in SetFallbackFunc and SetCustomFunc. It will be called when init before the Do() func.

func (*DataSculptor) Wait added in v0.1.3 

func (d *DataSculptor) Wait()

type DocumentType 

type DocumentType string

type Func 

type Func func(*DataSculptor) error

Func is alias of DataSculptor customFunc and fallbackFunc type

type Options 

type Options struct {
	Latency    time.Duration
	BufSize    int
	TagKey     string
	DebugLevel log.Level
}

Options is used for extra Control during DataSculptor.Do()

Source Files

View all Source files

Directories

Path Synopsis
Package parser is transformers collection of parsing any type of social_engineering_data to our own data struct
 Package parser is transformers collection of parsing any type of social_engineering_data to our own data struct
query
test
MergeTwoMoreDocument

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.