example1

Example gotro/W2 Project

How to use this template?

• install Go 1.16+ and clone this repo with --depth 1 flag
• copy this example1 directory to another folder (rename to projectName)
• go mod init projectName
• replace all word github.com/kokizzu/gotro/W2/internal/example1 and example1 with projectName

How to develop?

• modify or create new model/m*/*_tables.go, then run make gen-orm (will generate ORM), you may only add column/field at the end of model.
• create a new *_In, *_Out, *_Url, and the business logic methods inside domain, then run make gen-route (will generate routes and API docs).
• create an integration/unit test to make sure that your code is correct

How to release?

• change production/ configuration values
• setup the server, ssh to the production/staging server and run setup_server.sh
• cd production, run ./sync_service.sh
• run ./deploy_prod.sh

How to do multi server?

• replace id64 with string, for example1 lexid or standard uuid
• add an environment variable for SERVER_ID, and init it as lexid.ServerId
• before running deployment script, make sure to append environment variable SERVER_ID that are must unique per server

How to do multi database?

• see this blog post

Directory Structure

• 3rdparty - all third party wrapper should be here as a subfolder
• conf - all configuration constants
• domain - contains your business logic, these are the one that should be integration/unit tested
• model - contains your domains' data store
  • m[Domain] - contains data store that should be grouped inside that domain
    • rq[Domain] - read query (R from CQRS), you can add a new file here to extend the default ORM
    • sa[Domain] - statistics analytics (event source), you can add a new file here to extend the default ORM
    • wc[Domain] - write command (C from CQRS), you can add a new file here to extend the default ORM
    • *_table.go - the schema file for that domain, to generate the ORM and as an input for migration
• production - scripts and env for deploying to production
• svelte - frontend (can be replaced with any framework)

outer files:

• main_*.GEN.go - will be generated per transport/presentation/adapter (eg. gRPC, REST, WebSocket, CLI, etc)

Setup

# install tools required for codegen
make setup-deps

# install reverse proxy
make setup-webserver

# install dependencies for web frontend (Svelte with ESBuild): localhost:5500
make webclient

# start dependencies (Tarantool, Clickhouse, Mailhog): localhost:3301, localhost:9000, localhost:1025
make compose

# run api server (Go with Air auto-recompile): localhost:9090
make apiserver

# run reverse proxy (Caddy): localhost:80
make reverseproxy

Usage

# connect to OLTP database
tarantoolctl connect 3301

# connect to OLAP database
clickhouse-client

# generate ORM (after add new table or columns on models/m*/*_tables.go)
make gen-orm

# generate route (after add new _In+_Out struct, _Url const and business logic method on domain/*.go)
make gen-route

Every new *.svelte file will automatically generate corresponding *.html file, also will automatically generate index.GEN.go, so you can create proper handler to inject into _layout.html or json into the generated *.html file. For now, please rerun make webclient after adding new *.svelte to generate new route.

Gotchas

• Calling direct assignment (=) instead of wc*.Set*() before calling wc*.DoUpdateBy*() will do nothing, as direct assignment does not append mutation property

# proper way to update
x := mAuth.NewUsersMutator(s.Taran)
x.Id = ...
if !x.FindById() {
   // not found
   return // or x.DoInsert() then continue with update
}
x.SetBla(..) 
x.SetFoo(..)
x.SetBar(..)
x.SetBaz(..)
x.SetBaz(..) // calling twice on the same column will cause DoUpdate to fail
if !x.DoUpdateById() {
   // failed to update
}

# but if you need only insert or replace, you can use = directly
x := mAuth.NewUsersMutator(s.Taran)
x.Bla = ..
x.Foo = ..
x.Bar = ..
x.Baz = ..
x.DoInsert() or x.DoUpsert() // calling DoUpdateBy*() will do nothing, since mutation property only set when calling .Set*() method

• Clickhouse inserts are buffered using chTimedBuffer, so you must wait ~1s to ensure it's flushed
• Clickhouse have eventual consistency, so you must use FINAL query to make sure it's force-committed
• You cannot change Tarantool's datatype
• You cannot change Clickhouse's ordering keys datatype
• Currently migration only allowed for adding columns/fields at the end (you cannot insert new column in the middle/begginging)
• All Tarantool's columns always set not null after migration (I hate null values XD)
• Tarantool does not support client side transaction (so you must use Lua or 2PC or split into SAGAs)
• Current parser/codegen does not allow calling SetError with more than 1 concatenation or complex expression or non constant left-hand-side, eg. d.SetError(500, "error on" + Bla(bar) + Yay(baz)), you must repharase the error detail into something like this: d.SetError(500, "error on " + msg)

TODOs

• Add SEO pre-render: Rendora
• Add search-engine: TypeSense or MeiliSearch for single node
• Add more persisted cache option: IceFireDB or Aerospike
• Add external storage upload example1 (minio? wasabi?)
• Replace LightStep with SigNoz, tutorial, NewRelic and/or datav or just dump it to Clickhouse directly and MQ like RedPanda as fallback
• Add Analytics: Jitsu or Materialize
• Add example1 deployment script with LXC/LXD share for single server multi-tenant or Docker Compose or Docker Swarm
• Add NBIO codegen for websocket presentation/transport layer

File Upload Example

the schema (/model/m[Something]/[something]_tables.go), after creating this, run make gen-orm:

const (
	TableMediaUploads Tt.TableName = `mediaUploads`
	Id            = `id`
	CreatedBy     = `createdBy`
	CreatedAt     = `createdAt`
	UpdatedBy     = `updatedBy`
	UpdatedAt     = `updatedAt`
	DeletedBy     = `deletedBy`
	DeletedAt     = `deletedAt`
	IsDeleted     = `isDeleted`
	RestoredBy    = `restoredBy`
	RestoredAt    = `restoredAt`
	SizeByte      = `sizeByte`
	FilePath      = `filePath`
	ContentType   = `contentType`
	OrigName      = `origName`
)

var TarantoolTables = map[Tt.TableName]*Tt.TableProp{
	TableMediaUploads: {
		Fields: []Tt.Field{
			{Id, Tt.Unsigned},
			{CreatedAt, Tt.Integer},
			{CreatedBy, Tt.Unsigned},
			{UpdatedAt, Tt.Integer},
			{UpdatedBy, Tt.Unsigned},
			{DeletedAt, Tt.Integer},
			{DeletedBy, Tt.Unsigned},
			{IsDeleted, Tt.Boolean},
			{RestoredAt, Tt.Integer},
			{RestoredBy, Tt.Unsigned},
			{SizeByte, Tt.Unsigned},
			{FilePath, Tt.String},
			{ContentType, Tt.String},
			{OrigName, Tt.String},
		},
		Unique1: Id,
		Unique2: FilePath,
	},
}

func GenerateORM() {
	Tt.GenerateOrm(TarantoolTables)
}

// don't forget to add migration on model.go:
//	m.Taran.MigrateTables(mSomething.TarantoolTables)

the code for domain/business logic /domain/media.go, after creating this, run make gen-route:

type (
	MediaUpload_In struct {
		RequestCommon
		UploadId   uint64 `json:"uploadId,string" form:"uploadId" query:"uploadId" long:"uploadId" msg:"uploadId"`
		FileBinary string `json:"fileBinary" form:"fileBinary" query:"fileBinary" long:"fileBinary" msg:"fileBinary"`
	}
	MediaUpload_Out struct {
		ResponseCommon
		MediaUpload *rqSomething.MediaUploads `json:"mediaUpload" form:"mediaUpload" query:"mediaUpload" long:"mediaUpload" msg:"mediaUpload"`
	}
)

const MediaUpload_Url = `/MediaUpload`

func (d *Domain) MediaUpload(in *MediaUpload_In) (out MediaUpload_Out) {
	sess := d.mustAdmin(in.SessionToken, in.UserAgent, &out.ResponseCommon)
	if sess == nil {
		return
	}

	if len(in.Uploads) == 0 {
		out.SetError(400, `missing fileBinary, enctype not multipart/form-data?`)
		return
	}

	up := wcSomething.NewMediaUploadsMutator(d.Taran)
	up.Id = in.UploadId
	if in.UploadId > 0 {
		if !up.FindById() {
			out.SetError(404, `upload not found, wrong env?`)
			return
		}
	}
	if up.CreatedAt == 0 {
		up.Id = id64.UID()
		up.CreatedAt = in.UnixNow()
		up.CreatedBy = sess.UserId
	}
	up.UpdatedAt = in.UnixNow()
	up.UpdatedBy = sess.UserId
	for fileName, tmpFile := range in.Uploads {
		up.OrigName = fileName
		oldPath := up.FilePath
		uriPath := conf.UPLOAD_URI + conf.MEDIA_SUBDIR
		dir := conf.UPLOAD_DIR + conf.MEDIA_SUBDIR
		idStr := I.UToS(up.Id)
		if S.StartsWith(oldPath, uriPath) {
			oldPath = dir + S.RightOf(oldPath, uriPath)
		} else if oldPath != `` {
			L.Print(`ERROR weird name format to be replaced for mediaUpload.id: ` + idStr + `:` + oldPath)
		}

		mtype, err := mimetype.DetectFile(tmpFile)
		if L.IsError(err, `cannot detect file type: `+tmpFile) {
			out.SetError(500, `cannot detect file type: `+up.OrigName)
			return
		}

		err = os.MkdirAll(dir, 0755)
		if L.IsError(err, `failed to create upload directory: `+dir) {
			out.SetError(500, `cannot create upload directory`)
			return
		}
		ext := S.ToLower(filepath.Ext(fileName))
		newName := idStr + ext
		newPath := dir + newName
		err = os.Rename(tmpFile, newPath)
		if L.IsError(err, `failed to rename `+tmpFile+` to `+newPath) {
			out.SetError(500, `failed moving uploaded file`)
			return
		}
		in.Uploads[fileName] = oldPath // delete old file later
		up.FilePath = uriPath + newName
		stat, err := os.Stat(newPath)
		if L.IsError(err, `failed to stat moved file: `+newPath) {
			out.SetError(500, `failed to stat moved file`)
			return
		}
		up.SizeByte = uint64(stat.Size())
		up.ContentType = mtype.String()

		// ignore if upload more than one
		break
	}
	//if in.DoDelete {
	//	up.IsDeleted = true
	//	up.DeletedAt = in.UnixNow()
	//	up.DeletedBy = sess.UserId
	//}
	//if in.DoRestore {
	//	up.IsDeleted = false
	//	up.RestoredAt = in.UnixNow()
	//	up.RestoredBy = sess.UserId
	//}

	if !up.DoUpsert() {
		out.SetError(500, `cannot upsert media`)
		return
	}
	out.MediaUpload = &up.MediaUploads
	return
}

Testing with CURL

alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
time curl -X POST -H 'content-type: application/json' -d '{"email":"root@gmail.com","password":"123"}' http://localhost:9090/api/UserLogin
time curl -X POST -H 'content-type: application/json' -d '{"email":"root@gmail.com","password":"123","userName":"kokizzu"}' http://localhost:9090/api/UserRegister

cd; go install github.com/rakyll/hey@latest
hey -c 50 -n 200 http://localhost:9090/api/health
hey -c 1000 -n 100000 http://localhost:9090/api/UserLogin\?email\=root@gmail.com

Testing GraphQL

After starting docker-compose up and make apiserver, open localhost:9090/graphql and run these query:

mutation _ {
  UserLogin(email: "root@localhost", password: "test123", debug: true) {
    ResponseCommon {
      debug
      error
      sessionToken
      statusCode
    }
  }
}

TODOs:

• add more tests (eg. multiple cart)
• generate graphql schema with relationship to other model (and max nested and rate limiter)