hermes

package module
v0.7.17 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 19, 2023 License: MIT Imports: 5 Imported by: 0

README

Hermes Stars Watchers

Screenshot 2023-06-03 at 1 55 31 PM

How to contribute

Hermes is still in a developmental phase. Helping with test cases, benchmarks, speed, and memory usage are a great way to contribute to the project.

Hermes Go

Direct access to the hermes cache/nocache functions.

go get github.com/realTristan/hermes

Hermes Cloud

Access the hermes cache functions via websocket.

Python
pip install hermescloud

What is Hermes?

Hermes is an extremely fast full-text search and caching framework written in Go. Hermes has a cache algorithm where you can set, get, store, etc. keys and values into the cache, and a no-cache algorithm that reads data from an array or json file and uses an array to store data. Both of these algorithms provide full-text search query speeds from 10µs to 300µs.

Example of NoCache

If you want to use only the full-text-search features, then import hermes/nocache. Load the data using a .json file or array of maps. No Cache is much faster than the With-Cache algorithm, but you can't update the data.

Code

package main

import (
  "fmt"
  "time"

  hermes "github.com/realTristan/hermes/nocache"
)

func main() {
  // Initialize the full-text cache
  var ft, _ = hermes.InitWithJson("data.json")

  // Search for a word in the cache
  var res, _ = ft.Search(hermes.SearchParams{
    Query:  "tristan",
    Limit:  100,
    Strict: false,
  })

  fmt.Println(res)
}
Benchmarks
Dataset Map Entries: 4,115
Dataset Total Words: 208,092
Dataset Map Size: ≈ 2.3MB

Query: Computer, Limit: 100, Strict: False => 36.5µs
Query: Computer, Limit: 100, Strict: True => 12.102µs
data.json

Used with hermes.InitWithJson()

[
  {
    "id": 1,
    "name": {
      "$hermes.full_text": true,
      "$hermes.value": "Tristan Simpson"
    },
  },
]

With-Cache Example

The with-cache algorithm is notably slower than the no-cache algorithm. Why? Because the with-cache algorithm uses maps to store data and not an array. Import hermes directly to use the with-cache features.

Code

package main

import (
  "fmt"
  "time"
  hermes "github.com/realTristan/hermes"
)

func main() {
  // Initialize the cache
  cache := hermes.InitCache()

  // MaxSize: 10, MaxBytes: -1 (no limit), MinWordLength: 3
  cache.FTInit(10, -1, 3)

  // Set the value in the cache
  cache.Set("user_id", map[string]any{
    "name":       cache.WithFT("tristan"),
    "age":        17,
  })

  // Search for a word in the cache and print the result
  result, err := cache.Search(hermes.SearchParams{
    Query:  "tristan",
    Limit:  100,
    Strict: false,
  })

  fmt.Println(result)
}
Benchmarks
Dataset Map Entries: 4,115
Dataset Total Words: 208,092
Dataset Map Size: ≈ 2.3MB

Query: Computer, Limit: 100, Strict: False => 263.7µs
Query: Computer, Limit: 100, Strict: True => 40.84µs
data.json

Used with cache.FTInitWithJson()

{
  "user_id": {
    "age": 17,
    "name": {
      "$hermes.full_text": true,
      "$hermes.value": "Tristan Simpson"
    },
  },
}

Hermes Cloud App

Install

Coming Soon

Custom Implementation

import (
  "github.com/gofiber/fiber/v2"
  hermes "github.com/realTristan/hermes"
  socket "github.com/realTristan/hermes/cloud/socket"
)

func main() {
  // Cache and fiber app
  cache := hermes.InitCache()
  app := fiber.New()

  // Set the router
  socket.SetRouter(app, cache)

  // Listen on port 3000
  app.Listen(":3000")
}

Websocket API

Cache

cache.set
About
Set a value in the cache with the corresponding key.
Example Request
{
  "function": "cache.set",
  "key": "user_id",
  "value": base64{
    "name": "tristan"
  }
}
Response
{
  "success": true/false, 
  "data": nil
}
cache.delete
About
Delete the provided key, and the data correlated to it from the cache.
Example Request
{
  "function": "cache.delete",
  "key": "user_id"
}
Response
{
  "success": true/false, 
  "data": nil
}
cache.get
About
Get data from the cache using a key.
Example Request
{
  "function": "cache.get",
  "key": "user_id"
}
Response
{
  "success": true/false, 
  "data": map[string]any
}
cache.keys
About
Get all of the keys in the cache.
Example Request
{
  "function": "cache.keys"
}
Response
{
  "success": true/false, 
  "data": []string
}
cache.values
About
Get all of the values in the cache.
Example Request
{
  "function": "cache.values"
}
Response
{
  "success": true/false, 
  "data": []map[string]any
}
cache.length
About
Get the amount of keys stored in the cache.
Example Request
{
  "function": "cache.length"
}
Response
{
  "success": true/false, 
  "data": int
}
cache.clean
About
Clean all the data in the cache, and full-text storage.
Example Request
{
  "function": "cache.clean"
}
Response
{
  "success": true/false, 
  "data": nil
}
cache.info
About
Get the cache and full-text storage statistics.
Example Request
{
  "function": "cache.info"
}
Response
{
  "success": true/false, 
  "data": string
}
cache.exists
About
Get whether a key exists in the cache.
Example Request
{
  "function": "cache.exists",
  "key": "user_id"
}
Response
{
  "success": true/false, 
  "data": bool
}

Full-Text

ft.init
About
Intialize the full text cache.
Example Request
{
  "function": "ft.init",
  "maxbytes": -1,
  "maxsize": -1,
  "minwordlength": 3
}
Response
{
  "success": true/false, 
  "data": nil
}
ft.clean
About
Clean all of the data in the full-text storage.
Example Request
{
  "function": "ft.clean"
}
Response
{
  "success": true/false, 
  "data": nil
}
About
Search for a query in the full-text storage.
Example Request
{
  "function": "ft.search",
  "query": "tristan",
  "strict": false,
  "limit": 10
}
Response
{
  "success": true/false, 
  "data": []map[string]any
}
ft.search.oneword
About
Search for a single word in the full-text storage.
Example Request
{
  "function": "ft.search.oneword",
  "query": "tristan",
  "strict": false,
  "limit": 10
}
Response
{
  "success": true/false, 
  "data": []map[string]any
}
ft.search.values
About
Search in the cache data values. (Slower)
Example Request
{
  "function": "ft.search.values",
  "query": "tristan",
  "limit": 10,
  "schema": {
    "key": true
  }
}
Response
{
  "success": true/false, 
  "data": []map[string]any
}
ft.search.withkey
About
Search in the cache data values for a specific key.
Example Request
{
  "function": "ft.search.withkey",
  "query": "tristan",
  "key": "user_id",
  "limit": 10
}
Response
{
  "success": true/false, 
  "data": []map[string]any
}
ft.maxbytes.set
About
Set the maximum full-text storage size in bytes.
Example Request
{
  "function": "ft.maxbytes.set",
  "maxbytes": -1
}
Response
{
  "success": true/false, 
  "data": nil
}
ft.maxsize.set
About
Set the maximum full-text storage words allowed to be stored.
Example Request
{
  "function": "ft.maxsize.set",
  "maxsize": -1
}
Response
{
  "success": true/false, 
  "data": nil
}
ft.storage
About
Get the current full-text storage.
Example Request
{
  "function": "ft.storage"
}
Response
{
  "success": true/false, 
  "data": map[string][]int
}
ft.storage.size
About
Get the current full-text storage size in bytes.
Example Request
{
  "function": "ft.storage.size"
}
Response
{
  "success": true/false, 
  "data": int
}
ft.storage.length
About
Get the current full-text storage length.
Example Request
{
  "function": "ft.storage.length"
}
Response
{
  "success": true/false, 
  "data": int
}
ft.isinitialized
About
Get whether the full-text storage has been initialized.
Example Request
{
  "function": "ft.isinitialized"
}
Response
{
  "success": true/false, 
  "data": bool
}
ft.indices.sequence
About
Sequence the full-text storage indices.
Example Request
{
  "function": "ft.indices.sequence"
}
Response
{
  "success": true/false, 
  "data": nil
}

To-do

  • Testing Files
  • More Documentation
  • More Examples
  • Wrappers for other languages

License

MIT License

Copyright (c) 2023 Tristan

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WFTGetValue 

func WFTGetValue(value any) string

func WFTGetValueFromMap 

func WFTGetValueFromMap(value any) string

WFTGetValueFromMap is a function that gets the full-text value from a map.

Parameters:

  • value: any representing the value to get the full-text value from.

Returns:

  • A string representing the full-text value, or an empty string if the value is not a map or does not contain the correct keys.

Types

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a struct that represents an in-memory cache of key-value pairs. The cache can be used to store arbitrary data, and supports concurrent access through a mutex. Additionally, the cache can be configured to support full-text search using a FullText index. Fields:

  • data (map[string]map[string]any): A map that stores the data in the cache. The keys of the map are strings that represent the cache keys, and the values are sub-maps that store the actual data under string keys.
  • mutex (*sync.RWMutex): A RWMutex that guards access to the cache data.
  • ft (*FullText): A FullText index that can be used for full-text search. If nil, full-text search is disabled.

func InitCache 

func InitCache() *Cache

InitCache is a function that initializes a new Cache struct and returns a pointer to it.

Returns:

  • A pointer to a new Cache struct.

func (*Cache) Clean 

func (c *Cache) Clean()

Clean is a method of the Cache struct that clears the cache contents. If the full-text index is initialized, it is also cleared. This method is thread-safe.

Parameters:

  • None

Returns:

  • None

func (*Cache) Delete 

func (c *Cache) Delete(key string)

Delete is a method of the Cache struct that removes a key from the cache. If the full-text index is initialized, it is also removed from there. This method is thread-safe.

Parameters:

  • key: A string representing the key to remove from the cache.

Returns:

  • None

func (*Cache) Exists 

func (c *Cache) Exists(key string) bool

Exists is a method of the Cache struct that checks if a key exists in the cache. This method is thread-safe.

Parameters:

  • key: A string representing the key to check for existence in the cache.

Returns:

  • A boolean value indicating whether the key exists in the cache or not.

func (*Cache) FTClean 

func (c *Cache) FTClean() error

FTClean is a method of the Cache struct that clears the full-text cache contents. If the full-text index is not initialized, this method returns an error. Otherwise, the full-text index storage and indices are cleared, and this method returns nil. This method is thread-safe.

Returns:

  • error: An error object. If no error occurs, this will be nil.

func (*Cache) FTInit 

func (c *Cache) FTInit(maxSize int, maxBytes int, minWordLength int) error

Initialize the full-text for the cache This method is thread-safe. If the full-text index is already initialized, an error is returned.

Parameters: - maxSize: the maximum number of words to store in the full-text index. - maxBytes: the maximum size, in bytes, of the full-text index.

Returns: - error: If the full-text is already initialized.

func (*Cache) FTInitWithJson 

func (c *Cache) FTInitWithJson(file string, maxSize int, maxBytes int, minWordLength int) error

Initialize the full-text for the cache with a JSON file. This method is thread-safe. If the full-text index is already initialized, an error is returned.

Parameters: - file: the path to the JSON file to initialize the full-text index with. - maxSize: the maximum number of words to store in the full-text index. - maxBytes: the maximum size, in bytes, of the full-text index.

Returns: - error: If the full-text is already initialized.

func (*Cache) FTInitWithMap 

func (c *Cache) FTInitWithMap(data map[string]map[string]any, maxSize int, maxBytes int, minWordLength int) error

Initialize the full-text index for the cache with a map. This method is thread-safe. If the full-text index is already initialized, an error is returned.

Parameters: - data: the data to initialize the full-text index with. - maxSize: the maximum number of words to store in the full-text index. - maxBytes: the maximum size, in bytes, of the full-text index.

Returns: - error: If the full-text is already initialized.

func (*Cache) FTIsInitialized 

func (c *Cache) FTIsInitialized() bool

FTIsInitialized is a method of the Cache struct that returns a boolean value indicating whether the full-text index is initialized. If the full-text index is initialized, this method returns true. Otherwise, it returns false. This method is thread-safe.

Parameters:

  • None

Returns:

  • bool: A boolean value indicating whether the full-text index is initialized. If the full-text index is initialized, this method returns true. Otherwise, it returns false.

func (*Cache) FTSequenceIndices 

func (c *Cache) FTSequenceIndices()

When you delete a number of keys from the cache, the index remains the same. Over time, this number will grow to be very large, and will cause the cache to use a lot of memory. This function resets the indices to be sequential, starting from 0. This function is thread-safe.

func (*Cache) FTSetMaxBytes 

func (c *Cache) FTSetMaxBytes(maxBytes int) error

FTSetMaxBytes is a method of the Cache struct that sets the maximum size of the full-text index in bytes. If the full-text index is not initialized, this method returns an error. If the current size of the full-text index is greater than the new maximum size, this method returns an error. Otherwise, the maximum size of the full-text index is set to the specified value, and this method returns nil. This method is thread-safe.

Parameters:

  • maxBytes (int): An integer that represents the new maximum size of the full-text index, in bytes.

Returns:

  • error: An error object. If no error occurs, this will be nil.

func (*Cache) FTSetMaxSize 

func (c *Cache) FTSetMaxSize(maxSize int) error

FTSetMaxSize is a method of the Cache struct that sets the maximum number of words in the full-text index. If the full-text index is not initialized, this method returns an error. If the current size of the full-text index is greater than the new maximum size, this method returns an error. Otherwise, the maximum number of words in the full-text index is set to the specified value, and this method returns nil. This method is thread-safe.

Parameters:

  • maxSize (int): An integer that represents the new maximum number of words in the full-text index.

Returns:

  • error: An error object. If no error occurs, this will be nil.

func (*Cache) FTSetMinWordLength 

func (c *Cache) FTSetMinWordLength(minWordLength int) error

FTSetMinWordLength is a method of the Cache struct that sets the minimum word length for the full-text search. Parameters:

  • minWordLength (int): An integer representing the minimum word length.

Returns:

  • error: An error if the full-text search is not initialized or if the new minimum word length is greater than the maximum word length.

func (*Cache) FTStorage 

func (c *Cache) FTStorage() (map[string]any, error)

FTStorage is a method of the Cache struct that returns a copy of the full-text index storage map. If the full-text index is not initialized, this method returns an error. Otherwise, a copy of the full-text index storage map is returned, and this method returns nil. This method is thread-safe.

Returns:

  • map[string][]int: A copy of the full-text index storage map.
  • error: An error object. If no error occurs, this will be nil.

func (*Cache) FTStorageLength 

func (c *Cache) FTStorageLength() (int, error)

FTStorageLength is a method of the Cache struct that returns the number of words in the full-text index storage. If the full-text index is not initialized, this method returns an error. Otherwise, the number of words in the full-text index storage is returned as an integer, and this method returns nil. This method is thread-safe.

Returns:

  • int: An integer that represents the number of words in the full-text index storage.
  • error: An error object. If no error occurs, this will be nil.

func (*Cache) FTStorageSize 

func (c *Cache) FTStorageSize() (int, error)

FTStorageSize is a method of the Cache struct that returns the size of the full-text index storage in bytes. If the full-text index is not initialized, this method returns an error. Otherwise, the size of the full-text index storage is returned as an integer, and this method returns nil. This method is thread-safe.

Returns:

  • int: An integer that represents the size of the full-text index storage in bytes.
  • error: An error object. If no error occurs, this will be nil.

func (*Cache) Get 

func (c *Cache) Get(key string) map[string]any

Get is a method of the Cache struct that retrieves the value associated with the given key from the cache. This method is thread-safe.

Parameters:

  • key: A string representing the key to retrieve the value for.

Returns:

  • A map[string]any representing the value associated with the given key in the cache.

func (*Cache) Info 

func (c *Cache) Info() (map[string]any, error)

Info is a method of the Cache struct that returns a map with the cache and full-text info. This method is thread-safe. An error is returned if the full-text index is not initialized.

Returns:

  • A map[string]any representing the cache and full-text info.
  • An error if the full-text index is not initialized.

func (*Cache) InfoForTesting 

func (c *Cache) InfoForTesting() (map[string]any, error)

InfoForTesting is a method of the Cache struct that returns a map with the cache and full-text info for testing purposes. This method is thread-safe. An error is returned if the full-text index is not initialized.

Returns:

  • A map[string]any representing the cache and full-text info for testing purposes.
  • An error if the full-text index is not initialized.

func (*Cache) Keys 

func (c *Cache) Keys() []string

Keys is a method of the Cache struct that returns all the keys in the cache. This function is thread-safe.

Returns:

  • A slice of strings containing all the keys in the cache.

func (*Cache) Length 

func (c *Cache) Length() int

Length is a method of the Cache struct that returns the number of items stored in the cache. This function is thread-safe.

Returns:

  • An integer representing the number of items stored in the cache.

func (*Cache) Search 

func (c *Cache) Search(sp SearchParams) ([]map[string]any, error)

Search is a method of the Cache struct that searches for a query by splitting the query into separate words and returning the search results. Parameters:

  • c (c *Cache): A pointer to the Cache struct
  • sp (SearchParams): A SearchParams struct containing the search parameters.

Returns:

  • []map[string]any: A slice of maps containing the search results.
  • error: An error if the query is invalid or if the smallest words array is not found in the cache.

func (Cache) SearchOneWord 

func (c Cache) SearchOneWord(sp SearchParams) ([]map[string]any, error)

SearchOneWord searches for a single word in the FullText struct's data and returns a list of maps containing the search results. Parameters:

  • c (c *Cache): A pointer to the Cache struct
  • sp (SearchParams): A SearchParams struct containing the search parameters.

Returns:

  • []map[string]any: A slice of maps where each map represents a data record that matches the given query. The keys of the map correspond to the column names of the data that were searched and returned in the result.
  • error: An error if the query or limit is invalid or if the full-text is not initialized.

func (*Cache) SearchValues 

func (c *Cache) SearchValues(sp SearchParams) ([]map[string]any, error)

SearchValues searches for all records containing the given query in the specified schema with a limit of results to return. Parameters:

  • c (c *Cache): A pointer to the Cache struct
  • sp (SearchParams): A SearchParams struct containing the search parameters.

Returns:

  • []map[string]any: A slice of maps where each map represents a data record that matches the given query. The keys of the map correspond to the column names of the data that were searched and returned in the result.
  • error: An error if the query or limit is invalid

func (*Cache) SearchWithKey 

func (c *Cache) SearchWithKey(sp SearchParams) ([]map[string]any, error)

SearchWithKey searches for all records containing the given query in the specified key column with a limit of results to return. Parameters:

  • c (c *Cache): A pointer to the Cache struct
  • sp (SearchParams): A SearchParams struct containing the search parameters.

Returns:

  • []map[string]any: A slice of maps containing the search results
  • error: An error if the key, query or limit is invalid

func (*Cache) Set 

func (c *Cache) Set(key string, value map[string]any) error

Set is a method of the Cache struct that sets a value in the cache for the specified key. This function is thread-safe.

Parameters:

  • key: A string representing the key to set the value for.
  • value: A map[string]any representing the value to set.

Returns:

  • Error

func (*Cache) Values 

func (c *Cache) Values() []map[string]any

Values is a method of the Cache struct that gets all the values in the cache. This function is thread-safe.

Returns:

  • A slice of map[string]any representing all the values in the cache.

func (*Cache) WithFT 

func (cache *Cache) WithFT(value string) *WFT

WithFT is a function that creates a new WFT struct with the specified value.

Parameters:

  • value: A string representing the value to set in the cache and in the full-text cache.

Returns:

  • A WFT struct with the specified value or the initial string

type FullText 

type FullText struct {
	// contains filtered or unexported fields
}

FullText is a struct that represents a full-text index for a cache of data. The index is used to enable full-text search on the data in the cache.

Fields:

  • storage (map[string]any): A map that stores the indices of the entries in the cache that contain each word in the full-text index. The keys of the map are strings that represent the words in the index, and the values are slices of integers that represent the indices of the entries in the cache that contain the word.
  • indices (map[int]string): A map that stores the words in the full-text index. The keys of the map are integers that represent the indices of the words in the index, and the values are strings that represent the words.
  • index (int): An integer that represents the current index of the full-text index. This is used to assign unique indices to new words as they are added to the index.
  • maxSize (int): An integer that represents the maximum number of words that can be stored in the full-text index.
  • maxBytes (int): An integer that represents the maximum size of the text that can be stored in the full-text index, in bytes.
  • minWordLength (int): An integer that represents the minimum length of a word that can be stored in the full-text index.

type SearchParams 

type SearchParams struct {
	// The search query
	Query string
	// The limit of search results to return
	Limit int
	// A boolean to indicate whether the search should be strict or not
	Strict bool
	// A map containing the schema to search for
	Schema map[string]bool
	// Key to search in
	Key string
}

SearchParams is a struct that contains the search parameters for the Cache search methods.

type TempStorage 

type TempStorage struct {
	// contains filtered or unexported fields
}

NewTempStorage is a function that creates a new TempStorage object for a given FullText object. Parameters:

  • ft (*FullText): A pointer to the FullText object to create the TempStorage object for.

Returns:

  • (*TempStorage): A pointer to the newly created TempStorage object.

func NewTempStorage 

func NewTempStorage(ft *FullText) *TempStorage

NewTempStorage is a function that creates a new TempStorage object for a given FullText object. Parameters:

  • ft (*FullText): A pointer to the FullText object to create the TempStorage object for.

Returns:

  • (*TempStorage): A pointer to the newly created TempStorage object.

type WFT 

type WFT struct {
	// contains filtered or unexported fields
}

WFT is a struct that represents a value to be set in the cache and in the full-text cache.

func (*WFT) Set 

func (wft *WFT) Set(value string)

func (*WFT) Value 

func (wft *WFT) Value() string

Source Files

View all Source files

Directories

Path Synopsis
cloud
api
api/handlers
api/utils
socket
socket/handlers
socket/utils
wrappers/go
compression
gzip
zlib
examples
basic
////////////////////////////////////////////////////////////////////////////
 ////////////////////////////////////////////////////////////////////////////
router
/////////////////////////////////////////////////////////////////////////////
 /////////////////////////////////////////////////////////////////////////////
examples/basic
examples/router
/////////////////////////////////////////////////////////////////////////////
 /////////////////////////////////////////////////////////////////////////////
testing
api
cache
compression
fulltext
nocache/speed
socket
speed/func_speeds
speed/search_speeds

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.