Documentation ¶
Index ¶
- Constants
- Variables
- func CheckIfArticleExistsByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) (bool, error)
- func CompressData(inout []byte, algo int, lvl int) (error, int)
- func ConnectMongoDB(who string, cfg *MongoStorageConfig) (context.Context, context.CancelFunc, *mongo.Client, *mongo.Collection, error)
- func DecompressData(inout []byte, algo int) error
- func DeleteArticlesByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) error
- func DeleteManyArticles(ctx context.Context, collection *mongo.Collection, messageIDs []string) (int64, error)
- func DisConnectMongoDB(who *string, ctx context.Context, client *mongo.Client) error
- func EncodeToGob(data []byte) ([]byte, error)
- func ExtendContextTimeout(ctx context.Context, cancel context.CancelFunc, MongoTimeout int64) (context.Context, context.CancelFunc)
- func InsertManyArticles(ctx context.Context, collection *mongo.Collection, articles []*MongoArticle) error
- func InsertOneArticle(ctx context.Context, collection *mongo.Collection, article *MongoArticle) error
- func IsDup(err error) bool
- func Load_MongoDB(cfg *MongoStorageConfig)
- func MongoWorker_UpDn_Random()
- func MongoWorker_UpDn_Scaler(cfg *MongoStorageConfig)
- func RetrieveBodyByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) ([]byte, error)
- func RetrieveHeadByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) ([]byte, error)
- func SetDefaultsIfZero(cfg *MongoStorageConfig)
- func Strings2Byte(format string, input []string) ([]byte, int)
- type COUNTER
- type MongoArticle
- type MongoArticles
- type MongoDelRequest
- type MongoGetRequest
- type MongoStorageConfig
Constants ¶
const ( DefaultMongoURI string = "mongodb://localhost:27017" DefaultMongoDatabaseName string = "nntp" DefaultMongoCollection string = "articles" DefaultMongoTimeout int64 = 15 DefaultDelQueue int = 1 DefaultDelWorker int = 1 DefaultDeleteBatchSize int = 1 DefaultInsQueue int = 1 DefaultInsWorker int = 1 DefaultInsertBatchSize int = 1 DefaultGetQueue int = 1 DefaultGetWorker int = 1 DefaultFlushTimer int64 = 1000 )
const ( NOCOMP int = 0 GZIP_enc int = 1 ZLIB_enc int = 2 FLATE_enc int = 3 )
Compression constants
Variables ¶
var ( Counter COUNTER // _queue channels handle requests for read/get, delete and insert Mongo_Reader_queue chan *MongoGetRequest Mongo_Delete_queue chan *MongoDelRequest Mongo_Insert_queue chan *MongoArticle // pass a bool [true|false] to the UpDN_ channels and workers will start or stop // external access via: mongostorage.UpDn_***_Worker_chan UpDn_StopAll_Worker_chan = make(chan bool, 1) UpDn_Reader_Worker_chan = make(chan bool, 1) UpDn_Delete_Worker_chan = make(chan bool, 1) UpDn_Insert_Worker_chan = make(chan bool, 1) LOCK_UpDnScaler = make(chan struct{}, 1) READER string = "reader" DELETE string = "delete" INSERT string = "insert" )
var (
DEBUG bool = true
)
Functions ¶
func CheckIfArticleExistsByMessageID ¶
func CheckIfArticleExistsByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) (bool, error)
CheckIfArticleExistsByMessageID checks if an article with the given MessageID exists in the MongoDB collection. function written by AI.
func CompressData ¶
CompressData is a function that takes an input byte slice 'input' and an integer 'algo' representing the compression algorithm. It compresses the input data using the specified compression algorithm and returns the compressed data as a new byte slice. function written by AI.
func ConnectMongoDB ¶
func ConnectMongoDB(who string, cfg *MongoStorageConfig) (context.Context, context.CancelFunc, *mongo.Client, *mongo.Collection, error)
ConnectMongoDB is a function responsible for establishing a connection to the MongoDB server and accessing a specific collection. It takes the following parameters: - who: A string representing the name or identifier of the calling function or worker. function written by AI.
func DecompressData ¶
DecompressData is a function that takes an input byte slice 'input' and an integer 'algo' representing the compression algorithm. It decompresses the input data using the specified compression algorithm and returns the decompressed data as a new byte slice. function written by AI.
func DeleteArticlesByMessageID ¶
func DeleteArticlesByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) error
DeleteArticlesByMessageID deletes an article from the MongoDB collection by its MessageID. function written by AI.
func DeleteManyArticles ¶
func DeleteManyArticles(ctx context.Context, collection *mongo.Collection, messageIDs []string) (int64, error)
DeleteManyArticles is responsible for deleting multiple articles from the MongoDB collection based on a given set of MessageIDes. function written by AI.
func DisConnectMongoDB ¶
DisConnectMongoDB is a function responsible for disconnecting from the MongoDB server. function written by AI.
func ExtendContextTimeout ¶
func ExtendContextTimeout(ctx context.Context, cancel context.CancelFunc, MongoTimeout int64) (context.Context, context.CancelFunc)
ExtendContextTimeout extends the deadline of a given context by canceling the previous function written by AI.
func InsertManyArticles ¶
func InsertManyArticles(ctx context.Context, collection *mongo.Collection, articles []*MongoArticle) error
InsertManyArticles is a function that performs a bulk insert of multiple articles into a MongoDB collection. function written by AI.
func InsertOneArticle ¶
func InsertOneArticle(ctx context.Context, collection *mongo.Collection, article *MongoArticle) error
InsertOneArticle is a function that inserts a single article into a MongoDB collection. function written by AI.
func Load_MongoDB ¶
func Load_MongoDB(cfg *MongoStorageConfig)
Load_MongoDB initializes the MongoDB storage backend with the specified configuration parameters. The function starts the worker goroutines in separate background routines. function written by AI.
func MongoWorker_UpDn_Random ¶
func MongoWorker_UpDn_Random()
MongoWorker_UpDn_Random periodically sends random up/down (true/false) signals to the worker channels. The purpose of this function is to simulate random up/down requests to control the worker function not written by AI. ./mongodbtest -randomUpDN -test-num 0
func MongoWorker_UpDn_Scaler ¶
func MongoWorker_UpDn_Scaler(cfg *MongoStorageConfig)
MongoWorker_UpDn_Scaler runs in the background and listens on channels for up/down requests to start/stop workers. Explanation: MongoWorker_UpDn_Scaler is responsible for managing the scaling of worker goroutines based on up/down requests. It listens to UpDn_*_Worker_chan channels to receive requests for starting or stopping specific types of workers. The function uses updn_Set to update the worker counts accordingly, which effectively starts or stops worker goroutines. This mechanism allows the application to dynamically adjust the number of worker goroutines based on the workload or other factors. Note: The function runs in the background and continues to process requests as they arrive.
function not written by AI.
func RetrieveBodyByMessageID ¶
func RetrieveBodyByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) ([]byte, error)
RetrieveBodyByMessageID is a function that retrieves the "Body" data of an article based on its MessageID. function written by AI.
func RetrieveHeadByMessageID ¶
func RetrieveHeadByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) ([]byte, error)
RetrieveHeadByMessageID is a function that retrieves the "Head" data of an article based on its MessageID. function written by AI.
func SetDefaultsIfZero ¶
func SetDefaultsIfZero(cfg *MongoStorageConfig)
SetDefaultsIfZero takes a pointer to a MongoStorageConfig and sets integer-typed fields to their default values if their current value is less than or equal to zero. It also sets default values for empty strings. This function is useful for ensuring that all the fields in the MongoStorageConfig have valid values.
Types ¶
type COUNTER ¶
type COUNTER struct {
// contains filtered or unexported fields
}
COUNTER is a simple struct that implements basic counters with concurrent access protection using a mutex.
func (*COUNTER) Dec ¶
Dec decrements the counter for the given name by one, but only if the counter value is greater than 0.
type MongoArticle ¶
type MongoArticle struct { MessageID string `bson:"_id"` Hash string `bson:"hash"` Newsgroups []string `bson:"newsgroups"` Head []byte `bson:"head"` Headsize int `bson:"hs"` Body []byte `bson:"body"` Bodysize int `bson:"bs"` Arrival int64 `bson:"at"` HeadDate int64 `bson:"hd"` Enc int `bson:"enc"` Found bool } // end type MongoArticle struct
MongoArticle represents an article stored in MongoDB.
func RetrieveArticleByMessageID ¶
func RetrieveArticleByMessageID(ctx context.Context, collection *mongo.Collection, messageID string) (*MongoArticle, error)
RetrieveArticleByMessageID retrieves an article from the MongoDB collection by its MessageID. function written by AI.
func RetrieveArticlesByMessageIDs ¶
func RetrieveArticlesByMessageIDs(ctx context.Context, collection *mongo.Collection, messageIDs []string) ([]*MongoArticle, error)
RetrieveArticlesByMessageIDs is a function that retrieves articles from the MongoDB collection based on a list of MessageIDes. function written by AI.
type MongoArticles ¶
type MongoArticles struct {
List []*MongoArticle
}
type MongoDelRequest ¶
type MongoDelRequest struct { MessageIDs []string RetChan chan int64 } // end type MongoDelRequest struct
MongoDelRequest represents a delete request for deleting articles from MongoDB.
type MongoGetRequest ¶
type MongoGetRequest struct { MessageIDs []string STAT bool RetChan chan []*MongoArticle } // end type MongoGetRequest struct
MongoGetRequest represents a read request for fetching articles from MongoDB.
type MongoStorageConfig ¶
type MongoStorageConfig struct { // MongoURI is a string representing the MongoDB connection URI. // It should include the necessary authentication details and the address of the MongoDB server. MongoURI string // MongoDatabaseName is a string representing the name of the MongoDB database where the articles will be stored and retrieved from. MongoDatabaseName string // MongoCollection is a string representing the name of the collection within the database where the articles will be stored. MongoCollection string // MongoTimeout is an int64 value defining the timeout for a connection attempt to the MongoDB server. // If the connection is not successful within this duration, it will time out. MongoTimeout int64 // DelWorker is an integer specifying the number of worker goroutines to handle article deletions. // It determines the level of concurrency for deletion operations. // More DelWorker values can improve the speed at which articles are deleted from the database. DelWorker int // DelQueue is an integer specifying the size of the delete queue. // It specifies how many delete requests can be buffered before the send operation blocks. // If DelQueue is 0 or negative, a default value will be used. // The DelQueue parameter sets the maximum number of article deletion requests that can be buffered before the worker goroutines start processing them. DelQueue int // InsWorker is an integer specifying the number of worker goroutines to handle article insertions. // It determines the level of concurrency for insertion operations. // This parameter controls the concurrency for insertion operations, allowing multiple articles to be inserted simultaneously. InsWorker int // InsQueue is an integer specifying the size of the insert queue. // It specifies how many article insertion requests can be buffered before the send operation blocks. // If InsQueue is 0 or negative, a default value will be used. // The InsQueue parameter specifies the maximum number of article insertion requests that can be buffered in the queue before the worker goroutines process them. // If the number of pending insertions exceeds this limit, the send operation on the Mongo_Insert_queue channel will block. InsQueue int // GetWorker is an integer specifying the number of worker goroutines to handle article reads. // It determines the level of concurrency for read operations. // This parameter controls the level of concurrency for read operations, enabling multiple read requests to be processed concurrently. GetWorker int // GetQueue is an integer specifying the size of the read queue. // It specifies how many read requests can be buffered before the send operation blocks. // If GetQueue is 0 or negative, a default value will be used. // The GetQueue parameter defines the maximum length of the read request queue. // When the number of read requests exceeds this limit, the send operation on the Mongo_Reader_queue channel will block until space becomes available. GetQueue int // TestAfterInsert is a boolean flag indicating whether to perform a test after an article insertion. // The specific test details are not provided in the function, and the flag can be used for application-specific testing purposes. TestAfterInsert bool // DelBatch sets the number of MessageIDs a DeleteWorker will cache before deleting to batch into one process. DelBatch int // InsBatch sets the number of Articles an InsertWorker will cache before inserting to batch into one process. InsBatch int // FlushTimer represent the default time in milliseconds for flushing batched operations to MongoDB. FlushTimer int64 } // end type MongoStorageConfig
MongoStorageConfig represents the parameters for configuring MongoDB and worker goroutines. You can adjust the number of worker goroutines and queue sizes based on your application's requirements and available resources.
func GetDefaultMongoStorageConfig ¶
func GetDefaultMongoStorageConfig() MongoStorageConfig
GetDefaultMongoStorageConfig returns a MongoStorageConfig with default values.