util

func SplitToChunks ¶

func SplitToChunks(content []byte, maxChunkSize int, lookbackWindow int) ([]string, error)

SplitToChunks splits the contents into chunks, each chunk does not exceed maxChunkSize number of bytes. Assuming the content is encoded in UTF-8. The function will guarantee not to split a multi-byte Unicode into different chunks. The function will attempt to split the chunk as close to maxChunkSize as it can, but it will also prefer splitting at line breaks ("\r\n", otherwise "\r" or "\n"), or whitespaces. It will scan for the last lookbackWindow bytes for line break/white space to split. If there is no linebreak or whitespace within lookbackWindow bytes, it will split the chunk as close to maxChunkSize (without breaking a multi-byte UTF-8 character).