README
¶
streamtools
import "github.com/thejerf/streamtools"
One of the superpowers of Go is that the io.Reader and io.Writer
interfaces were in the library from the beginning. This has been quite
successful in encouraging the ecosystem to have pervasive streaming
available.
Streams have a lot of advantages in general, such as being able to operate on vast quantities of input while consuming only a window's worth of RAM. This is especially useful for network servers, which also happens to be Go's wheelhouse, so the match is natural.
However, the programming community has decades of experience with strings, which are resources fully in memory and with random access. While this is sometimes unavoidable, it is advantageous to try as hard as possible when dealing with streams to avoid turning them into strings. Turning streams into large strings eliminates the advantages of streams.
That is intrinsically a challenge because working natively with streams is difficult. I've seen some attempts to port other string APIs into Go, but it would be preferable to have native support for readers and writers.
The purpose of this library is to provide native stream support based on
io.Readers and io.Writers.
This native support can be broken into two main categories:
- Functions for interacting with streams directly, without intervening strings being created (until necessary).
- Decorators that can be wrapped around
io.Readers orio.Writers to provide some additional inline functionality.
Operating on streams has a fundamentally different set of idioms versus operating on strings. It is important to understand that you will need fundamentally different approaches to things in a streaming environment.
Release Status
This package is currently 100% EXPERIMENTAL. As I write this the one big function it has is even broken and needs a fundamental rewrite.
Strictness
The io.Reader and io.Writer interfaces have no room for issuing
warnings. There is only errors, likely terminating the stream, and panics,
certainly causing a mess.
However, there are a lot of things within this package that require certain
constraints on the incoming values, such as having buffers large enough to
hold certain results. It isn't necessarily appropriate to error out on
everything. The things that require such things will have a Strictness
parameter on them. Strict means that violations of the preconditions will
result in an error being returned, rather than values. Lax means that the
code will do its best to continue on.
There is a package-level DefaultStrictness; this should probably only be
set in your code. There is also a LaxWarning function that
Semantic Versioning
I expect this library to grow and develop over time. A true 1.0 may take some time. The way I will work this is that the primary version number is going to 0 probably for quite a while, but during development all API changes will result in a change in the second version number. The changelog will indicate what the change is.
I especially expect that it will take some development to reveal what the naming scheme should be.
Features
- TBD.
Changelog
- v0.0.4:
- Boundary code converted into a state machine and should be correct now.
- v0.0.3:
- Add io.Closer support to the BoundaryAtomicString. Start thinking about how to handle the variety of types that may be involved.
- v0.0.2:
- The broken boundary code remains, code for consuming a reader until a byte or one of a set of bytes is added.
- v0.0.1: Initial release.
- Broken code to make it possible to read an io.Reader stream for a specific string, and get a guarantee that that string will be atomically yielded as a .Read result if read.
Documentation
¶
Index ¶
- Constants
- Variables
- func NewBoundaryString(src io.Reader, search string) io.Reader
- func NewBoundaryStringCloser(src io.ReadCloser, search string) io.ReadCloser
- func ReadUntil(r io.Reader, b byte, buf []byte) (int, bool, error)
- func ReadUntilAny(r io.Reader, ends []byte, buf []byte) (int, bool, error)
- type ErrorType
- type StreamError
Constants ¶
const ( // ErrBufferTooSmall indicates that a Read call was made with a // buffer too small for some guarantee to be held. ErrBufferTooSmall = ErrorType(iota + 1) )
Variables ¶
var DebugStackTrace bool
DebugStackTrace indicates whether or not errors from this library will yield a stack trace. This is not thread-safe. It is intended to be set at start time, generally by test code. It should not be modified once streamtools are being used.
Functions ¶
func NewBoundaryString ¶ added in v0.0.4
NewBoundaryAtomicString returns a reader that will do its best to return the search string atomically in a Read call.
An example use for this is filtering an HTTP POST body or response for passwords. Create a NewBoundaryString with a search string of "password", and it becomes safe to .Read from the reader and simply check to see if the .Read value is == "password".
If the buffer is not large enough to contain the search string, it will still be chunked, but of course the search string will not be in one .Read call.
func NewBoundaryStringCloser ¶ added in v0.0.4
func NewBoundaryStringCloser(src io.ReadCloser, search string) io.ReadCloser
NewBoundaryStringCloser is the same as NewBoundaryString, except it returns something that can also be closed.
func ReadUntil ¶ added in v0.0.2
ReadUntil will read the given reader until the given byte is encountered, and put the values into the buffer. The int will return the number of bytes written into the buffer. The boolean will be true if the read completed, and false if there is more to read yet. error will be returned from the underlying reader, if any.
The checked byte will be consumed, but nothing else will be.
func ReadUntilAny ¶ added in v0.0.2
ReadUntilAny will read the given reader until one of the given bytes is encountered, and put the values into the buffer. The int will return the number of bytes written into the buffer. The boolean will be true if the read completed, and false if there is more to read yet. error will be returned from the underlying reader, if any.
The checked byte will be consumed, but nothing else will be.
If no ends bytes are passed, this degenerates into a call to io.ReadFull, except ErrUnexpectedEOF will be ignored because the caller is not trying to guarantee that there are enough bytes to fill the buffer.
Types ¶
type ErrorType ¶
type ErrorType int
ErrorType is a constant that indicates the type of error that has occurred.
type StreamError ¶
StreamError is the type of error returned by everything in this package. It supports having an ErrorType that can be examined without having to pull apart the inner error, and an optional stack trace.
func (StreamError) Error ¶
func (se StreamError) Error() string
Error implements the error interface.
func (StreamError) Unwrap ¶
func (se StreamError) Unwrap() []error
Unwrap implements the error unwrapping protocol.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package advstreamtools provides access to more generic streaming tools, based on the io.Reader and io.Writer interface, but generalized beyond byte streams to streams of anything.
|
Package advstreamtools provides access to more generic streaming tools, based on the io.Reader and io.Writer interface, but generalized beyond byte streams to streams of anything. |