go_with_concurrent_map_builtin

module
v0.0.0-...-a068d24 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 2, 2017 License: BSD-3-Clause

README

Concurrent Map Built-in for the Go Programming Language

Highlights of Project

In case you're someone perusing my profile and stumbled upon this repository, here are the 'highlights' of the project.

Concurrent Map Implementation

Comparison between fork and official

Awards

I have been granted two prestiguous awards from this project (one directly, one the main contribution).

Peer's Choice for Outstanding Project at Lehigh University's N.S.F-Sponsored R.E.U "Smart Spaces", school article here, official website down.

Honorable Mention for Outstanding Undergraduate Researcher for 2017 from the Computing Research Association's commitee sponsored by Microsoft Research, link

Why

This fork of the Go programming language solves a problem which the developers seek to ignore. Whether it be from laziness, a design decision, or it being deemed an impossibilty, for my academic research, I decided to solve this problem by tackling it head-on. The end result was a very, VERY high performance data structure which offered great utility, and is implemented in such a way that it is not only backwards compatible, but also compatible with the normal Go map syntax.

Progress

Now, as mentioned above, it originally was for academic research, hence I am the sole developer for this project. With this being the case, one person can only do so much, but, not to toot my own horn, I have done alot. Below you will see the current progress of the project in terms of what the map currently supports. It is segmented into the following sections...

  1. Core: This is what the data structure MUST support to even be remotely viable for public use.
  2. Sane: This is what the data structure SHOULD support to allow sane and reasonable usage.
  3. Nicety: This what the data structure COULD support, and would be nice additions.
Core

The backbone of any map; the operations it supports. Any and all maps, especially for compatibility with Go, NEED to support insert, lookup, removal, and of course, iteration. The below denotes the current status of the map in terms of completion. Keep in mind, for a concurrent map, this also includes the guarantee that these operations are fully atomic. Note that for these to pass, they MUST support the same syntax as Go's normal map.

  • Insertion
  • Lookup
  • Removal
  • Iteration
Sane

The most sane semantic that SHOULD be offered is the mutual exclusion of elements beyond a single operation. For reference, without this even something as simple as an increment (as basic of a Read-Modify-Write operation as you can get) are 3 distinct operations, two of which access the map. This is provided through the sync.Interlocked(map, key) function.

  • Single Key Exclusive Access
  • Multi Key Exclusive Access

Note:

Multi Key Exclusive Access CAN be supported, but due to time constraints, it will not be implemented in time. To support this, we must extend the above mentioned function to be variadic, (e.g., sync.Interlocked(map, key...)). From here, we may provide a global locking order of which all concurrent multi key mutual access must adhere to to prevent deadlock. The currently planned implementation is to order each key by the position they hash to, and acquire them (recursively) in that order. (e.g, Order by hash(key) % size and do this for whatever keys hash to the same nested bucketArray recursively, until you find the bucketData they correspond to).

Nicety

For true Go compatibility, the ability to use reflection on the concurrent map, while not a necessity, is a nicety. This does not only include adding support for the concurrent map in terms of it's underlying type (I.E, a way to determine, reflectively, if a map is concurrent or not), but also support for core functions (I.E, printing a concurrent map).

  • Reflection

Note:

Unfortunately, this has not been done, and truthfully, I doubt it will be (by me anyway), as the map, as is, is enough to demonstrate it's uses. Currently, you cannot even print out the map while it is being modified due to the implementation (thanks to Go's minimalism) does not use a conventional for loop, but instead calls the mapiternext function directly. While this is a very simple fix, I don't feel up to it.

Usage

Creation

How does one create the concurrent map? make has been modified to accept a third argument for creating a map. Previously, the first argument declared the type of the map, and the second argument was a hint to the runtime for how many buckets it should prepare in advance.

Now there is a third argument, which is the concurrency level, wherein this is the number of Goroutines that are expected to navigate the map. Just like how make normally works, if the second or third arguments are left out, they are replaced with 0 in the compiler. If concurrency level is 0, then it will create a normal map. If concurrency level > 0, then it will create an concurrent map.

cmap := make(map[int]int, NUM_ELEMS, NUM_CONCURRENCY)
Compatibility

The way Go works is that a map, I.E map[int]int, has its type managed by the compiler, and the actual object itself is just the header for the map. This header is opaque enough that we can modify it without effecting the user's code, and hence the concurrent map can be used wherever the normal hash map can be.

For example...

func fn(m map[int]int) {
	// Supports Insertion...
	m[k] = v
	// Supports Lookup...
	v = m[k]
	// Supports Removal...
	delete(m, k)
	// Supports iteration...
	for k, v := range m {
		k++
		v++
	}
}

func main() {
	// Declare map
	var m map[int]int
	
	// Default map
	m = make(map[int]int, NUM_ELEMS)
	// Pass default map
	fn(m)

	// Concurrent map
	m = make(map[int]int, NUM_ELEMS, NUM_CONCURRENCY)
	// Pass concurrent map
	fn(m)
}
Atomicity

All accesses to the map are guaranteed to be atomic, in repect to that operation (insert, lookup, and removal). Atomicity is extremely important in concurrent data structures, and synchronization is allowed through the usage of internal spinlocks that are coupled to each bucket. While a Goroutine holds a bucket, it's access is guaranteed to be atomic and synchronized.

// Assignment or update will not overlap with another operation.
map[key] = value
// Lookup will safely yield correct result.
value := map[key]
// Removal will safely leave the map in a well-defined state.
delete(map, key)
Read-Modify-Write

The API exposes the internal locking mechanism to the user to allow for prolonged access to individual elements. It is exposed through two new additions to the sync package, named Interlocked and Release. This allows the user to form a single-element mutex, but also forces the user to ONLY have access to that one element so long as that lock is held. Note that the release can also be deferred.

// Acquire lock on key.
sync.Interlocked(map, key)
// Atomically determine if the object is present, and perform some well-informed decision on it.
if obj, isPresent := map[key]; isPresent {
	update(obj)
} else {
	map[key] = NewObj()
}
// Release lock on current key
sync.Release(map)

As mentioned above, you may defer the release of the lock.

defer sync.Release(map)

Notice as well that sync.Release does not require the key as well; this is because it keeps track of the current key interlocked for you. With that said, there is a special invariant that will be brought up later.

Iteration

The map supports iteration that is not only scalable, but guarantees a very strong guarantee that you retain mutual exclusion so long as you are iterating over it. In consequence, since you only ever hold the lock on that element, and ONLY that element, it allows concurrent operations to occur while you are iterating (e.g., you may have other inserts, lookups, removals, and even iterations occuring in parallel/concurrently).

As well, iteration is handled in an elegant way in that iterators will skip over any locked buckets (eliminating lock convoying) to be processed later.

Bug

Unfortunately, if the user breaks from the loop early, it leaves the map in an inconsistent state... that is, since you hold the lock on that bucket, and it is only ever unlocked after it finishes processing that bucket, returning, breaking, or jumping out of the loop can cause deadlocks. This can be a very easy fix by looking for any such unconditional jumps and implementing some kind of function to cleanup the state.

However, in the meantime, the user may call sync.Release(map) if they need to do so. Note that, unfortunately, it will also leave the map in an inconsistent state if the user had deleted other items prior to making any unconditional jump... Again, this can be fixed, but will not be by me due to time constraints.

Invariant: One-Key-Per-Goroutine

This invariant must ALWAYS hold: Only one Goroutine may hold a lock at any given time. This is to prevent deadlocks from occuring, as well as for simplicity. If you attempt to access another key while one is interlocked (either with sync.Interlocked or during iteration) it will result in a panic.

Note:

There is a way around this, and it is rather simple...

If it is important to the user that they be able to access other buckets, then in the future there will be the ability to interlock multiple keys, however iteration will never allow this.

A clever way to read 'One-Key-Per-Goroutine' is that while only one Goroutine may have a key, you can still spawn another to access the map for you.

for key, value := range cmap {
	go func(key, value int) {
		cmap[key] = value
	}(key+1, value+1)
}

Note that this is deadlock free... even if the spawned Goroutine's key is held in the bucket we are currently iterating over, they will just be blocking on us (meaning, it is NOT SAFE to block on a spawned child accessing the map). Unfortunately, this invariant is something which is mandatory and cannot be side-stepped further without invoking deadlocks.

Performance

Default Map vs Concurrent Map

The default map, while very well optimized, is not concurrent safe, and so, does not scale well at all when it can be mutated at any time. Reader-Writer locks also do not help the issue where you have large amounts of writers, and simple Mutexes limit all access to a single Goroutine, while the concurrent map allows as many as possible.

The default map is faster for single-Goroutine access, such as if a mutex is not needed, however, the Concurrent Map scales extremely well and wins out at a breakpoint of 2 Goroutines. This means that, if you ever need a mutex, then the concurrent map wins out.

For raw insert, lookup, and removal the concurrent map is about 6 - 7x faster (on the hardware tested on). For read-write iteration, wherein the map can be mutated and a lock is needed to iterate the map, the concurrent map is about 8 - 9x faster. When you can have operations combined, it goes up to about 9 - 10x.

While the concurrent map does have limitations due to it's invariant, it is extremely useful.

Concurrent Map vs Others

The two most interesting attempts at providing a concurrent map was Streamrail's and Gotomic's. While it is clear that Streamrail fails to compete with our map, Gotomic does make a promising competitor. This is because it is an implementation of the Split Ordered List Hash Table, which is a lock-free implementation. While it does not support proper Read-Write iteration, it was an interesting contendor, and is faster in some cases where there are a low amount of elements, but loses in majority of others.

P.S: This is a temporary README, don't expect it to look good yet.

Benchmarks & Results

Intset

Integer set tests insert, lookup, and removal being done by multiple Goroutines. Performance measurements are taken in the amount of work being done in terms of how much time it takes to complete.

80/10/10 Lookup/Insert/Remove

Intset Lookup Biased

34/33/33 Lookup/Insert/Remove

Intset Benchmark

Iteration
Read-Only

Tests raw iteration without the need for a lock. This demonstrates that the concurrent map is, in fact, slower than an unsynchronized map. This is an unfair test, however, as the concurrent map is lock-based, and so it could never directly compete, however for credibility sake, we shows the ups and downs of the map.

Read-Only Benchmark

Read-Write

Tests iteration wherein you can have mutations to the map. This also tests mutations of the current key itself, and demonstrates the ability of fast-pathing when accessing the map (on the key being iterated over) while iterating.

Read-Write Benchmark

Combined

Tests all operations (insert, lookup, removal, and iteration), and shows the true scalability of the concurrent map. Note that, with a mutex, one must acquire it just to complete any one of these operations, while the concurrent map can freely do all of these operations safely.

Read-Only Iteration

Combined Benchmark

Read-Write Iteration

Combined Read-Write

Directories

Path Synopsis
doc
codewalk
Generating random text: a Markov chain algorithm
Generating random text: a Markov chain algorithm
play
An implementation of Conway's Game of Life.
An implementation of Conway's Game of Life.
misc
android
Cleaner removes anything from /data/local/tmp/goroot not on a builtin list.
Cleaner removes anything from /data/local/tmp/goroot not on a builtin list.
cgo/gmp
An example of wrapping a C library in Go.
An example of wrapping a C library in Go.
cgo/test
Test that the #cgo CFLAGS directive works, with and without platform filters.
Test that the #cgo CFLAGS directive works, with and without platform filters.
ios
This program can be used as go_darwin_arm_exec by the Go tool.
This program can be used as go_darwin_arm_exec by the Go tool.
linkcheck
The linkcheck command finds missing links in the godoc website.
The linkcheck command finds missing links in the godoc website.
nacl
Mkzip creates a zip file from a 'proto' file describing the contents.
Mkzip creates a zip file from a 'proto' file describing the contents.
sortac
Sortac sorts the AUTHORS and CONTRIBUTORS files.
Sortac sorts the AUTHORS and CONTRIBUTORS files.
src
archive/tar
Package tar implements access to tar archives.
Package tar implements access to tar archives.
archive/zip
Package zip provides support for reading and writing ZIP archives.
Package zip provides support for reading and writing ZIP archives.
bufio
Package bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or Writer) that also implements the interface but provides buffering and some help for textual I/O.
Package bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or Writer) that also implements the interface but provides buffering and some help for textual I/O.
builtin
Package builtin provides documentation for Go's predeclared identifiers.
Package builtin provides documentation for Go's predeclared identifiers.
bytes
Package bytes implements functions for the manipulation of byte slices.
Package bytes implements functions for the manipulation of byte slices.
cmd/addr2line
Addr2line is a minimal simulation of the GNU addr2line tool, just enough to support pprof.
Addr2line is a minimal simulation of the GNU addr2line tool, just enough to support pprof.
cmd/api
Binary api computes the exported API of a set of Go packages.
Binary api computes the exported API of a set of Go packages.
cmd/asm
Asm, typically invoked as “go tool asm”, assembles the source file into an object file named for the basename of the argument source file with a .o suffix.
Asm, typically invoked as “go tool asm”, assembles the source file into an object file named for the basename of the argument source file with a .o suffix.
cmd/asm/internal/asm
Package asm implements the parser and instruction generator for the assembler.
Package asm implements the parser and instruction generator for the assembler.
cmd/asm/internal/flags
Package flags implements top-level flags and the usage message for the assembler.
Package flags implements top-level flags and the usage message for the assembler.
cmd/asm/internal/lex
Package lex implements lexical analysis for the assembler.
Package lex implements lexical analysis for the assembler.
cmd/cgo
Cgo enables the creation of Go packages that call C code.
Cgo enables the creation of Go packages that call C code.
cmd/compile
Compile, typically invoked as “go tool compile,” compiles a single Go package comprising the files named on the command line.
Compile, typically invoked as “go tool compile,” compiles a single Go package comprising the files named on the command line.
cmd/cover
Cover is a program for analyzing the coverage profiles generated by 'go test -coverprofile=cover.out'.
Cover is a program for analyzing the coverage profiles generated by 'go test -coverprofile=cover.out'.
cmd/doc
Doc (usually run as go doc) accepts zero, one or two arguments.
Doc (usually run as go doc) accepts zero, one or two arguments.
cmd/fix
Fix finds Go programs that use old APIs and rewrites them to use newer ones.
Fix finds Go programs that use old APIs and rewrites them to use newer ones.
cmd/go
Go is a tool for managing Go source code.
Go is a tool for managing Go source code.
cmd/gofmt
Gofmt formats Go programs.
Gofmt formats Go programs.
cmd/internal/bio
Package bio implements common I/O abstractions used within the Go toolchain.
Package bio implements common I/O abstractions used within the Go toolchain.
cmd/internal/gcprog
Package gcprog implements an encoder for packed GC pointer bitmaps, known as GC programs.
Package gcprog implements an encoder for packed GC pointer bitmaps, known as GC programs.
cmd/internal/goobj
Package goobj implements reading of Go object files and archives.
Package goobj implements reading of Go object files and archives.
cmd/internal/objfile
Package objfile implements portable access to OS-specific executable files.
Package objfile implements portable access to OS-specific executable files.
cmd/internal/pprof/commands
Package commands defines and manages the basic pprof commands
Package commands defines and manages the basic pprof commands
cmd/internal/pprof/driver
Package driver implements the core pprof functionality.
Package driver implements the core pprof functionality.
cmd/internal/pprof/fetch
Package fetch provides an extensible mechanism to fetch a profile from a data source.
Package fetch provides an extensible mechanism to fetch a profile from a data source.
cmd/internal/pprof/plugin
Package plugin defines the plugin implementations that the main pprof driver requires.
Package plugin defines the plugin implementations that the main pprof driver requires.
cmd/internal/pprof/profile
Package profile provides a representation of profile.proto and methods to encode/decode profiles in this format.
Package profile provides a representation of profile.proto and methods to encode/decode profiles in this format.
cmd/internal/pprof/report
Package report summarizes a performance profile into a human-readable report.
Package report summarizes a performance profile into a human-readable report.
cmd/internal/pprof/svg
Package svg provides tools related to handling of SVG files
Package svg provides tools related to handling of SVG files
cmd/internal/pprof/symbolizer
Package symbolizer provides a routine to populate a profile with symbol, file and line number information.
Package symbolizer provides a routine to populate a profile with symbol, file and line number information.
cmd/internal/pprof/symbolz
Package symbolz symbolizes a profile using the output from the symbolz service.
Package symbolz symbolizes a profile using the output from the symbolz service.
cmd/internal/pprof/tempfile
Package tempfile provides tools to create and delete temporary files
Package tempfile provides tools to create and delete temporary files
cmd/link
Link, typically invoked as “go tool link,” reads the Go archive or object for a package main, along with its dependencies, and combines them into an executable binary.
Link, typically invoked as “go tool link,” reads the Go archive or object for a package main, along with its dependencies, and combines them into an executable binary.
cmd/nm
Nm lists the symbols defined or used by an object file, archive, or executable.
Nm lists the symbols defined or used by an object file, archive, or executable.
cmd/objdump
Objdump disassembles executable files.
Objdump disassembles executable files.
cmd/pack
Pack is a simple version of the traditional Unix ar tool.
Pack is a simple version of the traditional Unix ar tool.
cmd/pprof
Pprof interprets and displays profiles of Go programs.
Pprof interprets and displays profiles of Go programs.
cmd/trace
Trace is a tool for viewing trace files.
Trace is a tool for viewing trace files.
cmd/vet
Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string.
Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string.
cmd/vet/internal/cfg
This package constructs a simple control-flow graph (CFG) of the statements and expressions within a single function.
This package constructs a simple control-flow graph (CFG) of the statements and expressions within a single function.
cmd/vet/internal/whitelist
Package whitelist defines exceptions for the vet tool.
Package whitelist defines exceptions for the vet tool.
cmd/yacc
Yacc is a version of yacc for Go.
Yacc is a version of yacc for Go.
compress/bzip2
Package bzip2 implements bzip2 decompression.
Package bzip2 implements bzip2 decompression.
compress/flate
Package flate implements the DEFLATE compressed data format, described in RFC 1951.
Package flate implements the DEFLATE compressed data format, described in RFC 1951.
compress/gzip
Package gzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.
Package gzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.
compress/lzw
Package lzw implements the Lempel-Ziv-Welch compressed data format, described in T. A. Welch, “A Technique for High-Performance Data Compression”, Computer, 17(6) (June 1984), pp 8-19.
Package lzw implements the Lempel-Ziv-Welch compressed data format, described in T. A. Welch, “A Technique for High-Performance Data Compression”, Computer, 17(6) (June 1984), pp 8-19.
compress/zlib
Package zlib implements reading and writing of zlib format compressed data, as specified in RFC 1950.
Package zlib implements reading and writing of zlib format compressed data, as specified in RFC 1950.
container/heap
Package heap provides heap operations for any type that implements heap.Interface.
Package heap provides heap operations for any type that implements heap.Interface.
container/list
Package list implements a doubly linked list.
Package list implements a doubly linked list.
container/ring
Package ring implements operations on circular lists.
Package ring implements operations on circular lists.
context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
crypto
Package crypto collects common cryptographic constants.
Package crypto collects common cryptographic constants.
crypto/aes
Package aes implements AES encryption (formerly Rijndael), as defined in U.S. Federal Information Processing Standards Publication 197.
Package aes implements AES encryption (formerly Rijndael), as defined in U.S. Federal Information Processing Standards Publication 197.
crypto/cipher
Package cipher implements standard block cipher modes that can be wrapped around low-level block cipher implementations.
Package cipher implements standard block cipher modes that can be wrapped around low-level block cipher implementations.
crypto/des
Package des implements the Data Encryption Standard (DES) and the Triple Data Encryption Algorithm (TDEA) as defined in U.S. Federal Information Processing Standards Publication 46-3.
Package des implements the Data Encryption Standard (DES) and the Triple Data Encryption Algorithm (TDEA) as defined in U.S. Federal Information Processing Standards Publication 46-3.
crypto/dsa
Package dsa implements the Digital Signature Algorithm, as defined in FIPS 186-3.
Package dsa implements the Digital Signature Algorithm, as defined in FIPS 186-3.
crypto/ecdsa
Package ecdsa implements the Elliptic Curve Digital Signature Algorithm, as defined in FIPS 186-3.
Package ecdsa implements the Elliptic Curve Digital Signature Algorithm, as defined in FIPS 186-3.
crypto/elliptic
Package elliptic implements several standard elliptic curves over prime fields.
Package elliptic implements several standard elliptic curves over prime fields.
crypto/hmac
Package hmac implements the Keyed-Hash Message Authentication Code (HMAC) as defined in U.S. Federal Information Processing Standards Publication 198.
Package hmac implements the Keyed-Hash Message Authentication Code (HMAC) as defined in U.S. Federal Information Processing Standards Publication 198.
crypto/md5
Package md5 implements the MD5 hash algorithm as defined in RFC 1321.
Package md5 implements the MD5 hash algorithm as defined in RFC 1321.
crypto/rand
Package rand implements a cryptographically secure pseudorandom number generator.
Package rand implements a cryptographically secure pseudorandom number generator.
crypto/rc4
Package rc4 implements RC4 encryption, as defined in Bruce Schneier's Applied Cryptography.
Package rc4 implements RC4 encryption, as defined in Bruce Schneier's Applied Cryptography.
crypto/rsa
Package rsa implements RSA encryption as specified in PKCS#1.
Package rsa implements RSA encryption as specified in PKCS#1.
crypto/sha1
Package sha1 implements the SHA1 hash algorithm as defined in RFC 3174.
Package sha1 implements the SHA1 hash algorithm as defined in RFC 3174.
crypto/sha256
Package sha256 implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4.
Package sha256 implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4.
crypto/sha512
Package sha512 implements the SHA-384, SHA-512, SHA-512/224, and SHA-512/256 hash algorithms as defined in FIPS 180-4.
Package sha512 implements the SHA-384, SHA-512, SHA-512/224, and SHA-512/256 hash algorithms as defined in FIPS 180-4.
crypto/subtle
Package subtle implements functions that are often useful in cryptographic code but require careful thought to use correctly.
Package subtle implements functions that are often useful in cryptographic code but require careful thought to use correctly.
crypto/tls
Package tls partially implements TLS 1.2, as specified in RFC 5246.
Package tls partially implements TLS 1.2, as specified in RFC 5246.
crypto/x509
Package x509 parses X.509-encoded keys and certificates.
Package x509 parses X.509-encoded keys and certificates.
crypto/x509/pkix
Package pkix contains shared, low level structures used for ASN.1 parsing and serialization of X.509 certificates, CRL and OCSP.
Package pkix contains shared, low level structures used for ASN.1 parsing and serialization of X.509 certificates, CRL and OCSP.
database/sql
Package sql provides a generic interface around SQL (or SQL-like) databases.
Package sql provides a generic interface around SQL (or SQL-like) databases.
database/sql/driver
Package driver defines interfaces to be implemented by database drivers as used by package sql.
Package driver defines interfaces to be implemented by database drivers as used by package sql.
debug/dwarf
Package dwarf provides access to DWARF debugging information loaded from executable files, as defined in the DWARF 2.0 Standard at http://dwarfstd.org/doc/dwarf-2.0.0.pdf
Package dwarf provides access to DWARF debugging information loaded from executable files, as defined in the DWARF 2.0 Standard at http://dwarfstd.org/doc/dwarf-2.0.0.pdf
debug/elf
Package elf implements access to ELF object files.
Package elf implements access to ELF object files.
debug/gosym
Package gosym implements access to the Go symbol and line number tables embedded in Go binaries generated by the gc compilers.
Package gosym implements access to the Go symbol and line number tables embedded in Go binaries generated by the gc compilers.
debug/macho
Package macho implements access to Mach-O object files.
Package macho implements access to Mach-O object files.
debug/pe
Package pe implements access to PE (Microsoft Windows Portable Executable) files.
Package pe implements access to PE (Microsoft Windows Portable Executable) files.
debug/plan9obj
Package plan9obj implements access to Plan 9 a.out object files.
Package plan9obj implements access to Plan 9 a.out object files.
encoding
Package encoding defines interfaces shared by other packages that convert data to and from byte-level and textual representations.
Package encoding defines interfaces shared by other packages that convert data to and from byte-level and textual representations.
encoding/ascii85
Package ascii85 implements the ascii85 data encoding as used in the btoa tool and Adobe's PostScript and PDF document formats.
Package ascii85 implements the ascii85 data encoding as used in the btoa tool and Adobe's PostScript and PDF document formats.
encoding/asn1
Package asn1 implements parsing of DER-encoded ASN.1 data structures, as defined in ITU-T Rec X.690.
Package asn1 implements parsing of DER-encoded ASN.1 data structures, as defined in ITU-T Rec X.690.
encoding/base32
Package base32 implements base32 encoding as specified by RFC 4648.
Package base32 implements base32 encoding as specified by RFC 4648.
encoding/base64
Package base64 implements base64 encoding as specified by RFC 4648.
Package base64 implements base64 encoding as specified by RFC 4648.
encoding/binary
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.
Package binary implements simple translation between numbers and byte sequences and encoding and decoding of varints.
encoding/csv
Package csv reads and writes comma-separated values (CSV) files.
Package csv reads and writes comma-separated values (CSV) files.
encoding/gob
Package gob manages streams of gobs - binary values exchanged between an Encoder (transmitter) and a Decoder (receiver).
Package gob manages streams of gobs - binary values exchanged between an Encoder (transmitter) and a Decoder (receiver).
encoding/hex
Package hex implements hexadecimal encoding and decoding.
Package hex implements hexadecimal encoding and decoding.
encoding/json
Package json implements encoding and decoding of JSON as defined in RFC 4627.
Package json implements encoding and decoding of JSON as defined in RFC 4627.
encoding/pem
Package pem implements the PEM data encoding, which originated in Privacy Enhanced Mail.
Package pem implements the PEM data encoding, which originated in Privacy Enhanced Mail.
encoding/xml
Package xml implements a simple XML 1.0 parser that understands XML name spaces.
Package xml implements a simple XML 1.0 parser that understands XML name spaces.
errors
Package errors implements functions to manipulate errors.
Package errors implements functions to manipulate errors.
expvar
Package expvar provides a standardized interface to public variables, such as operation counters in servers.
Package expvar provides a standardized interface to public variables, such as operation counters in servers.
flag
Package flag implements command-line flag parsing.
Package flag implements command-line flag parsing.
fmt
Package fmt implements formatted I/O with functions analogous to C's printf and scanf.
Package fmt implements formatted I/O with functions analogous to C's printf and scanf.
go/ast
Package ast declares the types used to represent syntax trees for Go packages.
Package ast declares the types used to represent syntax trees for Go packages.
go/build
Package build gathers information about Go packages.
Package build gathers information about Go packages.
go/constant
Package constant implements Values representing untyped Go constants and their corresponding operations.
Package constant implements Values representing untyped Go constants and their corresponding operations.
go/doc
Package doc extracts source code documentation from a Go AST.
Package doc extracts source code documentation from a Go AST.
go/format
Package format implements standard formatting of Go source.
Package format implements standard formatting of Go source.
go/importer
Package importer provides access to export data importers.
Package importer provides access to export data importers.
go/internal/gccgoimporter
Package gccgoimporter implements Import for gccgo-generated object files.
Package gccgoimporter implements Import for gccgo-generated object files.
go/internal/gcimporter
Package gcimporter implements Import for gc-generated object files.
Package gcimporter implements Import for gc-generated object files.
go/parser
Package parser implements a parser for Go source files.
Package parser implements a parser for Go source files.
go/printer
Package printer implements printing of AST nodes.
Package printer implements printing of AST nodes.
go/scanner
Package scanner implements a scanner for Go source text.
Package scanner implements a scanner for Go source text.
go/token
Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).
Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).
go/types
Package types declares the data types and implements the algorithms for type-checking of Go packages.
Package types declares the data types and implements the algorithms for type-checking of Go packages.
hash
Package hash provides interfaces for hash functions.
Package hash provides interfaces for hash functions.
hash/adler32
Package adler32 implements the Adler-32 checksum.
Package adler32 implements the Adler-32 checksum.
hash/crc32
Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32, checksum.
Package crc32 implements the 32-bit cyclic redundancy check, or CRC-32, checksum.
hash/crc64
Package crc64 implements the 64-bit cyclic redundancy check, or CRC-64, checksum.
Package crc64 implements the 64-bit cyclic redundancy check, or CRC-64, checksum.
hash/fnv
Package fnv implements FNV-1 and FNV-1a, non-cryptographic hash functions created by Glenn Fowler, Landon Curt Noll, and Phong Vo.
Package fnv implements FNV-1 and FNV-1a, non-cryptographic hash functions created by Glenn Fowler, Landon Curt Noll, and Phong Vo.
html
Package html provides functions for escaping and unescaping HTML text.
Package html provides functions for escaping and unescaping HTML text.
html/template
Package template (html/template) implements data-driven templates for generating HTML output safe against code injection.
Package template (html/template) implements data-driven templates for generating HTML output safe against code injection.
image
Package image implements a basic 2-D image library.
Package image implements a basic 2-D image library.
image/color
Package color implements a basic color library.
Package color implements a basic color library.
image/color/palette
Package palette provides standard color palettes.
Package palette provides standard color palettes.
image/draw
Package draw provides image composition functions.
Package draw provides image composition functions.
image/gif
Package gif implements a GIF image decoder and encoder.
Package gif implements a GIF image decoder and encoder.
image/internal/imageutil
Package imageutil contains code shared by image-related packages.
Package imageutil contains code shared by image-related packages.
image/jpeg
Package jpeg implements a JPEG image decoder and encoder.
Package jpeg implements a JPEG image decoder and encoder.
image/png
Package png implements a PNG image decoder and encoder.
Package png implements a PNG image decoder and encoder.
index/suffixarray
Package suffixarray implements substring search in logarithmic time using an in-memory suffix array.
Package suffixarray implements substring search in logarithmic time using an in-memory suffix array.
internal/nettrace
Package nettrace contains internal hooks for tracing activity in the net package.
Package nettrace contains internal hooks for tracing activity in the net package.
internal/race
Package race contains helper functions for manually instrumenting code for the race detector.
Package race contains helper functions for manually instrumenting code for the race detector.
internal/singleflight
Package singleflight provides a duplicate function call suppression mechanism.
Package singleflight provides a duplicate function call suppression mechanism.
internal/syscall/windows/registry
Package registry provides access to the Windows registry.
Package registry provides access to the Windows registry.
internal/syscall/windows/sysdll
Package sysdll is an internal leaf package that records and reports which Windows DLL names are used by Go itself.
Package sysdll is an internal leaf package that records and reports which Windows DLL names are used by Go itself.
internal/testenv
Package testenv provides information about what functionality is available in different testing environments run by the Go team.
Package testenv provides information about what functionality is available in different testing environments run by the Go team.
io
Package io provides basic interfaces to I/O primitives.
Package io provides basic interfaces to I/O primitives.
io/ioutil
Package ioutil implements some I/O utility functions.
Package ioutil implements some I/O utility functions.
log
Package log implements a simple logging package.
Package log implements a simple logging package.
log/syslog
Package syslog provides a simple interface to the system log service.
Package syslog provides a simple interface to the system log service.
math
Package math provides basic constants and mathematical functions.
Package math provides basic constants and mathematical functions.
math/big
Package big implements arbitrary-precision arithmetic (big numbers).
Package big implements arbitrary-precision arithmetic (big numbers).
math/cmplx
Package cmplx provides basic constants and mathematical functions for complex numbers.
Package cmplx provides basic constants and mathematical functions for complex numbers.
math/rand
Package rand implements pseudo-random number generators.
Package rand implements pseudo-random number generators.
mime
Package mime implements parts of the MIME spec.
Package mime implements parts of the MIME spec.
mime/multipart
Package multipart implements MIME multipart parsing, as defined in RFC 2046.
Package multipart implements MIME multipart parsing, as defined in RFC 2046.
mime/quotedprintable
Package quotedprintable implements quoted-printable encoding as specified by RFC 2045.
Package quotedprintable implements quoted-printable encoding as specified by RFC 2045.
net
Package net provides a portable interface for network I/O, including TCP/IP, UDP, domain name resolution, and Unix domain sockets.
Package net provides a portable interface for network I/O, including TCP/IP, UDP, domain name resolution, and Unix domain sockets.
net/http
Package http provides HTTP client and server implementations.
Package http provides HTTP client and server implementations.
net/http/cgi
Package cgi implements CGI (Common Gateway Interface) as specified in RFC 3875.
Package cgi implements CGI (Common Gateway Interface) as specified in RFC 3875.
net/http/cookiejar
Package cookiejar implements an in-memory RFC 6265-compliant http.CookieJar.
Package cookiejar implements an in-memory RFC 6265-compliant http.CookieJar.
net/http/fcgi
Package fcgi implements the FastCGI protocol.
Package fcgi implements the FastCGI protocol.
net/http/httptest
Package httptest provides utilities for HTTP testing.
Package httptest provides utilities for HTTP testing.
net/http/httptrace
Package httptrace provides mechanisms to trace the events within HTTP client requests.
Package httptrace provides mechanisms to trace the events within HTTP client requests.
net/http/httputil
Package httputil provides HTTP utility functions, complementing the more common ones in the net/http package.
Package httputil provides HTTP utility functions, complementing the more common ones in the net/http package.
net/http/internal
Package internal contains HTTP internals shared by net/http and net/http/httputil.
Package internal contains HTTP internals shared by net/http and net/http/httputil.
net/http/pprof
Package pprof serves via its HTTP server runtime profiling data in the format expected by the pprof visualization tool.
Package pprof serves via its HTTP server runtime profiling data in the format expected by the pprof visualization tool.
net/internal/socktest
Package socktest provides utilities for socket testing.
Package socktest provides utilities for socket testing.
net/mail
Package mail implements parsing of mail messages.
Package mail implements parsing of mail messages.
net/rpc
Package rpc provides access to the exported methods of an object across a network or other I/O connection.
Package rpc provides access to the exported methods of an object across a network or other I/O connection.
net/rpc/jsonrpc
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpc package.
Package jsonrpc implements a JSON-RPC ClientCodec and ServerCodec for the rpc package.
net/smtp
Package smtp implements the Simple Mail Transfer Protocol as defined in RFC 5321.
Package smtp implements the Simple Mail Transfer Protocol as defined in RFC 5321.
net/textproto
Package textproto implements generic support for text-based request/response protocols in the style of HTTP, NNTP, and SMTP.
Package textproto implements generic support for text-based request/response protocols in the style of HTTP, NNTP, and SMTP.
net/url
Package url parses URLs and implements query escaping.
Package url parses URLs and implements query escaping.
os
Package os provides a platform-independent interface to operating system functionality.
Package os provides a platform-independent interface to operating system functionality.
os/exec
Package exec runs external commands.
Package exec runs external commands.
os/signal
Package signal implements access to incoming signals.
Package signal implements access to incoming signals.
os/user
Package user allows user account lookups by name or id.
Package user allows user account lookups by name or id.
path
Package path implements utility routines for manipulating slash-separated paths.
Package path implements utility routines for manipulating slash-separated paths.
path/filepath
Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.
Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.
reflect
Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types.
regexp
Package regexp implements regular expression search.
Package regexp implements regular expression search.
regexp/syntax
Package syntax parses regular expressions into parse trees and compiles parse trees into programs.
Package syntax parses regular expressions into parse trees and compiles parse trees into programs.
runtime
Package runtime contains operations that interact with Go's runtime system, such as functions to control goroutines.
Package runtime contains operations that interact with Go's runtime system, such as functions to control goroutines.
runtime/cgo
Package cgo contains runtime support for code generated by the cgo tool.
Package cgo contains runtime support for code generated by the cgo tool.
runtime/debug
Package debug contains facilities for programs to debug themselves while they are running.
Package debug contains facilities for programs to debug themselves while they are running.
runtime/internal/sys
package sys contains system- and configuration- and architecture-specific constants used by the runtime.
package sys contains system- and configuration- and architecture-specific constants used by the runtime.
runtime/pprof
Package pprof writes runtime profiling data in the format expected by the pprof visualization tool.
Package pprof writes runtime profiling data in the format expected by the pprof visualization tool.
runtime/race
Package race implements data race detection logic.
Package race implements data race detection logic.
runtime/trace
Go execution tracer.
Go execution tracer.
sort
Package sort provides primitives for sorting slices and user-defined collections.
Package sort provides primitives for sorting slices and user-defined collections.
strconv
Package strconv implements conversions to and from string representations of basic data types.
Package strconv implements conversions to and from string representations of basic data types.
strings
Package strings implements simple functions to manipulate UTF-8 encoded strings.
Package strings implements simple functions to manipulate UTF-8 encoded strings.
sync
Package sync provides basic synchronization primitives such as mutual exclusion locks.
Package sync provides basic synchronization primitives such as mutual exclusion locks.
sync/atomic
Package atomic provides low-level atomic memory primitives useful for implementing synchronization algorithms.
Package atomic provides low-level atomic memory primitives useful for implementing synchronization algorithms.
syscall
Package syscall contains an interface to the low-level operating system primitives.
Package syscall contains an interface to the low-level operating system primitives.
testing
Package testing provides support for automated testing of Go packages.
Package testing provides support for automated testing of Go packages.
testing/iotest
Package iotest implements Readers and Writers useful mainly for testing.
Package iotest implements Readers and Writers useful mainly for testing.
testing/quick
Package quick implements utility functions to help with black box testing.
Package quick implements utility functions to help with black box testing.
text/scanner
Package scanner provides a scanner and tokenizer for UTF-8-encoded text.
Package scanner provides a scanner and tokenizer for UTF-8-encoded text.
text/tabwriter
Package tabwriter implements a write filter (tabwriter.Writer) that translates tabbed columns in input into properly aligned text.
Package tabwriter implements a write filter (tabwriter.Writer) that translates tabbed columns in input into properly aligned text.
text/template
Package template implements data-driven templates for generating textual output.
Package template implements data-driven templates for generating textual output.
text/template/parse
Package parse builds parse trees for templates as defined by text/template and html/template.
Package parse builds parse trees for templates as defined by text/template and html/template.
time
Package time provides functionality for measuring and displaying time.
Package time provides functionality for measuring and displaying time.
unicode
Package unicode provides data and functions to test some properties of Unicode code points.
Package unicode provides data and functions to test some properties of Unicode code points.
unicode/utf16
Package utf16 implements encoding and decoding of UTF-16 sequences.
Package utf16 implements encoding and decoding of UTF-16 sequences.
unicode/utf8
Package utf8 implements functions and constants to support text encoded in UTF-8.
Package utf8 implements functions and constants to support text encoded in UTF-8.
unsafe
Package unsafe contains operations that step around the type safety of Go programs.
Package unsafe contains operations that step around the type safety of Go programs.
test
ken
stress
The runstress tool stresses the runtime.
The runstress tool stresses the runtime.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL