iconv

package module
v0.0.0-...-9b0f31d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2020 License: BSD-2-Clause Imports: 4 Imported by: 0

README

Install

The main method of installation is through "go get" (provided in $GOROOT/bin)

go get github.com/djimenez/iconv-go

This both downloads from github and installs the package into your $GOPATH. To just recompile the package after you've already "get"ed (e.g. for a new go version), use "go install" instead.

go install github.com/djimenez/iconv-go

See documentation for "go get" and "go install" for more information.

PLEASE NOTE that this package requires the use of cgo, since it is only a wrapper around iconv - either provided by libiconv or glibc on your system. Attempts to build without cgo enabled will fail.

Usage

To use the package, you'll need the appropriate import statement:

import (
	iconv "github.com/djimenez/iconv-go"
)

Converting string Values

Converting a string can be done with two methods. First, there's iconv.ConvertString(input, fromEncoding, toEncoding string)

output,_ := iconv.ConvertString("Hello World!", "utf-8", "windows-1252")

Alternatively, you can create a converter and use its ConvertString method. Reuse of a Converter instance is recommended when doing many string conversions between the same encodings.

converter := iconv.NewConverter("utf-8", "windows-1252")
output,_ := converter.ConvertString("Hello World!")

// converter can then be closed explicitly
// this will also happen when garbage collected
converter.Close()

ConvertString may return errors for the following reasons:

  • EINVAL - when either the from or to encoding is not supported by iconv
  • EILSEQ - when the input string contains an invalid byte sequence for the given from encoding

Converting []byte Values

Converting a []byte can similarly be done with two methods. First, there's iconv.Convert(input, output []byte, fromEncoding, toEncoding string). You'll immediately notice this requires you to give it both the input and output buffer. Ideally, the output buffer should be sized so that it can hold all converted bytes from input, but if it cannot, then Convert will put as many bytes as it can into the buffer without creating an invalid sequence. For example, if iconv only has a single byte left in the output buffer but needs 2 or more for the complete character in a multibyte encoding it will stop writing to the buffer and return with an iconv.E2BIG error.

in := []byte("Hello World!")
out := make([]byte, len(input))

bytesRead, bytesWritten, err := iconv.Convert(in, out, "utf-8", "latin1")

Just like with ConvertString, there is also a Convert method on Converter that can be used.

...
converter := iconv.NewConverter("utf-8", "windows-1252")

bytesRead, bytesWritten, error := converter.Convert(input, output)

Convert may return errors for the following reasons:

  • EINVAL - when either the from or to encoding is not supported by iconv

  • EILSEQ - when the input string contains an invalid byte sequence for the given from encoding

  • E2BIG - when the output buffer is not big enough to hold the full conversion of input

    Note on E2BIG: this is a common error value especially when converting to a multibyte encoding and should not be considered fatal. Partial conversion has probably occurred be sure to check bytesRead and bytesWritten.

Note on Shift Based Encodings

When using iconv.Convert convenience method it will automatically try to append to your output buffer with a nil input so that any end shift sequences are appropiately written. Using a Converter.Convert method however will not automatically do this since it can be used to process a full stream in chunks. So you'll need to remember to pass a nil input buffer at the end yourself, just like you would with direct iconv usage.

Converting an *io.Reader

The iconv.Reader allows any other *io.Reader to be wrapped and have its bytes transcoded as they are read.

// We're wrapping stdin for simplicity, but a File or network reader could
// be wrapped as well
reader,_ := iconv.NewReader(os.Stdin, "utf-8", "windows-1252")

Converting an *io.Writer

The iconv.Writer allows any other *io.Writer to be wrapped and have its bytes transcoded as they are written.

// We're wrapping stdout for simplicity, but a File or network reader could
// be wrapped as well
writer,_ := iconv.NewWriter(os.Stdout, "utf-8", "windows-1252")

Documentation

Overview

Wraps the iconv API present on most systems, which allows for conversion of bytes from one encoding to another. This package additionally provides some convenient interface implementations like a Reader and Writer.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Convert

func Convert(input []byte, output []byte, fromEncoding string, toEncoding string) (bytesRead int, bytesWritten int, err error)

All in one Convert method, rather than requiring the construction of an iconv.Converter

func ConvertString

func ConvertString(input string, fromEncoding string, toEncoding string) (output string, err error)

All in one ConvertString method, rather than requiring the construction of an iconv.Converter

Types

type Converter

type Converter struct {
	// contains filtered or unexported fields
}

func NewConverter

func NewConverter(fromEncoding string, toEncoding string) (converter *Converter, err error)

Initialize a new Converter. If fromEncoding or toEncoding are not supported by iconv then an EINVAL error will be returned. An ENOMEM error maybe returned if there is not enough memory to initialize an iconv descriptor

func (*Converter) Close

func (this *Converter) Close() (err error)

Close a Converter's iconv description explicitly

func (*Converter) Convert

func (this *Converter) Convert(input []byte, output []byte) (bytesRead int, bytesWritten int, err error)

Convert bytes from an input byte slice into a give output byte slice

As many bytes that can converted and fit into the size of output will be processed and the number of bytes read for input as well as the number of bytes written to output will be returned. If not all converted bytes can fit into output and E2BIG error will also be returned. If input contains an invalid sequence of bytes for the Converter's fromEncoding an EILSEQ error will be returned

For shift based output encodings, any end shift byte sequences can be generated by passing a 0 length byte slice as input. Also passing a 0 length byte slice for output will simply reset the iconv descriptor shift state without writing any bytes.

func (*Converter) ConvertString

func (this *Converter) ConvertString(input string) (output string, err error)

Convert an input string

EILSEQ error may be returned if input contains invalid bytes for the Converter's fromEncoding.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(source io.Reader, fromEncoding string, toEncoding string) (*Reader, error)

func NewReaderFromConverter

func NewReaderFromConverter(source io.Reader, converter *Converter) (reader *Reader)

func (*Reader) Close

func (this *Reader) Close()

Close close this reader, solve memory leak

func (*Reader) Read

func (this *Reader) Read(p []byte) (n int, err error)

implement the io.Reader interface

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter

func NewWriter(destination io.Writer, fromEncoding string, toEncoding string) (*Writer, error)

func NewWriterFromConverter

func NewWriterFromConverter(destination io.Writer, converter *Converter) (writer *Writer)

func (*Writer) Write

func (this *Writer) Write(p []byte) (n int, err error)

implement the io.Writer interface

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL