generator

Overview ¶

The code generator for the plugin for the Google protocol buffer compiler.
It generates Go code from the protocol buffer description files read by the
main routine.

The marshalto generates a Marshal and MarshalTo method for each message. The `Marshal() ([]byte, error)` method results in the fact that the message implements the Marshaler interface. This allows proto.Marshal to be faster by calling the generated Marshal method rather than using reflect to Marshal the struct.

The generation of marshalling tests are enabled using one of the following extensions:

• testgen
• testgen_all

And benchmarks given it is enabled using one of the following extensions:

• benchgen
• benchgen_all

Let us look at:

code.google.com/p/gogoprotobuf/test/example/example.proto

Btw all the output can be seen at:

code.google.com/p/gogoprotobuf/test/example/*

The following message:

  message B {
	option (gogoproto.description) = true;
	optional string A = 1 [(gogoproto.embed) = true];
	repeated int64 G = 2 [(gogoproto.customtype) = "github.com/dropbox/goprotoc/test.Id"];
  }

the marshalto code will generate the following code:

func (m *B) Marshal() (data []byte, err error) {
	size := m.Size()
	data = make([]byte, size)
	n, err := m.MarshalToUsingCachedSize(data)
	if err != nil {
		return nil, err
	}
	return data[:n], nil
}

func (m *B) MarshalTo(data []byte) (n int, err error) {
	m.Size()
	return m.MarshalToUsingCachedSize(data)
}

func (m *B) MarshalToUsingCachedSize(data []byte) (n int, err error) {
	var i int
	_ = i
	var l int
	_ = l
	if m.xxx_IsASet {
		data[i] = 0xa
		i++
		i = encodeVarintCustom(data, i, uint64(len(m.a)))
		i += copy(data[i:], m.a)
	}
	if m.xxx_LenG > 0 {
		for idx := 0; idx < m.xxx_LenG; idx++ {
			num := m.g[idx]
			data[i] = 0x10
			i++
			i = encodeVarintCustom(data, i, uint64(num))
		}
	}
	if m.XXX_unrecognized != nil {
		i += copy(data[i:], m.XXX_unrecognized)
	}
	return i, nil
}

As shown above Marshal calculates the size of the not yet marshalled message and allocates the appropriate buffer. This is followed by calling the MarshalTo method which requires a preallocated buffer. The MarshalTo method allows a user to rather preallocated a reusable buffer.

The Size method is generated using the size plugin and the gogoproto.sizer, gogoproto.sizer_all extensions. The user can also using the generated Size method to check that his reusable buffer is still big enough.

The generated tests and benchmarks will keep you safe and show that this is really a significant speed improvement.

This file generates a Size method for each message. This is useful with the MarshalTo method generated by the marshalto

The size code also generates a test given it is enabled using one of the following extensions:

• testgen
• testgen_all

And a benchmark given it is enabled using one of the following extensions:

• benchgen
• benchgen_all

Let us look at:

code.google.com/p/gogoprotobuf/test/example/example.proto

Btw all the output can be seen at:

code.google.com/p/gogoprotobuf/test/example/*

The following message:

  message B {
	option (gogoproto.description) = true;
	optional string A = 1 [(gogoproto.embed) = true];
	repeated int64 G = 2 [(gogoproto.customtype) = "github.com/dropbox/goprotoc/test.Id"];
  }

will generate the following code:

  func (m *B) Size() (n int) {
	var l int
	_ = l
	if m.xxx_IsASet {
		l = len(m.a)
		n += 1 + l + sovCustom(uint64(l))
	}
	if m.xxx_LenG > 0 {
		for i := 0; i < m.xxx_LenG; i++ {
			e := m.g[i]
			n += 1 + sovCustom(uint64(e))
		}
	}
	if m.XXX_unrecognized != nil {
		n += len(m.XXX_unrecognized)
	}
	m.xxx_sizeCached = n
	return n
  }

and the following test code:

  func TestBSize(t *testing5.T) {
	popr := math_rand5.New(math_rand5.NewSource(time5.Now().UnixNano()))
	p := NewPopulatedB(popr, true)
	data, err := dropbox_gogoprotobuf_proto2.Marshal(p)
	if err != nil {
	  panic(err)
	}
	size := g.Size()
	if len(data) != size {
	  t.Fatalf("size %v != marshalled size %v", size, len(data))
	}
  }

  func BenchmarkBSize(b *testing5.B) {
	popr := math_rand5.New(math_rand5.NewSource(616))
	total := 0
	pops := make([]*B, 1000)
	for i := 0; i < 1000; i++ {
	  pops[i] = NewPopulatedB(popr, false)
	}
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
	  total += pops[i%1000].Size()
	}
	b.SetBytes(int64(total / b.N))
  }

The sovExample function is a size of varint function for the example.pb.go file.

The unmarshal code generates a Unmarshal method for each message. The `Unmarshal([]byte) error` method results in the fact that the message implements the Unmarshaler interface. The allows proto.Unmarshal to be faster by calling the generated Unmarshal method rather than using reflect.

The generation of unmarshalling tests are enabled using one of the following extensions:

• testgen
• testgen_all

And benchmarks given it is enabled using one of the following extensions:

• benchgen
• benchgen_all

Let us look at:

code.google.com/p/gogoprotobuf/test/example/example.proto

Btw all the output can be seen at:

code.google.com/p/gogoprotobuf/test/example/*

The following message:

  option (gogoproto.unmarshaler_all) = true;

  message B {
	option (gogoproto.description) = true;
	optional string A = 1 [(gogoproto.embed) = true];
	repeated int64 G = 2 [(gogoproto.customtype) = "github.com/dropbox/goprotoc/test.Id"];
  }

the unmarshal will generate the following code:

func (m *B) Unmarshal(data []byte) error {
	l := len(data)
	index := 0
	for index < l {
		var wire uint64
		for shift := uint(0); ; shift += 7 {
			if index >= l {
				return io.ErrUnexpectedEOF
			}
			b := data[index]
			index++
			wire |= (uint64(b) & 0x7F) << shift
			if b < 0x80 {
				break
			}
		}
		fieldNum := int32(wire >> 3)
		wireType := int(wire & 0x7)
		switch fieldNum {
		case 1:
			if wireType != 2 {
				return fmt.Errorf("proto: wrong wireType = %d for field a", wireType)
			}
			m.xxx_IsASet = true
			var stringLen uint64
			for shift := uint(0); ; shift += 7 {
				if index >= l {
					return io.ErrUnexpectedEOF
				}
				b := data[index]
				index++
				stringLen |= (uint64(b) & 0x7F) << shift
				if b < 0x80 {
					break
				}
			}
			postIndex := index + int(stringLen)
			if postIndex > l {
				return io.ErrUnexpectedEOF
			}
			m.a = string(data[index:postIndex])
			index = postIndex
		case 2:
			if wireType != 0 {
				return fmt.Errorf("proto: wrong wireType = %d for field g", wireType)
			}
			m.xxx_LenG += 1
			var v int64
			for shift := uint(0); ; shift += 7 {
				if index >= l {
					return io.ErrUnexpectedEOF
				}
				b := data[index]
				index++
				v |= (int64(b) & 0x7F) << shift
				if b < 0x80 {
					break
				}
			}
			m.g = append(m.g, github_com_dropbox_goprotoc_test.Id(v))
		default:
			var sizeOfWire int
			for {
				sizeOfWire++
				wire >>= 7
				if wire == 0 {
					break
				}
			}
			index -= sizeOfWire
			skippy, err := proto.Skip(data[index:])
			if err != nil {
				return err
			}
			if (index + skippy) > l {
				return io.ErrUnexpectedEOF
			}
			m.XXX_unrecognized = append(m.XXX_unrecognized, data[index:index+skippy]...)
			index += skippy
		}
	}
	return nil
}

Remember when using this code to call proto.Unmarshal. This will call m.Reset and invoke the generated Unmarshal method for you. If you call m.Unmarshal without m.Reset you could be merging protocol buffers.