README
¶
SQS to Kafka forwarder 
Fetches messages from SQS and forwards them to Kafka.
Build
Given a properly setup go develop environment, just run
go get -d -t .
go build
The resulting binary should be named sqs-to-kafka (or sqs-to-kafka.exe
on Windows). For further detail, e.g. cross-platform builds and custom build
parameters, please refer to the go-documentation.
Usage
sqs-to-kafka takes the following commandline parameters:
--aws-access-key=STRING: AWS access key--aws-secret-key=STRING: AWS secret key--aws-region=STRING: AWS region--aws-profile=STRING: AWS profile--aws-read-config: read AWS configuration from~/.aws/config--aws-endpoint=STRING: URL of the AWS endpoint--sqs-url=STRING: URL of the SQS queue for incomming messages--kafka-brokers=STRING: list of Kafka brokers used for bootstrapping--kafka-topic=STRING: Kafka topic for outgoing messages--metrics-address=HOST:PORT: Listening address to serve metrics
sqs-url, kafka-brokers and kafka-topic are mandatory, everything else is
optional. When metrics-address is given sqs-to-kafka binds that address to
export prometheus compatible metrics. Parameters may set via environment
variables as well
AWS_ACCESS_KEY_ID: AWS access keyAWS_SECRET_ACCESS_KEY: AWS secret keyAWS_REGION: AWS regionAWS_ENDPOINT: URL of the AWS endpointAWS_PROFILE: AWS profileAWS_SDK_LOAD_CONFIG: if set to1read AWS configuration from~/.aws/configSQS_URL: URL of the SQS queue for incomming messagesKAFKA_BROKERS: list of Kafka brokers used for bootstrappingKAFKA_TOPIC: Kafka topic for outgoing messagesMETRICS_ADDRESS: Listening address to serve metrics
When both are specified, commandline parameters take precedence over environment variables.
In addition to that there a couple of AWS SDK specific parameters may be passed via environment variables, see the AWS SDK documentation for a complete list.
Docker
A docker-ized variant is available here meteogroup/sqs-to-kafka. Metrics are
exposed on port 8080. To run use
docker run -P meteogroup/sqs-to-kafka <additional commandline arguments>
License
Copyright © 2017 MeteoGroup Deutschland GmbH, all the files in this repository are released under the terms of Apache License 2.0.
Documentation
¶
Overview ¶
Copyright © 2017 MeteoGroup Deutschland GmbH
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2017 MeteoGroup Deutschland GmbH ¶
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2017 MeteoGroup Deutschland GmbH ¶
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2017 MeteoGroup Deutschland GmbH ¶
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2017 MeteoGroup Deutschland GmbH ¶
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2017 MeteoGroup Deutschland GmbH ¶
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Source Files