feedpushr

feedpushr

A simple feed aggregator service with sugar on top.

Features

• Single executable with an embedded database.
• Manage feed subscriptions.
• Import/Export feed subscriptions with OPML files.
• Aggressive and tunable aggregation process.
• Manage feed aggregation individually.
• Apply modifications on articles with a pluggable filter system.
• Push new articles to a pluggable output system (STDOUT, HTTP, Twitter ...).
• Customize the pipeline thanks to a powerful expression language.
• Support of PubSubHubbud the open, simple, web-scale and decentralized pubsub protocol.
• REST API with complete OpenAPI documentation.
• Full feature Web UI and CLI to interact with the API.
• Metrics production for monitoring.

Installation

Run the following command:

$ go get -v github.com/ncarlier/feedpushr/v2

Or download the binary regarding your architecture:

$ curl -s https://raw.githubusercontent.com/ncarlier/feedpushr/master/install.sh | bash

Or use Docker:

$ docker run -d --name=feedpushr ncarlier/feedpushr

Configuration

Feedpushr can be configured by using command line parameters or by setting environment variables.

Type feedpushr -h to display all parameters and related environment variables.

All configuration variables are described in etc/default/feedpushr.env file.

Tags

You can define tags on feeds using the Web UI or the API:

$ curl -XPOST http://localhost:8080/v1/feeds?url=http://www.hashicorp.com/feed.xml&tags=foo,bar

Tags can also be imported/exported in OPML format. When using OMPL, tags are stored into the category attribute. OPML category is a string of comma-separated slash-delimited category strings. For example, this OMPL attribute <category>/test,foo,/bar/bar</category> will be converted to the following tag list: test, foo, bar_bar.

Once feeds are configured with tags, each new article will inherit these tags and be pushed out with them.

Filters

Before being sent, articles can be modified through a filter chain.

Currently, there are some built-in filter:

Filters can be extended using plugins.

Conditional expression

Filters are activated according to a conditional expression. If the filter has no condition it will be activated regardless of the input article. Otherwise the expression will be applied to the article and its result will decide whether the filter is activated or not.

For example, if we want to activate a filter only on articles tagged news and mentioning Paris in their title. We can use the following expression: "news" in Tags and Title contains "Paris"

The conditional expression language is documented here.

Outputs

New articles are sent to outputs.

Currently, there are some built-in output providers:

Outputs can be extended using plugins.

Like filters, outputs are activated according to a conditional expression.

Output format

The output format for stdout and http providers is configurable.

If the format is not specified, the output will be formatted as the following JSON document:

{
	"title": "Article title",
	"text": "Article text description",
	"content": "Article HTML content",
	"link": "Article URL",
	"updated": "Article update date (String format)",
	"updatedParsed": "Article update date (Date format)",
	"published": "Article publication date (String format)",
	"publishedParsed": "Article publication date (Date format)",
	"guid": "Article feed GUID",
	"meta": {
		"key": "Metadata keys and values added by filters"
	},
	"tags": ["list", "of", "tags"]
}

If, on the other hand, you specify a format, it must comply with the Golang template syntax.

Basically, you have access to the JSON properties above but prefixed with a dot and a capital letter all surrounded by 2 brackets. For example, the title property is accessible with the syntax : {{.Title}}.

Let's say you want to send a JSON payload to a Mattermost Webhook. You can format the payload like this:

{
	"text": ":tada: {{.Title}} (<{{.Link}}|more>) cc @all",
}

Plugins

You can easily extend the application by adding plugins.

A plugin is a compiled library file that must be loaded when the application starts.

Plugins inside $PWD are automatically loaded. You can also load a plugin using the --plugin parameter.

Example:

$ feedpushr --plugin ./feedpushr-twitter.so

You can find some external plugins (such as for Twitter) into this directory.

Launcher

Feedpushr can be started in "desktop mode" thanks to a launcher. The purpose of feedpushr-launcher is to start the daemon and add an icon to your taskbar. This icon allows you to control the daemon and quickly access the user interface.

User Interface

You can access Web UI on http://localhost:8080/ui

Authentication

You can restrict access to Feedpushr using HTTP basic authentication.

To activate basic authentication, you have to create a htpasswd file:

$ # create passwd file the user 'admin'
$ htpasswd -B -c .htpasswd admin

This command will ask for a password and store it in the htpawsswd file.

Please note that by default, Feedpushr will try to load the .htpasswd file.

But you can override this behavior by specifying the location of the file:

$ export FP_PASSWD_FILE=/etc/feedpushr.htpasswd
$ # or
$ feedpushr --passwd-file /etc/webhookd/users.htpasswd

Use cases

Start the service

$ # Start service with default configuration:
$ feedpushr
$ # Start service with custom configuration:
$ export FP_DB="boltdb:///var/opt/feedpushr.db"
$ export FP_DELAY=20s
$ export FP_LOG_LEVEL=warn
$ feedpushr

Add feeds

$ # Add feed with the CLI
$ feedpushr-ctl create feed --url http://www.hashicorp.com/feed.xml
$ # Add feed with cURL
$ curl -XPOST http://localhost:8080/v1/feeds?url=http://www.hashicorp.com/feed.xml
$ # Import feeds from an OPML file
$ curl -XPOST http://localhost:8080/v1/opml -F"file=@subscriptions.opml"

Manage feeds

$ # List feeds
$ feedpushr-ctl list feed
$ # Get a feed
$ feedpushr-ctl get feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Remove a feed
$ feedpushr-ctl delete feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Stop aggregation of a feed
$ feedpushr-ctl stop feed --id=9090dfac0ccede1cfcee186826d0cc0d
$ # Start aggregation of a feed
$ feedpushr-ctl start feed --id=9090dfac0ccede1cfcee186826d0cc0d

Misc

$ # Get OpenAPI JSON
$ curl  http://localhost:8080/swagger.json
$ # Get runtime vars
$ curl  http://localhost:8080/v1/vars
$ # Here a quick ETL shell pipeline:
$ # Send transformed articles to HTTP endpoint using shell tools (jq and httpie)
$ feedpushr \
  | jq -c "select(.title) | {title:.title, content:.description, origin: .link}" \
  | while read next; do echo "$next" | http http://postb.in/b/i1J32KdO; done

For development

To be able to build the project you will need to:

• Install makefiles external helpers:

  $ git submodule init
  $ git submodule update
  

• Install goa:

  $ go get -u github.com/goadesign/goa/...@v1
  

Then you can build the project using make:

$ make

Type make help to see other possibilities.

License

GNU General Public License v3.0

See LICENSE to see the full text.