icarus

package module
v0.0.0-...-c4c1cbe Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 9, 2016 License: MIT Imports: 17 Imported by: 0

README

Icarus is a minimalistic blog platform for people who want to write Markdown or HTML documents in Git and have them served by a minimalistic server which still provides the basic nice-to-haves: trending content, full-text search and discovering similar content, while comfortably running on a small VPS.

It relies on quite a few opensource projects: Go, Redis, Bleve, Bootstrap 3, and many others.

Installation

First, you need to install some system dependencies, like Glide, and Redis. On OSX, that might look like:

brew install glide
brew install redis

Then you need to install further dependencies via Glide:

glide install

Then build or install the commands:

make build

or

make install

At which point the binaries will either be in cmd/icarus and cmd/icontent or will be in $GOPATH/bin/.

Configuring

Icarus is configured via a JSON configuration file, an example of which is available at config.json, and supporting these parameters:

{
  "server": {
    "loc": "127.0.0.1:8080",
    "proto": "http",
    "domain": "yourblog.com"
  },
  "rss": {
    "path": "/feeds/",
    "title": "Recent Pages"
  },
  "blog": {
    "name": "Your Blog",
    "results_per_page": 10,
    "pages_in_paginator": 10,
    "template_dir": "templates/",
    "static_dir": "static/"
  },
  "redis": {
    "loc": "localhost:6379"
  }
}

You'll certainly want to change server.domain and blog.name, and the easiest way to customize Icarus without changing the code is to supply your own templates in templates/ and your own static assets in static/ (you'll specify which files are used from static/ by customizing your templates).

By convention, you'll probably want to symlink your pages' static assets into static/ to allow the Go file server to serve your static assets, something along the lines of:

ln -s /Users/will/git/irrational_exuberance/static/ `pwd`/static/blog/

And you should be good to go.

Running

Once you've followed the Installation steps, you should be able to get things running via one of these options:

# if you did make install, recommended
$GOPATH/bin/icarus
# if you did make build
./cmd/icarus/icarus
# if you are making changes during development
make icarus

You can also pass in --config path/to/config.json if you're not running it from the icarus.git repository (or want to specify a different configuration file).

Adding Pages

Each article is either a Markdown or an HTML file (indicated via a trailing .md or .html respectively), but starting with a modified JSON blob:

"title": "This is my title",
"summary": "This is an exciting article about...",
"pub_date": 1184450111,
"slug": "a-unique-slug",
"tags": ["python", "programming"],
"draft": true,


Start writing your article here. Just make sure
you have an empty line after your JSON ends.
The trailing comma is optional.

The supported parameters are:

  • title is the human readable title for your page,
  • summary is the human readable description paragraph for a page,
  • pub_date is an optional timestamp for publishing date, defaults to time it is first sync'd,
  • slug is a unique URL component, such that /<slug>/ is the canonical URL for a page,
  • tags is a list of strings, for tags this page will be added to (tags are also used for calculating related/similar pages),
  • draft default to false and is optional, this governs if your page is included in analytics and the various article lists (e.g. a draft is only accessible if you type in its slug by hand, they are not discoverable).

From there you use the icontent tool to load the content:

$GOPATH/bin/icarus --config path/to/config.json blog/*.md
$GOPATH/bin/icarus --config path/to/config.json blog/*.html    

And you're done.

If you want to unpublish a piece of content, the easiest solution right now is to mark "draft": true in the page's configuration and it will be removed from all indexes.

Conceivably we might want to add a iremove command at some point to truly remove content, or you could just use "draft": true to remove the indexes and then do redis-cli REM page.<post-slug> if you like living on the edge!

History

For reasons which are hard to explain, I've spent a lot of time over the past decade building mediocre blog platforms, and Icarus is the next in that glorious heritage.

Icarus is a Go reimagining of Sisyphus, which was my second personal blogging platform, inspired by the many, many mistakes I made in my first generation "Lifeflow" blog and also by ideas around real-time analytics and content suggestions/ranking that came from working at Digg.

For the third generation, looking to explore some additional ideas:

  1. Writing it in Go and avoiding a heavy-weight framework like Django.
  2. Try to make it actually usable by someone other than myself, mostly as an exercise and not because I anticipate much adoption.
  3. Keep using Bootstrap, maybe Bootstrap 4.
  4. Be faster than 60ms to load the front page. Haven't profiled this in a very long time, but I'd guess Redis lookups are responsible for most of that delay (~85 Redis lookups for the frontpage to load). So... let's try to hit ~5 ms, which probably means keeping everything in-memory (and also updating analytics out-of-band, which will be easy in Go). We can use Redis Pub-Sub to invalid the cache if we're running with more than one instance.
  5. Disqus comments are mostly just ads for me, and should either be removed entirely or replaced with something better / different.
  6. Move away from Python-only Whoosh for search.
  7. Use rrssb for better, simpler sharing to social sites.

Documentation

Overview

Configuration for Icarus instance.

Utilities for preprocessing Markdown from the Python style to the new, different style...

Specifically:

1. strip out [TOC] since there is no support here, 2. call out to pygments to render code blocks starting with :::<lang>

TODO: move this into its own module w/its own namespace

Handling templates and such.

Index

Constants

View Source
const AnalyticsBackoff = "analytics.backoff.%v"
View Source
const DaySeconds = 60 * 60 * 24
View Source
const DirectReferrer = "DIRECT"
View Source
const HistoricalReferrer = "imported from Google Analytics"
View Source
const PageReferrers = "analytics.refer.%v"
View Source
const PageString = "page.%v"
View Source
const PageViewBonus = 60 * 60 * 24
View Source
const PageViewBucket = "analytics.pv_bucket"
View Source
const PageViewPageBucket = "analytics.pv_bucket.%v"
View Source
const PageViews = "analytics.pv"
View Source
const PageZsetByTime = "pages_by_time"
View Source
const PageZsetByTrend = "pages_by_trend"
View Source
const PagesInModules = 3

TODO: move this to Config

View Source
const RateLimitPeriod = 60
View Source
const Referrers = "analytics.refer"
View Source
const SimilarPagesByTrend = "similar_pages.%v"
View Source
const SimilarPagesExpire = 60 * 60 * 24
View Source
const TagPagesZsetByTime = "tag_pages_by_time.%v"
View Source
const TagPagesZsetByTrend = "tag_pages_by_trend.%v"
View Source
const TagZsetByPages = "tags_by_pages"
View Source
const TagZsetByTime = "tags_by_times"
View Source
const UserAgents = "analytics.useragent"

Variables

View Source
var BotAgents = []string{
	"-",
	"facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)",
	"mozilla/5.0 (compatible; yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp)",
	"disqus/1.0",
	"sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)",
	"baiduspider+(+http://www.baidu.com/search/spider.htm)",
	"baiduspider+(+http://www.baidu.jp/spider/)",
	"ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)",
	"java/1.6.0_24",
	"python-urllib/2.6",
	"ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)",
	"mozilla/5.0 (compatible; topblogsInfo/2.0; +topblogsinfo@gmail.com)",
	"mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
}
View Source
var FilteredRefs = []string{
	"www.google.com",
	"lethain.com",
	"dev.lethain.com",
	"DIRECT",
	HistoricalReferrer,
}

Functions

func BuildAtomFeed

func BuildAtomFeed(cfg *Config, ps []*Page) (*feeds.Feed, error)

func ConfigRedis

func ConfigRedis(cfg *Config) error

func ConfigSearch

func ConfigSearch(cfg *Config) error

func ConfigTemplates

func ConfigTemplates(cfg *Config) error

func CurrentTimestamp

func CurrentTimestamp() int64

func GetIP

func GetIP(r *http.Request) string

func GetRedisClient

func GetRedisClient() (*redis.Client, error)

func IndexPage

func IndexPage(p *Page) error

func IndexPages

func IndexPages(pgs []*Page) error

func IsRateLimited

func IsRateLimited(key string) bool

func PagesInList

func PagesInList(list string) (int, error)

func PutRedisClient

func PutRedisClient(rc *redis.Client)

func Referrer

func Referrer(r *http.Request) string

func RegisterPage

func RegisterPage(p *Page) error

func RegisterPageTag

func RegisterPageTag(p *Page, tag string) error

func ReindexAll

func ReindexAll() error
func Search(qs string) ([]string, error)

func Serve

func Serve(cfg *Config)

func ShouldIgnore

func ShouldIgnore(p *Page, r *http.Request) bool

func SlugsForList

func SlugsForList(list string, offset int, count int, reverse bool) ([]string, error)

func Track

func Track(p *Page, r *http.Request) error

func UnindexPage

func UnindexPage(p *Page) error

func UnregisterPage

func UnregisterPage(p *Page) error

func UnregisterPageTag

func UnregisterPageTag(p *Page, tag string) error

Types

type BlogConfig

type BlogConfig struct {
	Name             string
	ResultsPerPage   int    `json:"results_per_page"`
	PagesInPaginator int    `json:"pages_in_paginator"`
	TemplateDir      string `json:"template_dir"`
	StaticDir        string `json:"static_dir"`
}

.BlogName -> .Blog.Name .ListCount -> .Blog.ResultsPerPage .NumPages -> .Blog.PagesInPaginator

type Config

type Config struct {
	Server ServerConfig
	RSS    RSSConfig
	Blog   BlogConfig
	Redis  RedisConfig
}

func NewConfigFromFile

func NewConfigFromFile(path string) (*Config, error)

Build a new configuration file from disk.

func (*Config) BaseURL

func (cfg *Config) BaseURL() string

type NoSuchPagesError

type NoSuchPagesError struct {
	Slugs []string
	// contains filtered or unexported fields
}

func (*NoSuchPagesError) Error

func (e *NoSuchPagesError) Error() string

type Page

type Page struct {
	Slug        string   `json:"slug"`
	Tags        []string `json:"tags"`
	Title       string   `json:"title"`
	Summary     string   `json:"summary"`
	Content     string   `json:"html"`
	Draft       bool     `json:"draft"`
	PubDateStr  int64    `json:"pub_date"`
	EditDateStr int64    `json:"edit_date"`
}

func PageFromRedis

func PageFromRedis(slug string) (*Page, error)

Retrieve one page from Redis.

func PagesForList

func PagesForList(list string, offset int, count int, reverse bool) ([]*Page, error)

func PagesFromRedis

func PagesFromRedis(slugs []string) ([]*Page, error)

Retrieve a list of slugs from Redis.

func ReadHeaders

func ReadHeaders(content string) (*Page, string, error)

Create a new Page with the headers stripped out of it.

Headers look like this:

"tags": ["python", "redis"], "title": "Storing Bounded Timeboxes in Redis", "summary": "This is the summary...", "slug": "storing-bounded-timeboxes-in-redis",

Including a \n\n (outside of string) to designate the end of the section. In general, you can describe the header as a JSON dictionary without the opening or closing {}s.

func RecentPages

func RecentPages(offset int, count int) ([]*Page, error)

func Render

func Render(filename string, content string) (*Page, error)

func RenderHTML

func RenderHTML(content string) (*Page, error)

func RenderMarkdown

func RenderMarkdown(content string) (*Page, error)

func SimilarPages

func SimilarPages(p *Page, offset int, count int) ([]*Page, error)

func Surrounding

func Surrounding(p *Page, num int, reverse bool) ([]*Page, error)

Get up to N preceeding or following pages.

func TrendingPages

func TrendingPages(offset int, count int) ([]*Page, error)

func (*Page) EditDate

func (p *Page) EditDate() time.Time

func (*Page) EnsureEditDate

func (p *Page) EnsureEditDate()

func (*Page) EnsurePubDate

func (p *Page) EnsurePubDate()

func (*Page) InitEditDate

func (p *Page) InitEditDate()

func (*Page) InitPubDate

func (p *Page) InitPubDate()

func (*Page) Key

func (p *Page) Key() string

Generate the Redis key for this page.

func (*Page) PubDate

func (p *Page) PubDate() time.Time

func (*Page) Sync

func (p *Page) Sync() error

Synchronize this page to Redis.

type PageOpt

type PageOpt struct {
	Num      int
	Offset   int
	Selected bool
}

type Paginator

type Paginator struct {
	Pages      []PageOpt
	NextOffset int
	PrevOffset int
	HasNext    bool
	HasPrev    bool
	Show       bool
}

func NewPaginator

func NewPaginator(offset int, total int, pageSize int, numPages int) Paginator

Create a new Paginator.

offset is the current offset (in items). total is the total number of items. pageSize is the number of items per page. numPages is the number of pages to display

type RSSConfig

type RSSConfig struct {
	Path  string
	Title string
}

type RedisConfig

type RedisConfig struct {
	Loc      string
	Proto    string
	PoolSize int
}

type ServerConfig

type ServerConfig struct {
	Loc    string
	Proto  string
	Domain string
}

replacing:

NetLoc      string `json:"netloc"`
DomainUrl   string `json:"domain"`

type Tag

type Tag struct {
	Slug  string
	Count int
}

func GetAllTags

func GetAllTags() ([]Tag, error)

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL