mirror

Mirror

A Go Vanity URL Server

★ Bugs ★ Questions

Mirror is a vanity URL server that doesn't require a fixed project list, has no database, and supports numerous remote hosts. It works by taking any vanity request and checking upstream hosts to see if any have a repo by the same name. When it finds the repo, it caches the mapping in memory, and returns the magic redirect.

These things are a dime a dozen, but most

• Need a database, or
• Need a configuration file, or
• Are hard-coded to one service

The first two are suboptimal for me, because it means editing a file on a server and restarting a service every time I want to write a new project. The last is a non-starter because monocultures are toxic.

The upstream URLs are provided in environment variables or on the command line. Each is rooted in an organization, e.g., https://hg.sr.ht/~ser.

There are three basic assumptions here:

1. There will be no conflicting repos in multiple locations, or that if there are, the upstream sources can be ordered. If you have a git repo in github, and both git and Mercurial repos in Sourcehut, Mirror will use the first it finds in the order you provided. This scheme will not work for you if you have multiple duplicate projects in multiple places and they're all in different priority order. If you're doing something like that, use a vanity server that's configurable instead.
2. It's not prohibitive to query a small set of servers once looking for repositories.
3. There aren't enough repositories in the world to exhaust an in-memory cache of project => URL maps.

The "search" step will only happen the first time the vanity is requested. After a repo is found, it's cached in memory and the cache is used thereafter. If the server is restarted, projects have to be re-searched and re-cached when they're requested. If your servers are bouncing frequently enough for this to be an issue, you have bigger problems (possibly caused by my code); in that case, use a configurable vanity server.

Projects are the first part of the path, so submodules and versions are stripped; https://myhost/myproj/v3, https://myhost/myproj/v3/lib, https://myhost/myproj/lib and https://myhost/myproj should all warm the cache to the same redirection. As expected. Mirror ignores parameters; it doesn't need or recognize ?go-get=1, but that's mostly useful for reverse proxies anyway.

If called with no path, it returns an index of cached projects. This functionality is stolen directly from govanityurls; I thought it was a neat idea.

Caveat: There are no tests, and the code is undocumented. If you choose to run this, do some testing with your own projects. Maybe run it under a supervisor in case it crashes. It should fail fast if anything is misconfigured, but it's been pretty stable for me.

Usage

Install mirror with:

go install ser1.net/mirror@latest

Or, check out this project, go build it, and put the binary wherever.

Set the MIRROR_LISTEN environment variable to the listen IP:PORT (:8080, for example). On the command line, provide a list of URLs of your VCS hosts, prefixing each URL with the VCS that it serves:

MIRROR_LISTEN=:5555 mirror hg:https://hg.sr.ht/~ser git:https://github.com/xxxserxxx git:https://git.sr.ht/~ser git:https://gitlab.com/serussell

When a project is requested, the servers will be queried in order. Consequently, order your servers in descending popularity for maximum efficiency. Test it with

curl -i http://localhost:5555/YOURPROJECT?go-get=1

mirror does not implement SSL, so put it behind a reverse proxy. If you're using Caddy, you'd do something like:

@mirror {
  method GET
  path /*
  query go-get=1
}
reverse_proxy @mirror http://localhost:5555

You should have the VCS commands for each VCS type installed; if you don't, mirror will report an error, but skip over it. Mirror will try: hg, git, svn, and bzr.

Mirror uses Go's x/tools VCS package to Ping() the servers; it should be a reasonably lightweight query. It takes about a second per host to find, or fail to find, a project. This depends, of course, on the server, but for sr.ht, github, and gitlab, response times for a ping were similar.

I recommend, at least the first time you run mirror, querying your main projects. This will not only warm the cache, but make sure everything is working properly. You can do this with a shell script:

for proj in p1 p2 p3; do
  curl -i https://yourhost/${proj}\?go-get\=1
done

If it's working correctly, you should get the standard Go module redirect response.

Other things

If you happen to have a server that serves multiple VCSes with the same base URL (I can't imagine how, but I suppose it could be done), then just repeat the URL with a different VCS prefix, in the order you want them checked:

MIRROR_LISTEN=:5555 mirror hg:https://superserver/me git:https://superserver/me

You can also specify the repos as environment variables. Any environment variables that start with MIRROR_ and are not MIRROR_LISTEN will be parsed as host arguments the same as if they were on the command line. For example, these are valid (but not all equivalent! See below):

MIRROR_LISTEN=:5555 mirror hg:https://hg.sr.ht/~ser git:https://github.com/xxxserxxx git:https://git.sr.ht/~ser git:https://gitlab.com/serussell
MIRROR_LISTEN=:5555 MIRROR_SRHTHG=hg:https://hg.sr.ht/~ser MIRROR_GITHUB=git:https://github.com/xxxserxxx MIRROR_SRHTGIT=git:https://git.sr.ht/~ser MIRROR_GITLAB=git:https://gitlab.com/serussell mirror 
MIRROR_LISTEN=:5555 MIRROR_SRHTHG=hg:https://hg.sr.ht/~ser MIRROR_GITHUB=git:https://github.com/xxxserxxx mirror git:https://git.sr.ht/~ser git:https://gitlab.com/serussell

All arguments are ordered before any environment variables, but note: environment variables are not ordered. If precedence is not important to you, you can safely use env vars: if there are no project conflicts, it makes little difference how mirror is configured. While it will take longer for the initial query to find the projects if the host orders are suboptimal, the search happens only once per project.

Repeating services has no effect: the list of vcs:url is a proper set.