misfin-server

Misfin-Server in Golang

A very simple misfin server in golang that can now support multiple users. Supports gemmail parsing in gemmail subdirectory as well as a new gembox (mbox for gemmail) format in the gembox subdirectory. The default port for misfin is 1958, but this can be configured when you initialize your server.

Note: I have also created a misfin terminal client in golang, which you can find on the misfinmail repo.

Copyright: Christian Lee Seibold
License: BSD-3-Clause

You can find more information about the misfin protocol, created by lem, at its Gemini Capsule: gemini://misfin.org

Building/Installing the server

go build -o . ./...

You can instead install with golang using the following command:

go install gitlab.com/clseibold/misfin-server/misfin-server@master

Note: Upon releases, pre-compiled binary files are added to the release. To view these, go to Deploy > Releases in the sidebar on Gitlab.

Terms Used

• mailbox - a mailbox consists of an address using the mailbox name and hostname (mailbox_name@hostname), a full name (aka. blurb) that represents the user of the mailbox or the mailbox's purpose, and a place to store incoming messages.
• gemmail - the message format for misfin. It is a superset of gemtext.
• gembox - a single-file storage format for multiple gemmails, inspired by mbox.
• server certificate - this is the certificate used to make tls connections to communicate with clients and other server, and acts as a CA for all other mailboxes on the server. The server certificate therefore must be accessible by the server. A server certificate is also the cert for a mailbox itself, somtimes called the root mailbox by the software configuration files, and its mailbox is typically named "admin" on multi-mailbox setups. On single-mailbox setups, it may be the only mailbox and may use the user's preferred mailbox name.
• mailbox cert - this is the certificate for a mailbox hosted by the misfin server and is signed by the server certificate. A mailbox cert contains the hostname, username (mailbox name), and full name (aka. blurb) of the mailbox, which tells the server and recipient where they can contact the user and at which address. When a person sends a message to a misfin address, the client sends a request to the server using the mailbox cert and this mailbox cert is how a misfin server identifies the sender and their misfin address. A certificate can be copied and used from any computer as long as the user's misfin server is running. The IP of the sender is never used for replies. Instead, the address and hostname of the certificate is always used.
• gemini fetch address - this is the gemini address one can use to fetch (download) a gembox. Some gemboxes are public, and some require that the client cert be the mailbox cert of the mailbox trying to be downloaded, to ensure that only the mailbox's user has access to their gembox.
• geminimail - this is to misfin what webmail is to traditional email. "geminimail" is a misfin client implemented as a gemini interface accessible from any gemini browser.

Creating a Cert

Certs will be created when you use the init and make mailbox commands. More info on these below, in the Running the server section. Private key of users should be kept to the user, so the server does not store them. Instead, the server stores the certificate information (hostname, mailbox name, full name/blurb, and fingerprint). This is particularly useful on public access multi-user/timesharing systems (like pubnixes).

For the server certificate and mailinglist certificates, the private keys are stored on the server so that the server can use them to make requests to other misfin servers. Mailinglist certs will be stored in the configured mailboxes directory.

Running the server

Initialize the server and create your server certificate (which acts as a CA for all of the mailboxes) by running the init command, optionally passing in a config filepath:

misfin-server init [config_filepath]

You can instead initialize the server with a pre-existing server certificate (which must be a CA) using the following command:

misfin-server init [config_filepath] --server-cert server_cert_filepath

It will interactively guide you through setting up your server and cert, as well as ask you for a config_filepath where your configuration file is saved (if one was not provided). If you do not go with the default location, then this config_filepath must be passed to every command from here on.

You can start the server with the serve command. Give it your config_filepath that you set during the init step. If you do not, then the default of ./misfinserver.conf is used.

misfin-server serve [config_filepath]

Or you can add user mailboxes with this command. Give it your config_filepath that you set during the init step. If you do not, then the default of ./misfinserver.conf is used.

misfin-server make mailbox [config_filepath]

Currently, you must run the make mailbox command while the server is not running. More info on creating mailinglists and newsletters is provided below.

Finally, you can change the config using the config command. You can configure the following fields: port, bind-address, server-cert, mailbox-dir.

Creating a mailinglist or newsletter

You can run the make mailinglist command like below and it will guide you through creating a mailinglist. You can instead run the make newsletter command, which is just like the mailinglist command, except it will automatically choose the writers option for the send permission.

misfin-server make mailinglist [config_filepath]

The command will prompt you for a mailbox name, a full name (aka. blurb), and a description that is used in the mailinglist's welcome message (upon new subscriber). It will also prompt you for a send permission and a subs function. The send permission handles who can send to the mailinglist, based on the following options:

• subs - the default for mailinglists. Only subs can send to mailinglist.
• writers - the default for newsletters. Only designated writers or admins can send to mailinglist.
• open - anyone can send, even if not a sub.

The subs function handles whether people are auto-accepted as a sub when they send the Subscribe message. These are the options:

• accept-all - auto-accepts everyone as a sub who sends a subscribe message. Mailinglist will be publicly visible on the gemini interface if the gemini interface is enabled and set to public.
• accept-server-members - auto-accepts those who have a mailbox on the same mailserver as the mailinglist.
• closed - auto-rejects everyone. Mailinglist will not be publicly visible on the gemini interface and will require a certificate of a sub of the mailinglist to view.
• prompt - does not auto-accept, but will prompt mailinglist admins to add the user as a sub manually. Mailinglist will not be publicly visible on the gemini interface and will require a certificate of a sub of the mailinglist to view. Note: This option has not been implemented yet.

After prompting for these settings, the command will then generate the key, which will be stored on the mailingserver so that messages sent to the mailinglist can be forwarded.

To subscribe to a mailinglist, send a message with the subject of "Subscribe" to the mailinglist address. If your misfin client doesn't handle subjects separately, a subject is just a gemtext level-1 heading, so it would be "# Subject". The body of the message is ignored.

Only subs set to writer, admin, or federated (two mailinglists that have subscribed to each other with the "federated" sub type set) can forward messages to mailinglists. Use config mailinglist-sub command to configure a sub's type.

Importing Mailbox/Mailinglist/Newsletter into server

You can import a pre-existing mailbox into the server, as long as the certificate was signed by the server certificate and the hostnames match, using one of the following commands:

misfin-server import mailbox cert_file [config_file]
misfin-server import mailinglist cert_file [config_file]
misfin-server import newsletter cert_file [config_file]

These commands will ask for more information (if required) and add the mailbox to the server configuration files. It will also ask if you want to import a gembox. If so, it will copy gembox from the given filepath to the mailboxes directory.

Enabling the gemini server

By default, the gemini server is disabled. When enabled, it runs over the same port as the set port for the misfin server (1958 by default). You can enable it by running the following command, replacing value with the value you want to set the field to:

misfin-server config server gemini-permission value

Supported values for the gemini-permission field include:

• disabled: Gemini interface does not run, and the server rejects all gemini requests.
• admin-only: Only admins can access the interface.
• member-only: Only members can access the interface. Non-admins cannot register for a mailbox or create new mailboxes. Only admins can create new mailboxes or change server configuration settings.
• public: Gemini interface is accessible to anybody, but only those with a certificate from a mailbox on the server can login. Only admins can change server configuration settings. Mailinglists/newsletters with subs function set to accept-all will be publicly visible without client cert, otherwise they will require a client cert of a sub to view.

Enable Mailbox Registration via Gemini Interface

Set gemini-user-registration-permission to admin-only or public with the config server sub-command to allow admins and/or the public to register new mailboxes via the gemini interface. The gemini interface must also be enabled.

Setup Federated Mailinglist

You can federate your mailinglist with another mailinglist by having both mailinglists of different servers subscribe to each other, and enabling both to forward messages to each other. They should preferably have the same mailbox name, at the very least.

1. First, create your mailinglist to be federated (misfin-server make mailinglist) and start your server.
2. Then, subscribe your mailinglist to the other server's mailinglist using misfin-server config mailinglist-federate mailinglist_name mailinglist_to_sub_to, providing the mailbox name of your mailinglist and the address of the other server's mailinglist.
3. The other server's mailinglist should Subscribe to your mailinglist. Once that is done, stop your server and allow their mailinglist to forward messages to yours using misfin-server config mailinglist-sub mailinglist_name sub_address type federated, prividing the mailbox name of your mailinglist and the address of the other server's mailinglist.
4. Start your server back up. The two mailinglists should be federated now.

Configuring the server

One can configure the server using the following subcommand:

misfin-server config server field value [config_filepath]

You can configure a mailbox using the following subcommand:

misfin-server config mailbox mailbox_name field value [config_filepath]

Fields include: type, send, subs-function, gemini-permission, gemini-user-registration-permission, and description.

Subcommands

• init - initialize the server
• init --server-cert - initialize the server using a pre-existing server-cert
• serve [config_filepath] - runs the server, using the passed-in configuration file
• make mailbox [config_filepath] - makes a mailbox, storing its cert in the current directory (so it can be moved wherever you wish)
• make mailinglist [config_filepath] - makes a mailinglist, storing its cert in the configured mailboxes directory
• make newsletter [config_filepath] - makes a newsletter, storing its cert in the configured mailboxes directory
• config server field value [config_filepath] - configured the server by setting the field to the value
• config mailbox mailbox_name field value [config_filepath] - configure mailbox
• config mailinglist-sub mailinglist_name sub_addr field value [config_filepath] - configure a subscriber of a mailinglist
• config mailinglist-federate mailinglist_name sub_to_addr [config_filepath] - Subscribe your mailinglist to another's mailinglist. This is part of the setup to join a mailinglist federation.
• import mailbox [config_filepath] - imports a mailbox, prompting for its cert, and optionally its gembox
• import mailinglist [config_filepath] - imports a mailinglist, prompting for its cert, and optionally its gembox
• import newsletter [config_filepath] - imports a newsletter, prompting for its cert, and optionally its gembox
• help subcommand
• version - prints the current version

How Mailinglist Federation is Implemented

Before being able to implemnet mailinglist federation, a server needs to implement mailinglists. A mailinglist will allow subscribers to send a mail to the mailinglist address, and it will forward that mail to every other subscriber. That's the basics of a mailinglist in misfin. However, because forwards can be used to spoof the sender's list, mailinglists should not allow mails to be forwarded to them unless it's from a trusted source.

For federation, two mailinglists subscribe to each other and allow each other to forward messages to each other.

The last implementatin detail is to prevent duplicate messages from looping in the system. First, messages should never be forwarded to addresses that are already part of the current senders list of the mail. Lastly, to prevent the rest of the looping that can occur, one can check if a mail is already in the mailinglist's mailbox by checking the original sender and the first timestamp of the mail, which should always remain the same during transmission of the message around the whole network. If the original sender and timestamp are already in the mailbox, then the message is likely a duplicate. One could optionally check the message contents as well. If a duplicate is found, then the message is not added to the mailbox and it is not forwarded to subs.

Here's an image detailing how this works:

{width=250px}

Spec-Compliant Incoming Mail Handling

The server follows the spec in adding the sender and timestamp to the gemmail file, as per the sections below, from the spec (gemini://misfin.org/specification.gmi):

Sender lines should be added by the server when saving or retransmitting a message. If a message is forwarded, the original sender line should be preserved and sent alongside, so the final recipient will see both senders:

• misfin spec, section 4.1, "Sender line"

and...

Like sender lines, timestamp lines should be added by the receiving mailserver, and only sent if forwarding a message, in which case they should be left as-is.

• misfin spec, section 4.3, "Timestamp line"

Importing gembox and gemmail packages

This repository offers three useful packages for creating both misfin servers and clients. The misfin_client package will implement a simple client that can send requests to a misfin server and receive the response. The gemmail package handles parsing gemmail files (as per the misfin spec), and allows you to make changes (like prepend a sender or timestamp) to the gemmail. The gembox package is a parser for the gembox file format outlined in the Gembox section below. It is a flat file of multiple gemmails separated by a separator line. The gembox package will split these up into multiple gemmails and wrap around the gemmail package.

You should be able to import the misfin_client, gembox, and gemmail packages into your own project like so:

import "gitlab.com/clseibold/misfin-server/misfin_client"
import "gitlab.com/clseibold/misfin-server/gembox"
import "gitlab.com/clseibold/misfin-server/gemmail"

The Gemmail Format (From the Spec)

The gemmail format is like a gemtext file, with three new linetypes. The three new linetypes are as follows:

• Sender Line - a sender line starts with < followed by the sender misfin address. You can add a space and the blurb (aka. full name) after the misfin address, but this is optional. The general syntax is like so, with square brackets for optional characters <[ ]mailbox@hostname.com[ First Last]. When a message comes in to the receiving mailserver, you should prepend the incoming sender just above the first sender line of the gemmail, at the top.
• Recipients Line - a receivers line is a list of those who have received the gemmail. The linetype starts with a colon (:) and is followed by misfin addresses separated with a space. This line is useful if sending one gemmail to multiple people and you want those people to be able to know who else the gemmail was sent to as well as reply to all people involved. If a gemmail is sent to just one person, then there is no need for a recipients line. Note that the person you are directly sending a copy of the gemmail to does not need to be on the recipients line. When sending a copy to two@example.com, their address does not go on the recipients line, and then when you send to the next recipient three@example.com, include the address for two@example.com, but not for three@example.com.
• Timestamp Line - a timestamp line tells you when a message was received. The receiving mailserver adds a timestamp to the gemmail when the message is received. The line begin with @ and is followed by an ISO-8601 format (and I believe always in UTC). Theoretically, this means each receiving mailserver will add their own timestamp, so if a gemmail is being forwarded, there will be multiple timestamps. The last timestamp should be the first one listed, assuming mailservers always add their timestamp above the previous timestamp in the gemmail so that the timestamps are in the reverse order they are received, much like the sender lines.

Lastly, a message's subject is always assumed to be the first gemtext level-1 heading (the first line starting with #). All other lines should be interpreted as per the Gemini gemtext spec.

If you are forwarding a gemmail, keep all sender lines. Each receiving mailserver will add a sender line for each sender.

Gemmail Parser

The gemmail parser (in the gemmail subdirectory) will parse both the spec-specific gemmail linetypes as well as all gemtext linetypes. Gemtext linetypes are placed in a separate array (GemMail.GeminiLines) of the GemMail struct to distinguish them from gemmail linetypes.

Call the String() function on a GemMail variable to convert the gemmail back into a string for saving. You can also append Senders, Receivers, and Timestamps to the GemMail before converting it back into a string.

Gembox (.gembox or .gmbox file) and Gembox parser

The Gembox format was something I created to be a single-file storage of multiple gemmails, inspired by the concept of mbox files in Multics and Unix, and other Operating Systems.

Gemboxes are gemmails with the divider <=====\n appearing on a separate line in-between each gemmail. This divider never appears at the beginning or end of a gembox file (they are always between two gemmails). When you receive mail, the mail will be appended to the gembox file and will be printed to the terminal.

The Gembox parser (in the gembox subdirectory) will simply split a multi-line string based on the divider linetype <=====\n to get each individual gemmail. Then, each gemmail will be passed to the Gemmail parser mentioned above. The GemBox struct contains a slice of GemMails.

Call the String() function on a GemBox variable to convert the gembox back into a string for saving. You can also append GemMails to the GemBox before converting it back into a string.

The divider was chosen because it starts with a gemmail linetype (the sender line, using the < character), however ===== is an invalid address. This means readers that don't support gembox will either fail on the line or ignore it if they are trying to parse addresses correctly. The intention is that this hopefully introduces some backwards compatibility, however, there may be some other way of doing this better. I am not set on the specific divider that I have chosen, so feedback is welcome.

Misfin Client Library (under misfin_client package)

The misfin_client package is a client library for misfin that was heavily based off of makeworld's go-gemini client library. The ISC license for makeworld's client library can be found in LICENSE-CLIENT. Additions and modifications by me are licensed with BSD-3-Clause.

Below is a simple example of a client that sends a message to clseibold@auragem.letz.dev:

package main

import (
    "fmt"
    "os"
    "gitlab.com/clseibold/misfin-server/misfin_client"
)

func main() {
    message := "# Test\nThis is a test gemmail!"
    certFile, _ := os.ReadFile("mailbox.pem")
    client := misfin_client.Client{}
    resp, err := client.SendWithCert("misfin://clseibold@auragem.letz.dev", certFile, certFile, message)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%d %s\n", resp.Status, resp.Meta) // Should print `20 <fingerprint>` if server is running
}

Contact Me

You can contact me via misfin at the address clseibold@auragem.letz.dev.