go-pst

A library for reading PST files (written in Go/Golang).

Introduction

go-pst is a library for reading PST files (written in Go/Golang).

The PFF (Personal Folder File) and OFF (Offline Folder File) format is used to store Microsoft Outlook e-mails, appointments and contacts. The PST (Personal Storage Table), OST (Offline Storage Table) and PAB (Personal Address Book) file format consist of the PFF format.

References

Special thanks to James McLeod for helping me with some debugging.

Documentation

• Outlook Personal Folders (.pst) File Format
• Personal Folder File (PFF) file format specification

Libraries

• java-libpst
• libpff
• XstReader
• pstreader
• PANhunt

Datasets

This library is tested on the following datasets:

• enron.pst
  • Enron Corporation
• 32-bit.pst
  • DFRWS 2009 Rodeo
• support.pst
  • Hacking Team
  • 50GB worth of PST files from Hacking Team is available via this torrent magnet link (see the folders mail, mail2, mail3):

    magnet:?xt=urn:btih:51603bff88e0a1b3bad3962614978929c9d26955&dn=Hacked%20Team&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=http%3A%2F%2Fmgtracker.org%3A2710%2Fannounce&tr=http%3A%2F%2Fbt.careland.com.cn%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Fexodus.desync.com%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Ftracker.pomf.se&tr=udp%3A%2F%2Ftracker.blackunicorn.xyz%3A6969
    

Usage

package main

import (
  "fmt"
  pst "github.com/mooijtech/go-pst/v2/pkg"
)

func main() {
  pstFile := pst.New("data/enron.pst")

  fmt.Printf("Parsing file: %s\n", pstFile.Filepath)

  isValidSignature, err := pstFile.IsValidSignature()

  if err != nil {
    fmt.Printf("Failed to read signature: %s\n", err)
    return
  }

  if !isValidSignature {
    fmt.Printf("Invalid file signature.\n")
    return
  }

  contentType, err := pstFile.GetContentType()

  if err != nil {
    fmt.Printf("Failed to get content type: %s\n", err)
    return
  }

  fmt.Printf("Content type: %s\n", contentType)

  formatType, err := pstFile.GetFormatType()

  if err != nil {
    fmt.Printf("Failed to get format type: %s\n", err)
    return
  }

  fmt.Printf("Format type: %s\n", formatType)

  encryptionType, err := pstFile.GetEncryptionType(formatType)

  if err != nil {
    fmt.Printf("Failed to get encryption type: %s\n", err)
    return
  }

  fmt.Printf("Encryption type: %s\n", encryptionType)

  fmt.Printf("Initializing B-Trees...\n")

  err = pstFile.InitializeBTrees(formatType)

  if err != nil {
    fmt.Printf("Failed to initialize node and block b-tree.\n")
    return
  }

  rootFolder, err := pstFile.GetRootFolder(formatType, encryptionType)

  if err != nil {
    fmt.Printf("Failed to get root folder: %s\n", err)
    return
  }

  err = GetSubFolders(pstFile, rootFolder, formatType, encryptionType)

  if err != nil {
    fmt.Printf("Failed to get sub-folders: %s\n", err)
    return
  }
}

// GetSubFolders is a recursive function which retrieves all sub-folders for the specified folder.
func GetSubFolders(pstFile pst.File, folder pst.Folder, formatType string, encryptionType string) error {
  subFolders, err := pstFile.GetSubFolders(folder, formatType, encryptionType)

  if err != nil {
    return err
  }

  for _, subFolder := range subFolders {
    fmt.Printf("Parsing sub-folder: %s\n", subFolder.DisplayName)

    messages, err := pstFile.GetMessages(subFolder, formatType, encryptionType)

    if err != nil {
      return err
    }

    if len(messages) > 0 {
      fmt.Printf("Found %d messages.\n", len(messages))

      var attachmentsCount int

      for _, message := range messages {
        // Do something with the message.
        attachments, err := pstFile.GetAttachments(&message, formatType, encryptionType)

        if err != nil {
          fmt.Printf("Failed to get attachments: %s\n", err)
          continue
        }

        for _, attachment := range attachments {
          // Do something with the attachment.
          err = pstFile.WriteAttachmentToFile(attachment, "data/" + attachment.GetLongFilename(), formatType, encryptionType)

          if err != nil {
            fmt.Printf("Failed to write attachment to file: %s\n", err)
            continue
          }
        }

        attachmentsCount += len(attachments)
      }

      if attachmentsCount > 0 {
        fmt.Printf("Found %d attachments.\n", attachmentsCount)
      }
    }

    err = GetSubFolders(pstFile, subFolder, formatType, encryptionType)

    if err != nil {
      return err
    }
  }

  return nil
}

Implementation

This implementation is based on the references.
The source code of go-pst will reference this implementation.

File Header

Content Types

Format Types

The 64-bit header data

The 32-bit header data

Encryption Types

The node and block b-tree

The following offsets start from the (node/block) b-tree offset.

64-bit

64-bit 4k

32-bit

The b-tree entries

Note: When searching for an identifier make sure to only return the last found identifier as there can be two nodes (branch and leaf) with the same identifier.

The 64-bit block b-tree branch node entry

The 64-bit block b-tree leaf node entry

The 64-bit node b-tree leaf node entry

The 32-bit block b-tree branch node entry

The 32-bit block b-tree leaf node entry

The 32-bit node b-tree leaf node entry

Identifier

The 32-bit integer (identifier) can be used to search for b-tree nodes.

Identifier types

Heap-on-Node

If the encryption type was set in the file header, the entire Heap-on-Node is encrypted.

It is best to implement an input stream (Heap-on-Node input stream) to deal with encryption, local descriptors (which point to data) and blocks (XBlock and XXBlock) which point to block b-tree identifiers that contains the data.

Compressible encryption

Compressible encryption is a simple byte-substitution cipher with a fixed substitution table.

Heap-on-Node HID

Heap-on-Node header

The first block contains the Heap-on-Node header.

Heap-on-Node page map

Table types

B-Tree-on-Heap

B-Tree-on-Heap header

All tables should have a BTree-on-Heap header at HID 0x20 (the start offset to the BTree-on-Heap header is in the allocation table). This is the HID User Root from the Heap-on-Node header.

Property Context

The property context starts at the HID root of the B-Tree-on-Heap header.

A list of available properties can be found here.

Property Context B-Tree-on-Heap Record

Property types

The root folder

The identifier 290 refers to the root folder.

Sub-Folders

A folder identifier + 11 refers to the related sub folders (for example, for the root folder this is 290 + 11 = 301).

The related sub folders consists of the Table Context (7c table).

Each property ID of 26610 has a reference HNID which points to a message.

Messages

Messages have a property context where all data is stored.

Attachments

The message property context contains the property ID of 3591 (PidTagMessageFlags), reference HNID & 0x10 != 0 indicates if the message contains attachments.

The message local descriptors contains an identifier of 1649 which points to the attachments table context.

Table Context

The table context has a B-Tree-on-Heap.

The table context starts at the HID User Root of the Heap-on-Node.

Table Context Info

Table Context Column Descriptor

Blocks

Block sizes:

• Unicode: 8192
• Unicode4k: 65536
• ANSI: 8192

Block trailer sizes:

• Unicode: 16
• Unicode4k: 16
• ANSI: 12

XBlock

Number of records

Number of blocks: Size of the table context / (block size - block trailer size). The size of the table context is retrieved by the allocation table.

Rows per block: (block size - block trailer size) / row size

Row count: (number of blocks * rows per block) + ((table row matrix size % (block size - block trailer size)) / row size)

Current row start offset: (((startAtRow + currentRowIteration) / rowsPerBlock) * (blockSize - blockTrailerSize)) + (((startAtRow + currentRowIteration) % rowsPerBlock) * rowSize)

Cell Existence Block

Checks if a column exists.

Cell existence block size: math.Ceil(columnCount / 8)

Cell existence block: tableRowMatrix[currentRowStartOffset + tci1b:currentRowStartOffset + tci1b + cellExistenceBlockSize]

Cell existence block exists: cellExistenceBlock[column.CellExistenceBitmapIndex / 8] & (1 << (7 - (column.CellExistenceBitmapIndex % 8))) != 0

Table Context Item

If the column data size is 1, 2 or 4, the bytes contain a HNID which points to data in the Heap-on-Node, these offsets are in the allocation table.

Local Descriptors

The local descriptors identifier and size is in the b-tree entries.

If the local descriptors' identifier is 0, there are no local descriptors.

The Heap-on-Node HNID (in the allocation table) may point to a local descriptor which contains it's data in the block b-tree (using the data identifier).

Local descriptor entry size:

• 64-bit and 64-bit-with-4k: 24
• 32-bit: 12

The 64-bit local descriptors

The 64-bit-with-4k local descriptors are the same format as the 64-bit local descriptors.

The 64-bit local descriptors leaf node

The 32-bit local descriptors

The 32-bit local descriptors leaf node

Contact

Feel free to contact me if you have any questions.
Name: Marten Mooij
Email: info@mooijtech.com
Phone: +31 6 30 53 47 67

License

MIT