bam

type AdjacentBAMShard ¶

type AdjacentBAMShard struct {
	// ShardIdx is the index of the shard.
	// Indexing starts at zero.
	ShardIdx int
	// contains filtered or unexported fields
}

AdjacentBAMShard represents an ordered subset of records from an AdjacentShardedBAMReader. The order of AdjacentBAMShard is determined by the shard's ShardIdx. If all of an AdjacentShardedBAMReader's AdjacentBAMShards were read sequentially from shard 0 to n, all the records would be read in the same order they appear in the underlying BAM file.

func (s *AdjacentBAMShard) Record() Pair

func (s *AdjacentBAMShard) Scan() bool

type AdjacentShardedBAMReader ¶

type AdjacentShardedBAMReader struct {
	// contains filtered or unexported fields
}

AdjacentShardedBAMReader provides a deterministic way to read a BAM file, provided that all records are grouped into adjacent pairs. Each AdjacentBAMShard has sequentially increasing shard numbers starting at zero. The records in each AdjacentBAMShard are in the same order as they appear in the underlying BAM file.

AdjacentBAMShards are returned one at a time by GetShard() as soon as a shard is ready. Shards are created automatically in a goroutine when NewAdjacentShardedBAMReader() is called. Each AdjacentBAMShard is thread-safe.

When records are adjacent, there is no BAM index so the the number of shards is indeterminate. Because of this, it is recommended that the caller limits the number of goroutines used for reading a BAM file. In this example, the number of goroutines is limited to the number of available CPUs. If there are more shards that can be concurrently processed than CPUs, there will be multiple shards sequentially processed in the same goroutine.

Example Use of AdjacentShardedBAMReader:

ctx := context.Background()
f, _ := os.Create("input.bam")
r, _ := NewAdjacentShardedBAMReader(ctx, f, 100000, 2)

err = traverse.CPU(func() error {
   for {
      shard := r.GetShard()
      if shard == nil { break }
      for shard.Scan() {
            pair := shard.Record()
            if pair.Err != nil { return pair.Err }
            // Do something with the record and
            // and use shard.ShardIdx to denote
            // the order of the shards.
      }
   }
   return nil
})

func NewAdjacentShardedBAMReader(ctx context.Context, r io.Reader, recordsPerShard int, queueSize int) (*AdjacentShardedBAMReader, error)

func (r *AdjacentShardedBAMReader) GetShard() *AdjacentBAMShard

func (r *AdjacentShardedBAMReader) Header() *sam.Header

type Bin ¶

type Bin struct {
	BinNum uint32
	Chunks []Chunk
}

Bin represents the bin data within a .bai file.

type Chunk ¶

type Chunk struct {
	Begin bgzf.Offset
	End   bgzf.Offset
}

Chunk represents the Chunk data within a .bai file.

type CoordGenerator ¶

type CoordGenerator struct {
	LastRec biopb.Coord
}

CoordGenerator is a helper class for computing the Coord.Seq value from a sam.Record. This object must be created per pam shard. Generate() must be called for every record that is being read or written to the pam file in order.

func NewCoordGenerator() CoordGenerator

func (g *CoordGenerator) Generate(refID, pos int32) biopb.Coord

func (g *CoordGenerator) GenerateFromRecord(rec *sam.Record) biopb.Coord

type FieldType ¶

type FieldType uint8

FieldType defines a sam.Record field. Each field is stored in a separate file.

const (
	// FieldCoord combines <sam.Reference.ID(), sam.Record.Pos>. They need to be
	// read together when seeking to a specific coordinate.
	FieldCoord FieldType = iota
	// Rest of Field* stands for the sam.Record field with the same name.
	FieldFlags
	FieldMapq
	FieldCigar
	FieldMateRefID
	FieldMatePos
	FieldTempLen
	FieldName
	FieldSeq
	FieldQual
	FieldAux

	// FieldInvalid is a sentinel
	FieldInvalid
	NumFields = int(FieldInvalid)
)

func ParseFieldType(v string) (FieldType, error)

func (f FieldType) String() string

type GIndex ¶

type GIndex []GIndexEntry

GIndex is an alternate .bam file index format that uses the .gbai file extension. The .gbai file format contains mappings from genomic position to .bam file voffset. This index format is simpler than the legacy style .bai file, but allows a user to seek into a .bam file much more efficiently for some genomic positions.

The .gbai format exists because the .bai format can point into a .bam file with a minimum genomic spacing of 16 kbp. The problem with this minimum spacing is that if there are many alignments in the .bam file within a 16 kbp region, then seeking to a target genomic position within the 16 kbp region requires the reader to seek to the beginning for the 16 kbp region and then scan through bam records until reaching the target genomic position. This scanning requires unnecessary IO and CPU time for reading and decompressing records that come before the target position.

The .gbai file format contains a set of mappings from (genomic position, and record number at that position) to the voffset in the bam file where the record begins. In typical use, the spacing between the genomic positions in the .gbai file are chosen so that the spacing between voffsets in the .bam file are uniform and relatively small. This allows a user to divide the .bam file into uniform sized shards. For example, 64 KBytes is a reasonable default spacing between voffsets. This spacing allows a reader to seek directly to within 64 KBytes of any target genomic position.

The on disk .gbai format is a header followed by a sequence of entries. The header consists of the magic byte sequence {0x47, 0x42, 0x41, 0x49, 0x01, 0xf1, 0x78, 0x5c,

0x7b, 0xcb, 0xc1, 0xba, 0x08, 0x23, 0xb1, 0x19}

which is "GBAI1" followed by 11 random bytes.

Each entry consists of 4 values, each in little-endian byte order:

1. int32 RefID to match the .bam file RefIDs. The unmapped records at the end of the .bam have RefID equal to -1.
2. int32 Position to match the .bam file Positions
3. uint32 Sequence number of the record at the particular (RefID, Position) pair. If the record is the first record with this (RefID, Position) pair, then Sequence will be 0. If the record is the second, then Sequence will be 1, and so on.
4. uint64 VOffset of the record in the .bam file as described in the .bam specification.

The .gbai index entries are sorted in ascending order using the key (RefID, Position, Sequence) and the .gbai index requires that the corresponding .bam file is also sorted by position.

If the bam file contains a bam record for a given RefID, then the gindex contains an entry for the first bam record with the given RefID. This implies that the first entry in the gindex points to the first record in the bam file. If there are no bam records with RefID R, then there will be no entries in the gindex with RefID R.

The series of index entries is then compressed with gzip before writing to the .gbai file.

func ReadGIndex(r io.Reader) (gindex *GIndex, err error)

func (idx *GIndex) RecordOffset(refID, pos int32, seq uint32) bgzf.Offset

func (idx *GIndex) UnmappedOffset() bgzf.Offset

type GIndexEntry ¶

type GIndexEntry struct {
	RefID   int32
	Pos     int32
	Seq     uint32
	VOffset uint64
}

GIndexEntry is one entry of the .gbai index.

type Index ¶

type Index struct {
	Magic         [4]byte
	Refs          []Reference
	UnmappedCount *uint64
}

Index represents the content of a .bai index file (for use with a .bam file).

func ReadIndex(rawr io.Reader) (*Index, error)

func (i *Index) AllOffsets() map[int][]bgzf.Offset

type Metadata ¶

type Metadata struct {
	UnmappedBegin uint64
	UnmappedEnd   uint64
	MappedCount   uint64
	UnmappedCount uint64
}

Metadata represents the Metadata data within a .bai file.

type Pair ¶

type Pair struct {
	R1  *sam.Record
	R2  *sam.Record
	Err error
}

Pair encapsulates a pair of SAM records for a pair of reads, and whether any error was encountered in retrieving them.

type Reference ¶

type Reference struct {
	Bins      []Bin
	Intervals []bgzf.Offset
	Meta      Metadata
}

Reference represents the reference data within a .bai file.

type Shard ¶

type Shard struct {
	StartRef *sam.Reference
	EndRef   *sam.Reference
	Start    int
	End      int
	StartSeq int
	EndSeq   int

	Padding  int
	ShardIdx int
}

Shard represents a genomic interval. The <StartRef,Start,StartSeq> and <EndRef,End,EndSeq> coordinates form a half-open, 0-based interval. An iterator for such a range will return reads whose start positions fall within that range.

The StartSeq, EndSeq fields are used to distinguish a list of reads that start at the same coordinate. The Nth read that start at the coordinate is assigned the seq value of N-1 (assuming N is 1-based). For example, Passing range [(startref=10,start=100,startseq=15), (limitref=10,limit=100,limitseq=20)] will read 16th to 20th read sequences at coordinate (10,100)

Uses of non-zero {Start,End}Seq is supported only in PAM files. For BAM files, *Seq must be zero.

An unmapped sequence has coordinate (nil,0,seq), and it is stored after any mapped sequence. Thus, a shard that contains an unmapped sequence will have EndRef=nil, End=1, EndSeq=0> (in theory, End can be any value > 0, but in practice we use End=1).

Padding must be >=0. It expands the read range to [PaddedStart, PaddedEnd), where PaddedStart=max(0, Start-Padding) and PaddedEnd=min(EndRef.Len(), End+Padding)). The regions [PaddedStart,Start) and [End,PaddedEnd) are not part of the shard, since the padding regions will overlap with another Shard's [Start, End).

The Shards are ordered according to the order of the bam input file. ShardIdx is an index into that ordering. The first Shard has index 0, and the subsequent shards increment the ShardIdx by one each.

func CoordRangeToShard(header *sam.Header, r biopb.CoordRange, padding, shardIdx int) Shard

func GetByteBasedShards(bamPath, baiPath string, bytesPerShard int64, minBases, padding int, includeUnmapped bool) (shards []Shard, err error)

func GetPositionBasedShards(header *sam.Header, shardSize int, padding int, includeUnmapped bool) ([]Shard, error)

func UniversalShard(header *sam.Header) Shard

func (s *Shard) CoordInShard(padding int, coord biopb.Coord) bool

func (s *Shard) MateInShard(r *sam.Record) bool

func (s *Shard) PadEnd(padding int) int

func (s *Shard) PadStart(padding int) int

func (s *Shard) PaddedEnd() int

func (s *Shard) PaddedStart() int

func (s *Shard) RecordInPaddedShard(r *sam.Record) bool

func (s *Shard) RecordInShard(r *sam.Record) bool

func (s *Shard) RecordInStartPadding(r *sam.Record) bool

func (s *Shard) String() string

type ShardedBAMCompressor ¶

type ShardedBAMCompressor struct {
	// contains filtered or unexported fields
}

ShardedBAMCompressor contains the state of an in-progress compressed shard. A caller should create a ShardedBAMCompressor using ShardedBAMWriter.GetCompressor(). The ShardedBAMCompressor will compress the records and store the compressed bytes until the caller is finished with the shard. When the caller is finished adding records, the caller should call CloseShard(). More than one ShardedBAMCompressor can exist at once, and they can all compress records in parallel with each other.

func (c *ShardedBAMCompressor) AddRecord(r *sam.Record) error

func (c *ShardedBAMCompressor) CloseShard() error

func (c *ShardedBAMCompressor) StartShard(shardNum int) error

type ShardedBAMWriter ¶

type ShardedBAMWriter struct {
	// contains filtered or unexported fields
}

ShardedBAMWriter writes out ShardedBAMBuffers in the order of their shard numbers.

func NewShardedBAMWriter(w io.Writer, gzLevel, queueSize int, header *sam.Header) (*ShardedBAMWriter, error)

func (bw *ShardedBAMWriter) Close() error

func (bw *ShardedBAMWriter) GetCompressor() *ShardedBAMCompressor

type StrandType ¶

type StrandType int

const (
	// StrandNone denotes an unmapped read.
	StrandNone StrandType = iota
	// StrandFwd denotes a read mapped to the R1+ strand.
	StrandFwd
	// StrandRev denotes a read mapped to the R1- strand.
	StrandRev
)

func GetStrand(flags sam.Flags) StrandType