README
¶
ndn-dpdk/app/fileserver
This package is the file server, implemented as a traffic generator producer module.
It speaks the ndn6-file-server protocol.
It requires at least one thread, running the FileServer_Run function.
The file server is configured with one or more mountpoints.
Each mountpoint maps from an NDN name prefix to a path on the filesystem.
The top directory of each mountpoint is opened during file server initialization, and the user must not delete them while the file server is running.
Request Processing Workflow
Upon receiving an Interest, the name prefix consisting of only GenericNameComponents is used to lookup the list of mountpoints and determine a filesystem path, while the suffix consisting of non-GenericNameComponents classifies the request into one of the following kinds:
- request for file or directory metadata
- request for a directory listing segment
- request for a file segment
- unrecognized request - dropped
The file server invokes openat2 to open the file or directory (or dup in case of a request to the mountpoint directory itself), and then gathers information about file size, etc, via statx syscall.
Metadata and directory listing requests are responded right away.
For each file segment request, the file server constructs a reply Data packet that contains everything except the Content payload. It then creates iovec(s) of the payload and then enqueue a READV operation to the io_uring. Later, the file server polls for io_uring completions and transmits Data packets filled with Content payload.
File Descriptor Caching
The file server maintains a hashtable of open file descriptors.
If a request refers to a file or directory that already has an open file descriptor, the same file descriptor is reused instead of calling openat2 again.
Each open file descriptor is associated with a reference count, which indicates how many inflight READV operations are using this file descriptor.
As soon as the reference count reaches zero, i.e. the file descriptor becomes unused, it is placed in a cleanup queue (doubly linked list).
This cleanup queue has a limited capacity (configurable through keepFds option); if it's full, the oldest unused file descriptor is closed.
If a new request locates a file descriptor in the cleanup queue (i.e. its reference count is zero), the file descriptor is removed from the cleanup queue.
In short, the hashtable contains both active and unused file descriptors, while the cleanup queue forms a FIFO cache of unused file descriptors.
The result of statx syscall is stored together with each file descriptor, to avoid invoking statx for every request.
It is refreshed every few seconds (configurable through statValidity option) to keep the information up-to-date.
If the file server is configured to have multiple threads, each thread has its own file descriptor hashtable.
InputDemux for incoming Interests can dispatch Interests based on their name prefixes consisting of only GenericNameComponents, so that requests for the same file go to the same thread, eliminating the overhead of opening the same file in multiple threads.
Directory Listing
As specified in the ndn6-file-server protocol, directory listing is a segmented object that contains a textual payload.
Upon receiving a directory listing request, the file server invokes getdents64 syscall to gather directory listing, and then transforms the result into the textual format.
getdents64 is chosen over readdir because the latter often involves dynamic memory allocation.
During directory listing preparation, the file server uses dirent64.d_type field to determine filesystem object type.
If the underlying filesystem does not support this field or the directory entry is a symbolic link, the file server invokes statx syscall, either as a fallback or to resolve the symbolic link.
The directory listing response is stored together with the file descriptor, so that it can be used to satisfy requests for all segments.
In case file descriptor statx refresh detects that the directory has changed, the directory listing response is invalidated.
Limitations
The file server does not perform Data signing. Each Data packet has a Null signature, which provides no integrity or authenticity protection.
Directory listing response is limited to 256 KiB (MaxLsResult constant).
Large directories may be truncated.
Documentation
¶
Overview ¶
Package fileserver implements a file server.
Index ¶
- Constants
- Variables
- type Config
- type Counters
- type Mount
- type Server
- func (p *Server) Close() error
- func (p *Server) ConnectRxQueues(demuxI *iface.InputDemux)
- func (p *Server) Counters() (cnt Counters)
- func (p *Server) Face() iface.Face
- func (p *Server) Launch()
- func (p Server) Mounts() []Mount
- func (p *Server) Stop() error
- func (p *Server) Workers() []ealthread.ThreadWithRole
Constants ¶
const ( MaxMounts = 8 MaxLsResult = 262144 MinSegmentLen = 64 MaxSegmentLen = 16384 DefaultSegmentLen = 4096 MinUringCapacity = 256 MaxUringCapacity = 32768 // KERN_MAX_ENTRIES in liburing DefaultUringCapacity = 4096 DefaultUringCongestionThres = 0.7 DefaultUringWaitThres = 0.9 MinOpenFds = 16 MaxOpenFds = 16384 DefaultOpenFds = 256 MinKeepFds = 4 MaxKeepFds = 16384 DefaultKeepFds = 64 DefaultStatValidityMilliseconds = 10 * 1000 // 10 seconds EstimatedMetadataSize = 4 + 2 + 10 + 7*(4+8) // NNI fields MetadataFreshness = 1 )
Limits and defaults.
Variables ¶
var ( GqlMountInput *graphql.InputObject GqlConfigInput *graphql.InputObject GqlCountersType *graphql.Object GqlServerType *gqlserver.NodeType[*Server] )
GraphQL types.
var GqlRetrieveByFaceID func(id iface.ID) *Server
GqlRetrieveByFaceID returns *FileServer associated with a face. It is assigned during package tg initialization.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
NThreads int `json:"nThreads,omitempty"`
RxQueue iface.PktQueueConfig `json:"rxQueue,omitempty"`
// Mounts is a list of name prefix and filesystem path.
// There must be between 1 and MaxMounts entries.
// Prefixes should not overlap.
Mounts []Mount `json:"mounts"`
// SegmentLen is maximum TLV-LENGTH of Data Content payload.
// This value must be set consistently in every producer of the same name prefix.
SegmentLen int `json:"segmentLen,omitempty" gqldesc:"Maximum TLV-LENGTH of Data Content payload."`
// UringCapacity is io_uring submission queue size.
UringCapacity int `json:"uringCapacity,omitempty" gqldesc:"uring submission queue size."`
// UringCongestionThres is the uring occupancy threshold to start inserting congestion marks.
// If uring occupancy ratio exceeds this threshold, congestion marks are added to some outgoing Data packets.
// This must be between 0.0 (exclusive) and 1.0 (exclusive); it should be smaller than UringWaitThres.
UringCongestionThres float64 `json:"uringCongestionThres,omitempty" gqldesc:"uring occupancy threshold to start inserting congestion marks."`
// UringWaitThres is the uring occupancy threshold to start waiting for completions.
// If uring occupancy ratio exceeds this threshold, uring submission will block and wait for completions.
// This must be between 0.0 (exclusive) and 1.0 (exclusive).
UringWaitThres float64 `json:"uringWaitThres,omitempty" gqldesc:"uring occupancy threshold to start waiting for completions."`
// OpenFds is the limit of open file descriptors (including KeepFds) per thread.
// You must also set `ulimit -n` or systemd `LimitNOFILE=` appropriately.
OpenFds int `json:"openFds,omitempty" gqldesc:"Maximum open file descriptors per thread."`
// KeepFds is the number of unused file descriptors per thread.
// A file descriptor is unused if no I/O operation is ongoing on the file.
// Keeping them open can speed up subsequent requests referencing the same file.
KeepFds int `json:"keepFds,omitempty" gqldesc:"Maximum unused file descriptors per thread."`
// ResolveBeneath disallows absolute symbolic links during path resolution.
ResolveBeneath bool `json:"resolveBeneath,omitempty" gqldesc:"Disallow absolute symbolic links during path resolution."`
// StatValidity is the validity period of statx result.
StatValidity nnduration.Nanoseconds `json:"statValidity,omitempty" gqldesc:"statx result validity period."`
// WantVersionBypass allows setting special values in version component to bypass version check.
// This is intended for fileserver benchmarks and should not be set in normal operation.
WantVersionBypass bool `json:"wantVersionBypass,omitempty" gqldesc:"Allow bypassing version check in benchmarks."`
// contains filtered or unexported fields
}
Config contains FileServer configuration.
type Counters ¶
type Counters struct {
ReqRead uint64 `json:"reqRead" gqldesc:"Received read requests."`
ReqLs uint64 `json:"reqLs" gqldesc:"Received directory listing requests."`
ReqMetadata uint64 `json:"reqMetadata" gqldesc:"Received metadata requests."`
FdNew uint64 `json:"fdNew" gqldesc:"Successfully opened file descriptors."`
FdNotFound uint64 `json:"fdNotFound" gqldesc:"File not found."`
FdUpdateStat uint64 `json:"fdUpdateStat" gqldesc:"Update stat on already open file descriptors."`
FdClose uint64 `json:"fdClose" gqldesc:"Closed file descriptors."`
UringAllocError uint64 `json:"uringAllocErrs" gqldesc:"uring SQE allocation errors."`
UringSubmitted uint64 `json:"uringSubmitted" gqldesc:"uring submitted SQEs."`
UringSubmitNonBlock uint64 `json:"uringSubmitNonBlock" gqldesc:"uring non-blocking submission batches."`
UringSubmitWait uint64 `json:"uringSubmitWait" gqldesc:"uring waiting submission batches."`
UringCqeFail uint64 `json:"cqeFail" gqldesc:"uring failed CQEs."`
}
Counters contains file server counters.
type Mount ¶
type Mount struct {
Prefix ndn.Name `json:"prefix" gqldesc:"NDN name prefix."`
Path string `json:"path" gqldesc:"Filesystem path."`
// contains filtered or unexported fields
}
Mount defines a mapping between name prefix and filesystem path.
type Server ¶
type Server struct {
VersionBypassHi uint32
// contains filtered or unexported fields
}
Server represents a file server.
func (*Server) ConnectRxQueues ¶
func (p *Server) ConnectRxQueues(demuxI *iface.InputDemux)
ConnectRxQueues connects Interest InputDemux to RxQueues.
func (*Server) Workers ¶
func (p *Server) Workers() []ealthread.ThreadWithRole
Workers returns worker threads.
Source Files