tko

WARNING: This repository is published by the Nephio Authors but is neither endorsed nor maintained by the Nephio Technical Steering Committee (TSC). It is intended to be used for reference only. The Nephio distribution repositories are located in the nephio-project organization. For more information see this page.

TKO

A PoC demonstrating scalability opportunities for Nephio with a focus on decoupling the various subsystems, specifically the data backend and API access, as well as integration with external site inventories and blueprint catalogs.

This PoC is a complete rewrite of the Nephio core. It comprises three controllers (API server, Preparer, and Meta-Scheduler) that can run independently or be embedded into a control plane, including first-class support for Kubernetes management clusters.

Included is an RDBMS backend, specifically for the PostgreSQL dialect of SQL. RDBMSes provides scalability, reliability, resiliency, and atomic updates via transactions. They are widely available as managed cloud services, e.g. Google's planetary-scale Spanner (which speaks the PostgreSQL dialect).

A git backend is also possible (a.k.a. "GitOps"). Note that such an implementation may be suitable for storing templates, but it's probably not a good backend for sites and deployments, which are expected to number in the millions in real-world telco environments.

The TKO API is exposed in three flavors:

1. gRPC, which is widely supported, including in loadbalancers (see Envoy) and thus service meshes. TKO makes good use of gRPC streaming directly from the backend to the clients, allowing scalable access to extremely large result sets.
2. KRM via a Kubernetes aggregated API. When running in a Kubernetes management cluster, it may be preferrable for controllers to use KRM instead of gRPC, though the latter is still readily available inside Kubernetes. Note that this KRM API is not implemented as CRs and is not stored in Kubernetes's etcd database. Rather, it's a KRM facade over the TKO backend.
3. A simple JSON-over-HTTP API for web browser applications, such as GUIs.

Ways in which TKO differs from Nephio:

• A different approach to "specialization" (here called "preparation"), replacing the kpt file with per-resource plugins: no pipeline, no conditions. Enforces atomic updates. Kpt functions still get first-class support.
• A different approach to "instantiation" based on meta-scheduling of complete sites together with all their associated workload deployments, including infrastructure resources for infrastructure managers. Can work with existing sites and also provision new ones.
• A different approach to topologies, incorporating topology decomposition as part of the preparation process. Topologies can be nested. Also supports TOSCA topologies as an alternative frontend, implemented with Puccini.
• A different approach to KRM validation, supporting custom validation plugins that can go far beyond OpenAPIv3 schemas. Can also work on templates via "partial" validation. Invalid KRM is never allowed into the backend. Relies on Kubeconform for validating standard Kubernetes resources via their published OpenAPIv3 schemas.

Additional features demonstrated:

• SDK for Python-based plugins for preparation, meta-scheduling, and validation. The SDK does most of heavy lifting, allowing devs to focus on the network function or cloud platform vendor logic.
• Rich metadata support, enabling powerful and scalable package querying and template/site selection for topologies.
• Support for Helm charts with an expression language for pulling in chart input values (as an alternative to full-blown preparation plugins).
• Web dashboard.
• Rich terminal dashboard, with mouse support.
• IPv6 first, with support for dual-stack IP.
• Unified structured logging via CommonLog, including support for logging to journald.

Documentation

• Installation guide
• User guide
• Package reference
• KRM API
• How preparation works
• TODO

Architecture Diagram

%%{init: {'themeVariables': { 'edgeLabelBackground': 'transparent'}}}%%
flowchart TD
    D[(Data Backends)]

    C(TKO CLI and TUI)
    W(TKO Web GUI)
    O(Orchestrators)

    A[TKO API Server]
    P[TKO Preparer]
    MS[TKO Meta-Scheduler]

    PP[/preparer plugins/]
    VP[/validator plugins/]
    SP[/scheduler plugins/]

    C-. gRPC ..-A
    W-. HTTP ..-A
    O-. gRPC ..-A
    A-. gRPC ..- P
    A-. gRPC ..-MS

    D-..-A

    PP---P
    VP---P
    SP---MS

    P-.-DPA
    PR-.-MS
    MS-.-WC
    MS-.-SR

    subgraph TC [Template Catalog]
        ST[\Site Templates/]
        WT[\Workload Templates/]
        TT[\Topology Templates/]
    end

    subgraph SI [Site Inventory]
        PS{{Provisioned Sites}}
        ES{{Existing Sites}}
    end

    subgraph DPA [Deployment Preparation Area]
        UT{{Unprepared Topologies}}
        UR{{Unprepared Resources}}
        PR{{Prepared Resources}}
    end

    subgraph WC [Workload Cluster]
        SR{{Scheduled Resources}}
    end

    TT-->UT-->UR-->PR-->SR
    WT-->UR
    ST-->PS-->WC

    style A fill:blue,color:white
    style C fill:blue,color:white
    style W fill:blue,color:white
    style P fill:blue,color:white
    style PP fill:darkblue,color:white
    style VP fill:darkblue,color:white
    style MS fill:blue,color:white
    style SP fill:darkblue,color:white
    style ST fill:red,color:white
    style TT fill:red,color:white
    style WT fill:red,color:white
    style UT fill:yellow
    style UR fill:yellow
    style PR fill:green,color:white
    style SR fill:green,color:white
    style PS fill:green,color:white
    style ES fill:green,color:white
    linkStyle 0,1,2,3,4,9,10 stroke:blue,color:blue
    linkStyle 6,7,8 stroke:darkblue,color:darkblue

Additional Resources

• Presentation recording
• Components proposal (presentation recording)
• Nephio R2 Summit presentation slides