srctx: source context

A library for extracting and analyzing definition/reference graphs from your codebase. Powered by tree-sitter and LSIF/SCIP.

Name	Status
Latest Version
Unit Tests
Code Coverage
Code Style

About this tool

This library processes your code into precise function-level graphs, seamlessly integrated with Git, and then you can apply some analysis to them.

With this lib developers can know exactly what happened in every lines of your code. Such as definition, reference. And understand the actual impacts of your git commits.

Some "dangerous" line changes can be found automatically.

You can see a dangerous change in file cmd/srctx/diff/cmd.go#L29-#143, .

In addition, as a library, it also provides convenient APIs for secondary development, allowing you to freely access the content in the graph.

src := filepath.Dir(filepath.Dir(curFile))
lsif := "../dump.lsif"
lang := core.LangGo

funcGraph, _ := function.CreateFuncGraphFromDirWithLSIF(src, lsif, lang)

functions := funcGraph.GetFunctionsByFile("cmd/srctx/main.go")
for _, each := range functions {
    // about this function
    log.Infof("func: %v", each.Id())
    log.Infof("decl location: %v", each.FuncPos.Repr())
    log.Infof("func name: %v", each.Name)

    // context of this function
    outVs := funcGraph.DirectReferencedIds(each)
    log.Infof("this function reached by %v other functions", len(outVs))
    for _, eachOutV := range outVs {
        outV, _ := funcGraph.GetById(eachOutV)
        log.Infof("%v directly reached by %v", each.Name, outV.Name)
    }
}

Output:

time="2023-06-17T19:48:52+08:00" level=info msg="func: cmd/srctx/main.go:#16-#26:main||mainFunc|[]string|"
time="2023-06-17T19:48:52+08:00" level=info msg="decl location: cmd/srctx/main.go#16-26"
time="2023-06-17T19:48:52+08:00" level=info msg="func name: mainFunc"
time="2023-06-17T19:48:52+08:00" level=info msg="this function reached by 7 other functions"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestDiffDir"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestDiffNoDiff"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestRenderHtml"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestDiffRaw"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestDiffSpecificLang"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by TestDiff"
time="2023-06-17T19:48:52+08:00" level=info msg="mainFunc directly reached by main"

Currently, srctx is still in an active development phase. If you're interested in its iteration direction and vision, you can check out our roadmap page.

Usage

Quick Start

We provide a one-click script for quickly deploying srctx anywhere. Common parameters include:

• SRCTX_LANG: Required, specifies the language, such as GOLANG/JAVA/KOTLIN.
• SRCTX_BUILD_CMD: Optional, specifies the compilation command.

For Golang

curl https://raw.githubusercontent.com/williamfzc/srctx/main/scripts/quickstart.sh \
| SRCTX_LANG=GOLANG bash

For Java/Kotlin

As there is no unique compilation toolchain for Java (it could be Maven or Gradle, for example), so at the most time, you also need to specify the compilation command to obtain the invocation information.

You should replace the SRCTX_BUILD_CMD with your own one.

Java:

curl https://raw.githubusercontent.com/williamfzc/srctx/main/scripts/quickstart.sh \
| SRCTX_LANG=JAVA SRCTX_BUILD_CMD="clean package -DskipTests" bash

Kotlin:

Change the SRCTX_LANG=JAVA to SRCTX_LANG=KOTLIN.

In Production

In proudction, it is generally recommended to separate the indexing process from the analysis process, rather than using a one-click script to complete the entire process. This can make the entire process easier to maintain.

1. Generate LSIF file

Tools can be found in https://lsif.dev/ .

You will get a dump.lsif file after that.

2. Run srctx

Download our prebuilt binaries from release page.

For example, diff from HEAD~1 to HEAD:

./srctx diff \
  --before HEAD~1 \
  --after HEAD \
  --lsif dump.lsif \
  --outputCsv output.csv \
  --outputDot output.dot \
  --outputHtml output.html

See details with ./srctx diff --help.

Prefer a real world sample?

Our CI is a good start.

Usage as Github Action (Recommendation)

Because LSIF files require dev env heavily, it's really hard to provide a universal solution in a single binary file for all the repos.

We are currently working on diffctx, which will provide a GitHub Actions plugin that allows users to use it directly in a Pull Request.

Usage as Lib

API

This API allows developers accessing the data of FuncGraph.

• Start up example: example/api_test.go
• Real world example: cmd/srctx/diff/cmd.go

Low level API

Low level API allows developers consuming LSIF file directly.

See example/api_base_test.go for details.

Correctness / Accuracy

We wanted it to provide detection capabilities as accurate as an IDE.

Roadmap

See Roadmap Issue.

Contribution

Issues and PRs are always welcome.

References

LSIF is a standard format for persisted code analyzer output. Today, several companies are working to support its growth, including Sourcegraph and GitHub/Microsoft. The LSIF defines a standard format for language servers or other programming tools to emit their knowledge about a code workspace.

• https://lsif.dev/
• https://microsoft.github.io/language-server-protocol/overviews/lsif/overview/
• https://code.visualstudio.com/blogs/2019/02/19/lsif#_how-to-get-started

Thanks

• SCIP/LSIF toolchains from https://github.com/sourcegraph
• LSIF from Microsoft
• LSIF parser from GitLab
• IDE support from JetBrains

License

Apache 2.0