envd

module
v0.3.45 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 24, 2024 License: Apache-2.0

README ΒΆ

envd cat wink envd cat wink

Development environment for AI/ML

discord invitation link trackgit-views Python Version all-contributors envd package downloads continuous integration Coverage Status

What is envd?

envd (ΙͺnˈvdΙͺ) is a command-line tool that helps you create the container-based development environment for AI/ML.

Creating development environments is not easy, especially with today's complex systems and dependencies. With everything from Python to CUDA, BASH scripts, and Dockerfiles constantly breaking, it can feel like a nightmare - until now!

Instantly get your environment running exactly as you need with a simple declaration of the packages you seek in build.envd and just one command: envd up!

Why use envd?

Environments built with envd provide the following features out-of-the-box:

Simple CLI and language

envd enables you to quickly and seamlessly integrate powerful CLI tools into your existing Python workflow to provision your programming environment without learning a new language or DSL.

def build():
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")
    config.jupyter()

Isolation, compatible with OCI image

With envd, users can create an isolated space to train, fine-tune, or serve. By utilizing sophisticated virtualization technology as well as other features like buildkit, it's an ideal solution for environment setup.

envd environment image is compatible with OCI image specification. By leveraging the power of an OCI image, you can make your environment available to anyone and everyone! Make it happen with a container registry like Harbor or Docker Hub.

Local, and cloud

envd can now be used on a hybrid platform, ranging from local machines to clusters hosted by Kubernetes. Any of these options offers an efficient and versatile way for developers to create their projects!

$ envd context use local
# Run envd environments locally
$ envd up
...
$ envd context use cluster
# Run envd environments in the cluster with the same experience
$ envd up

Check out the doc for more details.

Build anywhere, faster

envd offers a wealth of advantages, such as remote build and software caching capabilities like pip index caches or apt cache, with the help of buildkit - all designed to make your life easier without ever having to step foot in the code itself!

Reusing previously downloaded packages from the PyPI/APT cache saves time and energy, making builds more efficient. No need to redownload what was already acquired before – a single download is enough for repeat usage!

With Dockerfile v1, users are unable to take advantage of PyPI caching for faster installation speeds - but envd offers this support and more!

Besides, envd also supports remote build, which means you can build your environment on a remote machine, such as a cloud server, and then push it to the registry. This is especially useful when you are working on a machine with limited resources, or when you expect a build machine with higher performance.

Knowledge reuse in your team

Forget copy-pasting Dockerfile instructions - use envd to easily build functions and reuse them by importing any Git repositories with the include function! Craft powerful custom solutions quickly.

envdlib = include("https://github.com/tensorchord/envdlib")

def build():
    base(os="ubuntu20.04", language="python")
    envdlib.tensorboard(host_port=8888)
envdlib.tensorboard is defined in github.com/tensorchord/envdlib
def tensorboard(
    envd_port=6006,
    envd_dir="/home/envd/logs",
    host_port=0,
    host_dir="/tmp",
):
    """Configure TensorBoard.

    Make sure you have permission for `host_dir`

    Args:
        envd_port (Optional[int]): port used by envd container
        envd_dir (Optional[str]): log storage mount path in the envd container
        host_port (Optional[int]): port used by the host, if not specified or equals to 0,
            envd will randomly choose a free port
        host_dir (Optional[str]): log storage mount path in the host
    """
    install.python_packages(["tensorboard"])
    runtime.mount(host_path=host_dir, envd_path=envd_dir)
    runtime.daemon(
        commands=[
            [
                "tensorboard",
                "--logdir",
                envd_dir,
                "--port",
                str(envd_port),
                "--host",
                "0.0.0.0",
            ],
        ]
    )
    runtime.expose(envd_port=envd_port, host_port=host_port, service="tensorboard")

Getting Started πŸš€

Requirements

  • Docker (20.10.0 or above)

Install and bootstrap envd

envd can be installed with pip, or you can download the binary release directly. After the installation, please run envd bootstrap to bootstrap.

pip install --upgrade envd

After the installation, please run envd bootstrap to bootstrap:

envd bootstrap

Read the documentation for more alternative installation methods.

You can add --dockerhub-mirror or -m flag when running envd bootstrap, to configure the mirror for docker.io registry:

envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn

Create an envd environment

Please clone the envd-quick-start:

git clone https://github.com/tensorchord/envd-quick-start.git

The build manifest build.envd looks like:

def build():
    base(os="ubuntu20.04", language="python3")
    # Configure the pip index if needed.
    # config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")

Note that we use Python here as an example but please check out examples for other languages such as R and Julia here.

Then please run the command below to set up a new environment:

cd envd-quick-start && envd up
$ cd envd-quick-start && envd up
[+] ⌚ parse build.envd and download/cache dependencies 2.8s βœ… (finished)
 => download oh-my-zsh                                                    2.8s
[+] πŸ‹ build envd environment 18.3s (25/25) βœ… (finished)
 => create apt source dir                                                 0.0s
 => local://cache-dir                                                     0.1s
 => => transferring cache-dir: 5.12MB                                     0.1s
...
 => pip install numpy                                                    13.0s
 => copy /oh-my-zsh /home/envd/.oh-my-zsh                                 0.1s
 => mkfile /home/envd/install.sh                                          0.0s
 => install oh-my-zsh                                                     0.1s
 => mkfile /home/envd/.zshrc                                              0.0s
 => install shell                                                         0.0s
 => install PyPI packages                                                 0.0s
 => merging all components into one                                       0.3s
 => => merging                                                            0.3s
 => mkfile /home/envd/.gitconfig                                          0.0s
 => exporting to oci image format                                         2.4s
 => => exporting layers                                                   2.0s
 => => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f  0.0s
 => => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717  0.0s
 => => sending tarball                                                    0.4s
envd-quick-start via Py v3.9.13 via πŸ…’ envd
⬒ [envd]❯ # You are in the container-based environment!

Set up Jupyter notebook

Please edit the build.envd to enable jupyter notebook:

def build():
    base(os="ubuntu20.04", language="python3")
    # Configure the pip index if needed.
    # config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")
    config.jupyter()

You can get the endpoint of the running Jupyter notebook via envd envs ls.

$ envd up --detach
$ envd envs ls
NAME                    JUPYTER                 SSH TARGET              CONTEXT                                 IMAGE                   GPU     CUDA    CUDNN   STATUS          CONTAINER ID
envd-quick-start        http://localhost:42779   envd-quick-start.envd   /home/gaocegege/code/envd-quick-start   envd-quick-start:dev    false   <none>  <none>  Up 54 seconds   bd3f6a729e94

Difference between v0 and v1

Note To use the v1 config file, add # syntax=v1 to the first line of your build.envd file.

Features v0 v1
is default for envd<v1.0 βœ… ❌
support dev βœ… βœ…
support CUDA βœ… βœ…
support serving ⚠️ βœ…
support custom base image ⚠️ βœ…
support installing multiple languages ⚠️ βœ…
support moby builder ❌ βœ… (a)

Note (a) To use the moby builder, you will need to create a new context with envd context create --name moby-test --builder moby-worker --use. For more information about the moby builder, check the issue-1693.

Important For more details, check the upgrade to v1 doc.

More on documentation πŸ“

See envd documentation.

Roadmap πŸ—‚οΈ

Please checkout ROADMAP.

Contribute 😊

We welcome all kinds of contributions from the open-source community, individuals, and partners.

Open in Gitpod

Contributors ✨

Thanks goes to these wonderful people (emoji key):

 Friends A.
Friends A.

πŸ“– 🎨
Aaron Sun
Aaron Sun

πŸ““ πŸ’»
Aka.Fido
Aka.Fido

πŸ“¦ πŸ“– πŸ’»
Alex Xi
Alex Xi

πŸ’»
Bingtan Lu
Bingtan Lu

πŸ’»
Bingyi Sun
Bingyi Sun

πŸ’»
Ce Gao
Ce Gao

πŸ’» πŸ“– 🎨 πŸ“†
Frost Ming
Frost Ming

πŸ’» πŸ“–
Guangyang Li
Guangyang Li

πŸ’»
Gui-Yue
Gui-Yue

πŸ’»
Haiker Sun
Haiker Sun

πŸ’»
Ikko Ashimine
Ikko Ashimine

πŸ’»
Isaac
Isaac

πŸ’»
JasonZhu
JasonZhu

πŸ’»
Jian Zeng
Jian Zeng

🎨 πŸ€” πŸ”¬
Jinjing Zhou
Jinjing Zhou

πŸ› πŸ’» 🎨 πŸ“–
Jun
Jun

πŸ“¦ πŸ’»
Kaiyang Chen
Kaiyang Chen

πŸ’»
Keming
Keming

πŸ’» πŸ“– πŸ€” πŸš‡
Kevin Su
Kevin Su

πŸ’»
Ling Jin
Ling Jin

πŸ› πŸš‡
Manjusaka
Manjusaka

πŸ’»
Nino
Nino

🎨 πŸ’»
Pengyu Wang
Pengyu Wang

πŸ“–
Sepush
Sepush

πŸ“–
Shao Wang
Shao Wang

πŸ’»
Siyuan Wang
Siyuan Wang

πŸ’» πŸš‡ 🚧
Suyan
Suyan

πŸ“–
To My
To My

πŸ“–
Tumushimire Yves
Tumushimire Yves

πŸ’»
Wei Zhang
Wei Zhang

πŸ’»
Weixiao Huang
Weixiao Huang

πŸ’»
Weizhen Wang
Weizhen Wang

πŸ’»
XRW
XRW

πŸ’»
Xu Jin
Xu Jin

πŸ’»
Xuanwo
Xuanwo

πŸ’¬ 🎨 πŸ€” πŸ‘€
Yijiang Liu
Yijiang Liu

πŸ’»
Yilong Li
Yilong Li

πŸ“– πŸ› πŸ’»
Yuan Tang
Yuan Tang

πŸ’» 🎨 πŸ“– πŸ€”
Yuchen Cheng
Yuchen Cheng

πŸ› πŸš‡ 🚧 πŸ”§
Yuedong Wu
Yuedong Wu

πŸ’»
Yunchuan Zheng
Yunchuan Zheng

πŸ’»
Zheming Li
Zheming Li

πŸ’»
Zhenguo.Li
Zhenguo.Li

πŸ’» πŸ“–
Zhenzhen Zhao
Zhenzhen Zhao

πŸš‡ πŸ““ πŸ’»
Zhizhen He
Zhizhen He

πŸ’» πŸ“–
cutecutecat
cutecutecat

πŸ’»
dqhl76
dqhl76

πŸ“– πŸ’»
heyjude
heyjude

πŸ’»
jimoosciuc
jimoosciuc

πŸ““
kenwoodjw
kenwoodjw

πŸ’»
li mengyang
li mengyang

πŸ’»
nullday
nullday

πŸ€” πŸ’»
rrain7
rrain7

πŸ’»
tison
tison

πŸ’»
wangxiaolei
wangxiaolei

πŸ’»
wyq
wyq

πŸ› 🎨 πŸ’»
x0oo0x
x0oo0x

πŸ’»
xiangtianyu
xiangtianyu

πŸ“–
xieydd
xieydd

πŸ’»
xing0821
xing0821

πŸ€” πŸ““ πŸ’»
xxchan
xxchan

πŸ“–
zhyon404
zhyon404

πŸ’»
ζ¨ζˆι”΄
ζ¨ζˆι”΄

πŸ’»

This project follows the all-contributors specification. Contributions of any kind welcome!

License πŸ“‹

Apache 2.0

trackgit-views

Directories ΒΆ

Path Synopsis
cmd
envd-sshd
ssh is the CLI running in the container as the sshd.
ssh is the CLI running in the container as the sshd.
e2e
v0
v1
pkg
app
buildkitd/mock
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
driver
The following code is from https://github.com/fsouza/go-dockerclient/ Copyright (c) 2013-2021, go-dockerclient authors
The following code is from https://github.com/fsouza/go-dockerclient/ Copyright (c) 2013-2021, go-dockerclient authors
lang/frontend/starlark/mock
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
progress/compileui/mock
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
ssh

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL