PaddlePaddle Cloud
PaddlePaddle Cloud is a Distributed Deep-Learning Cloud Platform for both cloud
providers and enterprises.
PaddlePaddle Cloud use Kubernetes as it's backend job
dispatching and cluster resource management center. And use PaddlePaddle
as the deep-learning frame work. Users can use web pages or command-line tools
to submit their deep-learning training jobs remotely to make use of power of
large scale GPU clusters.
Using Command-line To Submit Cloud Training Jobs
中文手册
English tutorials(comming soon...)
Deploy PaddlePaddle Cloud
Pre-Requirements
- PaddlePaddle Cloud use kubernetes as it's backend core, deploy kubernetes cluster
using Sextant or any tool you like.
Run on kubernetes
-
Build Paddle Cloud Docker Image
# build docker image
git clone https://github.com/PaddlePaddle/cloud.git
cd cloud/paddlecloud
docker build -t [your_docker_registry]/pcloud .
# push to registry so that we can submit paddlecloud to kubernetes
docker push [your_docker_registry]/pcloud
-
We use volume to mount MySQL data,
cert files and settings, in k8s/
folder we have some samples for how to mount
stand-alone files and settings using hostpath. Here's
a good tutorial of creating kubernetes certs: https://coreos.com/kubernetes/docs/latest/getting-started.html
- create data folder on a Kubernetes node, such as:
mkdir -p /home/pcloud/data/mysql
mkdir -p /home/pcloud/data/certs
- Copy Kubernetes CA files (ca.pem, ca-key.pem, ca.srl) to
/home/pcloud/data/certs
folder
- Copy Kubernetes admin user key (admin.pem, admin-key.pem) to
/home/pcloud/data/certs
folder
- Optianal: copy CephFS Key file(admin.secret) to
/home/pcloud/data/certs
folder
- Copy
paddlecloud/settings.py
file to /home/pcloud/data
folder
-
Configure cloud_deployment.yaml
spec.template.spec.containers[0].volumes
change the hostPath
which match your data folder.
spec.template.spec.nodeSelector.
, edit the value kubernetes.io/hostname
to host which data folder on.You can use kubectl get nodes
to list all the Kubernetes nodes.
-
Configure settings.py
- Add your domain name to
ALLOWED_HOSTS
.
- Configure
DATACENTERS
to your backend storage, supports CephFS and HostPath currently.
You can use HostPath mode to make use of shared file-systems like "NFS".
-
Configure cloud_ingress.yaml
is your kubernetes cluster is using ingress
to proxy HTTP traffics, or you can configure cloud_service.yaml
to use NodePort
- if using ingress, configure
spec.rules[0].host
to your domain name
-
Deploy cloud on Kubernetes
kubectl create -f k8s/cloud_deployment.yaml
kubectl create -f k8s/cloud_service.yaml
kubectl create -f k8s/cloud_ingress.yaml
(optianal)
To test or visit the website, find out the kubernetes ingress IP
addresses, or the NodePort.
Then open your browser and visit http://, or
http://:
Run locally without docker
virtualenv paddlecloudenv
# enable the virtualenv
source paddlecloudenv/bin/activate
To run for the first time, you need to:
npm install
pip install -r requirements.txt
./manage.py migrate
./manage.py loaddata sites
npm run dev
Browse to http://localhost:8000/
If you are starting the server for the second time, just run:
./manage.py runserver
If you want to use mail
command to send confirmation emails, change the below settings:
EMAIL_BACKEND = 'django_sendmail_backend.backends.EmailBackend'
You may need to use hostNetwork
for your pod when using mail command.
Or you can use django smtp bindings just refer to https://docs.djangoproject.com/en/1.11/topics/email/