VCS GitOps

GitOps setup with ArgoCD

This page provides a detailed overview of how to set up the GitOps functionality for Seldon Deploy. Setting up GitOps is highly recommended.

Installation of ArgoCD

We start with installing ArgoCD in your cluster. Official documentation can be found here.

Following set of commands will install argocd in the argocd namespace.

ARGOCD_VERSION=v1.6.2

kubectl create namespace argocd || echo "namespace exist"
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml

kubectl rollout status deployment/argocd-application-controller -n argocd
kubectl rollout status deployment/argocd-repo-server -n argocd
kubectl rollout status deployment/argocd-server -n argocd
kubectl rollout status deployment/argocd-redis -n argocd
kubectl rollout status deployment/argocd-dex-server -n argocd

We will also need the argocd command line tool that can be obtained directly from GitHub

ARGOCD_VERSION=v1.6.2
wget -q -O argocd "https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64"
chmod +x ./argocd

Password and ingress configuration

We can set ArgoCD password by patching a corresponding kubernetes secret:

ARGOCDINITIALPASS=12341234
kubectl patch secret \
    -n argocd argocd-secret \
    -p '{"stringData": { "admin.password": "'$(htpasswd -bnBC 10 "" ${ARGOCDINITIALPASS} | tr -d ':\n')'"}}'

If the SSL termination happens at the earlier stage we need to inform ArgoCD that it is suppose to run in the insecure mode

kubectl patch deploy argocd-server \
    -n argocd \
    -p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--insecure"}]' \
    --type json

Assuming that base path will be /argocd we need to patch argocd-server accordingly

kubectl patch deploy argocd-server \
    -n argocd \
    -p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--rootpath"}, {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "/argocd"}]' \
    --type json

Finally, we create VirtualService named argocd-vs.yaml, pointing at the Istio Gateway created for Seldon (see the Istio page):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: argocd-server
  namespace: argocd
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /argocd/
    route:
    - destination:
        host: argocd-server
        port:
          number: 80

and apply it with

kubectl apply -f argocd-vs.yaml

Accessing ArgoCD UI and API endpoints

If you created virtual service in the previous step the ArgoCD UI should be accessible under following URL:

ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "ArgoCD UI URL: http://${ISTIO_INGRESS}/argocd/"

It is also possible to access the UI at localhost:8080 using port-forward method:

kubectl port-forward -n argocd svc/argocd-server 8080:80

To log ArgoCD API using Istio ingress:

./argocd login ${ISTIO_INGRESS}:80 --username admin --password ${ARGOCDINITIALPASS} --insecure --grpc-web-root-path /argocd

Create ArgoCD project and application

In this section we assume that following environmental variables are set

GIT_USER=...      # git user
GIT_TOKEN=...     # git token or password
GIT_REPO_NAME=... # repository name, e.g. seldon-gitops-repo
GIT_REPO_URL=...  # repository URL, e.g. github.com/${GIT_USER}/${GIT_REPO_NAME}

Note that $GIT_REPO_URL should not contain http or http protocol. In addition we will use $namespace to refer to the name of namespace for which we are enabling the GitOps functionality.

The only requirements for the git repository are that it is non-empty (empty README file suffice) and cloneable from within the cluster. The latter can be verified with the following one-liner command

kubectl run --quiet=true -it --rm git-clone --image=radial/busyboxplus:git  --restart=Never -- git clone https://$GIT_USER:$GIT_TOKEN@$GIT_REPO_URL

ArgoCD projects and applications can be created via the ArgoCD UI, argocd CLI tool and through CRDs. As declarative approach guarantees best reproducibility this is the approach that we recommend.

We create ArgoCD project by creating the following AppProject resources named argocd-project.yaml

cat << EOF > ./argocd-project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: seldon
  namespace: argocd
spec:
  description: Seldon Deploy Project
  sourceRepos:
  - https://${GIT_REPO_URL}
  destinations:
  - namespace: ${namespace}
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  roles:
  - name: seldon-admin
    policies:
    - p, proj:seldon:seldon-admin, applications, get, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, create, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, update, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, delete, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, sync, seldon/*, allow
EOF

kubectl apply -f argocd-project.yaml

Similarly we create the ArgoCD Application resource:

ARGO_APP_NAME=seldon-gitops-"${namespace}"

cat << EOF > ./argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ${ARGO_APP_NAME}
  namespace: argocd
spec:
  project: seldon
  destination:
    namespace: ${namespace}
    server: https://kubernetes.default.svc
  source:
    directory:
      recurse: true
    path: ${namespace}
    repoURL: https://${GIT_REPO_URL}
  syncPolicy:
    automated: {}
EOF

kubectl apply -f argocd-application.yaml

Seldon namespace configuration

For a namespace to be used with GitOps a following label and annotation are required

kubectl create ns $namespace || echo "namespace $namespace already exists"

kubectl label namespace $namespace seldon.gitops=enabled --overwrite=true
kubectl annotate namespace $namespace git-repo="https://${GIT_REPO_URL}" --overwrite=true

If you intend to use Batch jobs on the namespace then you’ll need additional config. See the Argo section.

Git credentials configuration

The last required step is to create Git credentials. We configure ArgoCD’s access to the repository with:

./argocd repo add "https://${GIT_REPO_URL}" \
    --username ${GIT_USER} \
    --password ${GIT_TOKEN} \
    --upsert

Seldon Deploy also requires access to the repository. To achieve this we create a secret:

kubectl create secret generic git-creds -n seldon-system \
  --from-literal=username="${GIT_USER}" \
  --from-literal=token="${GIT_TOKEN}" \
  --from-literal=email="${GIT_EMAIL}"  \
  --dry-run=client -o yaml | kubectl apply -f -

and enable GitOps in the deploy-values.yaml values file:

gitops:
  git:
    secret: git-creds
  argocd:
    enabled: true

It is also possible to configure ssh key based access by including id_rsa and known_hosts in the secret:

kubectl create secret generic git-creds -n seldon-system \
  --from-file ~/.ssh/id_rsa --from-file ~/.ssh/known_hosts \
  --from-literal=passphrase="${GIT_SSHKEY_PASSPHRASE}" \
  --from-literal=username="${GIT_USER}" \
  --from-literal=token="${GIT_TOKEN}" \
  --from-literal=email="${GIT_EMAIL}"  \
  --dry-run=client -o yaml | kubectl apply -f -

Configure Webhooks (optional)

Here we show how it’s possible to programmatically set up the webhooks programmatically, however you can also set them up manually from the repo.

Note: In case webhooks are not arriving at its destination it is helpful to enable webhook’s history.

It may be necessary to also select Skip certificate verification option.

Github

First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:

whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -X POST -d "{\"name\": \"web\", \"active\": true, \"events\": [\"*\"], \"config\": {\"url\": \"http://${whIp}/api/git-webhook\", \"content_type\": \"json\" }}" https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks

Then we set up the webhook that will be sending the updates to ArgoCD.

curl -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -X POST -d "{\"name\": \"web\", \"active\": true, \"events\": [\"*\"], \"config\": {\"url\": \"https://${ARGOCDURL}/api/webhook\", \"content_type\": \"json\", \"secret\": \"${ARGOCDINITIALPASS}\", \"insecure_ssl\": \"1\" }}" https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks

Bitbucket

First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:

whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -v -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -d '
{
    "description": "web",
    "events": ["repo:push"],
    "url": "'"http://${whIp}/api/git-webhook"'",
    "active": true
}' https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks

Then we set up the webhook that will be sending the updates to ArgoCD.

curl -v -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -d '
{
    "description": "web",
    "events": ["repo:push"],
    "url": "'"https://${ARGOCDURL}/api/webhook"'",
    "active": true
}' https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks
Last modified November 20, 2020: batch steps using minio browser (a438c0e)