KFserving Installation

Install and configure KFServing

This section will walk you through installation procedure of KFserving such that it is ready to be used with Seldon Deploy.

Install KNative Serving

Install KNATIVE Serving

KNATIVE_SERVING_URL=https://github.com/knative/serving/releases/download
SERVING_VERSION=v0.18.1
SERVING_BASE_VERSON=v0.18.0

kubectl apply -f ${KNATIVE_SERVING_URL}/${SERVING_VERSION}/serving-crds.yaml
kubectl apply -f ${KNATIVE_SERVING_URL}/${SERVING_VERSION}/serving-core.yaml

kubectl apply -f https://github.com/knative-sandbox/net-istio/releases/download/${SERVING_BASE_VERSON}/release.yaml

If you are using seldon core analytics for prometheus then for knative metrics add these annotations:

kubectl annotate -n knative-serving service autoscaler prometheus.io/scrape=true
kubectl annotate -n knative-serving service autoscaler prometheus.io/port=9090

kubectl annotate -n knative-serving service activator-service prometheus.io/scrape=true
kubectl annotate -n knative-serving service activator-service prometheus.io/port=9090

Configure Cluster Local Gateway

KNative requires a Cluster Local Gateway to work properly. This can be added to an existing Istio 1.6.x installation by generating required manifests

cat << EOF > ./local-cluster-gateway.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: empty
  components:
    ingressGateways:
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              name: http2
            - port: 443
              name: https
  values:
    gateways:
      istio-ingressgateway:
        debug: error
EOF

istioctl manifest generate -f local-cluster-gateway.yaml > manifest.yaml

Note the profile: empty line. This ensures that generated manifest only contain gateway related resources.

Once manifest is generated, inspect them, and then use kubectl to apply them

kubectl apply -f manifest.yaml

The above manifest and istiocl command no longer works with istioctl 1.7. For 1.7 we suggest installing istio with

   cat << EOF > ./local-cluster-gateway.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
  addonComponents:
    pilot:
      enabled: true
    prometheus:
      enabled: false
  components:
    ingressGateways:
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https
  values:
    gateways:
      istio-ingressgateway:
        debug: error
EOF

   ./istioctl install -f local-cluster-gateway.yaml

Read more about gateway configuration in Istio’s documentation.

Install KFServing

Clone KFserving repository and checkout the right branch

KFSERVING_VERSION = v0.4.1
KFSERVING_RESOURCE_DIR = kfserving-${KFSERVING_VERSION}

git -C ${KFSERVING_RESOURCE_DIR} fetch origin || git clone https://github.com/kubeflow/kfserving.git ${KFSERVING_RESOURCE_DIR}
#have to checkout commit rather than tag as v0.4.1 yaml is missing in v0.4.1 tag
git -C ${KFSERVING_RESOURCE_DIR} checkout f253dd529ba25c080ca458d8dc36fd0a19bbd473

Install and configure KFserving

cd ${KFSERVING_RESOURCE_DIR}
kubectl apply --validate=false -f ./install/${KFSERVING_VERSION}/kfserving.yaml || true
sleep 3
kubectl patch cm -n kfserving-system inferenceservice-config -p '{"data":{"explainers":"{\n    \"alibi\":{\n        \"image\" : \"docker.io/seldonio/kfserving-alibi\",\n        \"defaultImageVersion\" : \"0.4.1\"\n    }\n}","ingress":"{\n    \"ingressGateway\" : \"istio-system/seldon-gateway\",\n    \"ingressService\" : \"istio-ingressgateway.istio-system.svc.cluster.local\"\n}","logger":"{\n    \"image\" : \"gcr.io/kfserving/logger:v0.4.0\",\n    \"memoryRequest\": \"100Mi\",\n    \"memoryLimit\": \"1Gi\",\n    \"cpuRequest\": \"100m\",\n    \"cpuLimit\": \"1\",\n    \"defaultUrl\": \"http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default\"\n}"}}'
# Create self-signed certs because we dont have cert-manager
./hack/self-signed-ca.sh

Configure Seldon Deploy

Enable KFserving in Seldon Deploy by adding a following section to your deploy-values.yaml files

kfserving:
  protocol: "http"
  enabled: true
Last modified November 26, 2020: be consistent on VERSION not TAG (aefcfcb)