Request Logging

Install KNative Eventing

KNATIVE_EVENTING_URL=https://github.com/knative/eventing/releases/download
EVENTING_VERSION=v0.18.3

kubectl apply --filename ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing-crds.yaml
kubectl apply --filename ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing-core.yaml

kubectl apply --filename ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/in-memory-channel.yaml
kubectl apply --filename ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/mt-channel-broker.yaml

kubectl rollout status -n knative-eventing deployment/imc-controller

Install EFK Stack

Initial Configuration

Copy default fluentd-config (and edit as required)

cp ./seldon-deploy-install/prerequisites-setup/efk/fluentd-values.yaml fluentd-values.yaml

Elasticsearch

Install Elasticsearch using the following script

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"

helm upgrade --install elasticsearch elasticsearch \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --set antiAffinity="soft" \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/elasticsearch/elasticsearch-oss

kubectl rollout status statefulset/elasticsearch-master -n seldon-logs

Fluentd and Kibana

Install fluentd and kibana using the following script

helm upgrade --install fluentd fluentd-elasticsearch \
    --version 8.0.0 \
    --namespace seldon-logs -f fluentd-values.yaml \
    --repo https://kiwigrid.github.io

helm upgrade --install kibana kibana \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP 
    --repo https://helm.elastic.co

kubectl rollout status deployment/kibana-kibana -n seldon-logs

Configure EFK Ingress (Optional)

Kibana

It can be useful to access kibana’s UI without having to port-forward.

To expose kibana externally it needs to have its own path. This means a custom values file:

extraEnvs:
- name: SERVER_BASEPATH
  value: "/kibana"

That should be referenced with -f as an additional parameter on the previous helm install command.

Then create a following VirtualService for Kibana to enable its ingress

cat << EOF > kibana-vs.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kibana
  namespace: seldon-logs
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kibana/
    rewrite:
      uri: /
    route:
    - destination:
        host: kibana-kibana
        port:
          number: 5601
EOF

Kubectl apply -f kibana-vs.yaml

Configure KNative Event broker

Create knative event broker that will handle the logging.

kubectl create -f - <<EOF
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: default
  namespace: seldon-logs
EOF

Seldon Core and Deploy Configuration

For Seldon Core add to your core-values.yaml following options

executor:
  requestLogger:
    defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default"

For Seldon Deploy add to your deploy-values.yaml following options

requestLogger:
  create: true

Custom Request Logger

It’s possible for you to add your own custom request logger, with any custom loggic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core helm chart’s executor.requestLogger.defaultEndpoint.

Below we show an example of how you would do this for our non-knative default request logger

Running without KNative

It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.

Important: you need to make sure that you are running a request logger in every namespace where you plan to deploy seldon core models

apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-request-logger
  namespace: default
  labels:
    app: seldon-request-logger
spec:
  replicas: 2
  selector:
    matchLabels:
      app: seldon-request-logger
  template:
    metadata:
      labels:
        app: seldon-request-logger
    spec:
      containers:
        - name: user-container
          image: docker.io/seldonio/seldon-request-logger:0.2.0
          imagePullPolicy: Always
          env:
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-request-logger
  namespace: default
spec:
  selector:
    app: seldon-request-logger
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Configure Seldon Core to use that endpoint

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

        - name: EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT
          value: "http://seldon-request-logger."

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.

Overriding Request Logger Endpoint for specific Seldon Deployment

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

      logger:
        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all.

The url is where this should be logged to. There’s a similar example for KFServing.