Model Accuracy Metrics with Iris Classifier

Iris is the family in the flower which contains the several species such as the setosa, versicolor, virginica,etc. This demo is based on Iris classification model based on flower properties like Sepal length, Sepal width, Petal length, Petal width. Here we will :

• Launch an iris classifier model
• Setup an metrics server for this particular model
• Send a request to get a iris classification
• Send feedback requests to get a gather accuracy metrics

Create Model

Use the following model url with seldon runtime. Set the protocol to ‘seldon’:

gs://seldon-models/sklearn/iris

Setup Metrics Server

Setup an metrics server with model name multiclassserver using the default settings (which sets Reply URL as seldon-request-logger in the logger’s default namespace - change if you modified this at install time) and storage URI as follows:

adserver.cm_models.multiclass_numeric.MultiClassNumeric

And set the protocol as Seldon Feedback for port 8080.

Make Predictions

Run a single prediction using the ndarray payload format. Make a couple of these requests at random using the predict tool in the UI.

{
	"data": {
		"names": ["Sepal length", "Sepal width", "Petal length", "Petal Width"],
		"ndarray": [
			[6.8, 2.8, 4.8, 1.4]
		]
	}
}

The prediction response is versicolor.

{
	"data": {
		"names": [
			"t:0",
			"t:1",
			"t:2"
		],
		"ndarray": [
			[
				0.008074020139120223,
				0.7781601484223125,
				0.21376583143856714
			]
		]
	},
	"meta": {}
}

Send Feedback

As we saw the prediction response was versicolor. In numeric form the response is,

{
  "data": {
    "ndarray": [
      1
    ]
  }
}

Now prepare a feedback with the prediction response and the truth for the numeric metrics server as follows:

{
	"response": {
		"data": {
			"ndarray": [
				1
			]
		}
	},
	"truth": {
		"data": {
			"ndarray": [
				1
			]
		}
	}
}

Now send a feedback request as following with the auth token from the curl form in the predict tool. This feedback represents a case of true positive.

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k -H "X-Auth-Token: $AUTH_TOKEN" -H "Content-Type: application/json" https://$CLUSTER_IP/seldon/seldon/iris-classifier/api/v0.1/feedback -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'

Also prepare a feedback with the prediction response and the truth for the numeric metrics server such that the truth is different from the prediction.

{
	"response": {
		"data": {
			"ndarray": [
				1
			]
		}
	},
	"truth": {
		"data": {
			"ndarray": [
				0
			]
		}
	}
}

Now send a feedback request as following with the auth token from the curl form in the predict tool. This feedback represents a case of false positives and negetives.

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k -H "X-Auth-Token: $AUTH_TOKEN" -H "Content-Type: application/json" https://$CLUSTER_IP/seldon/seldon/iris-classifier/api/v0.1/feedback -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'

Monitor accuracy metrics on the Monitor Screen

Go to the monitor screen’s the accuracy metrics tab to view all the metrics. Set the time range to view the metrics. You can see metrics like accuracy, precision, recall and sepcificity here. Notice the drop in accuracy metrics after the false feedback was received.

Submit batch feedback using Batch Processor component

Now we will submit a feedback as a batch using the Batch Processor component.

We will use two files, each containing 10k feedback instances:

• 90% success rate feedback file
• 40% success rate feedback file

Ww need to upload the files to MinIO’s data bucket. For details on interacting with MinIO UI please see Batch Demo.

Once files are in S3 bucket, we go to Batch Requests screen using following button available at the main model screen

and submit batch request using following form values for both feedback-input-90.txt and feedback-input-40.txt files

Input Data Location: s3://data/feedback-input-40.txt
Output Data Location: s3://data/output-data-{{workflow.name}}.txt
Number of Workers: 15
Number of Retries: 3
Method: Feedback
Transport Protocol: REST
Input Data Type: Raw Data
Object Store Secret Name: seldon-job-secret

Now go to the monitor view and observe how metrics value evolve over time.