Sedna/examples/incremental_learning_modelbox/mnist

• Using Incremental Learning Job In Mnist
• • Mnist Experiment
  • • Prepare Model
    • Prepare dataset
    • Prepare Image
    • Project creation and running
    • Check Incremental Learning Job

Using Incremental Learning Job In Mnist

This document introduces how to use the inference of incremental learning job in Mnist. Using the incremental learning inference job, our application can automatically retrains, evaluates, and updates models based on the data generated at the edge.

Mnist Experiment

Prepare Model

Link：https://pan.baidu.com/s/1Gi5BJ_NQzqj66R8N5OXPzA 
Extract code：OSPP

Prepare dataset

Link：https://pan.baidu.com/s/1Gi5BJ_NQzqj66R8N5OXPzA 
Extract code：OSPP

Prepare Image

This example uses the image:

ymh13383894400/mnist-new:v1

This image is generated by the script used for creating training, eval and inference worker.

Project creation and running

Create a Mnist project

├─flowunit：# Flowunit directory
│  ├─mnist_preprocess：# Preprocessing functional unit
│  ├─mnist_infer：# TensorFlow Inference Functional Unit
│  ├─mnist_response：# HTTP responses construct functional units
└─graph：# Flowchart catalog
│  ├─mnist.toml：# Inference flowchart
│  └─test_mnist.py # Inference python file

create the job

WORKER_NODE="edge-node1"
INFER_NODE="edge-node2"

• Create Dataset

  kubectl create -f - <<EOF
  apiVersion: sedna.io/v1alpha1
  kind: Dataset
  metadata:
    name: incremental-dataset
  spec:
    url: "/data/train_data.txt"
    format: "txt"
    nodeName: $WORKER_NODE
  EOF
  

• Create Initial Model to simulate the initial model in incremental learning scenario.

  kubectl create -f - <<EOF
  apiVersion: sedna.io/v1alpha1
  kind: Model
  metadata:
    name: initial-model
  spec:
    url : "/models/base_model"
    format: "ckpt"
  EOF
  

• Create Deploy Model

  kubectl create -f - <<EOF
  apiVersion: sedna.io/v1alpha1
  kind: Model
  metadata:
    name: deploy-model
  spec:
    url : "/models/deploy_model/saved_model.pb"
    format: "pb"
  EOF
  

• Start The Incremental Learning Job

The inference part uses the modelbox image to run the pod.

IMAGE=ymh13383894400/mnist-new:v1

kubectl create -f - <<EOF
apiVersion: sedna.io/v1alpha1
kind: IncrementalLearningJob
metadata:
  name: Mnist-demo
spec:
  initialModel:
    name: "initial-model"
  dataset:
    name: "incremental-dataset"
    trainProb: 0.8
  trainSpec:
    template:
      spec:
        nodeName: $WORKER_NODE
        containers:
          - image: $IMAGE
            name:  train-worker
            imagePullPolicy: IfNotPresent
            args: ["train.py"]
    trigger:
      checkPeriodSeconds: 60
      timer:
        start: 02:00
        end: 20:00
      condition:
        operator: ">"
        threshold: 500
        metric: num_of_samples
  evalSpec:
    template:
      spec:
        nodeName: $WORKER_NODE
        containers:
          - image: $IMAGE
            name:  eval-worker
            imagePullPolicy: IfNotPresent
            args: ["eval.py"]
  deploySpec:
    model:
      name: "deploy-model"
      hotUpdateEnabled: true
      pollPeriodSeconds: 60
    trigger:
      condition:
        operator: ">"
        threshold: 0.1
        metric: precision_delta
    hardExampleMining:
      name: "IBT"
      parameters:
        - key: "threshold_img"
          value: "0.9"
        - key: "threshold_box"
          value: "0.9"
    template:
      spec:
        nodeName: $INFER_NODE
        containers:
        - image: $IMAGE
          name:  infer-worker
          imagePullPolicy: IfNotPresent
          args: ["test_mnist.py"]
          volumeMounts:
          - name: localvideo
            mountPath: /video/
          - name: hedir
            mountPath: /he_saved_url
          resources:  # user defined resources
            limits:
              memory: 2Gi
        volumes:   # user defined volumes
          - name: localvideo
            hostPath:
              path: /incremental_learning/video/
              type: DirectoryOrCreate
          - name: hedir
            hostPath:
              path:  /incremental_learning/he/
              type: DirectoryOrCreate
  outputDir: "/output"
EOF

Check Incremental Learning Job

Query the service status:

kubectl get incrementallearningjob Mnist-detection-demo