V-function

Function Introduction

V-Function is a set of multiple visual recognition firmware developed by the M5Stack team for V-series devices. Based on different firmware functions (object tracking, motion detection, etc.), users can quickly build visual recognition functions. This tutorial will introduce you to how to burn firmware to your device and call it through UIFlow graphical programming.

Driver Installation

Burn Firmware

UIFlow Reference

Import Extensions

Notes

Motion Detection

• Initialize

  • Initialize

• Set Change Rate Threshold

  • Set the change rate threshold: When the change amount is less than this value of pixels, it will not be considered as a change, and its change amount will not be included in the frame change rate.

• Set Detection Mode

  • dynamic: Dynamic detection mode, continuously takes pictures and compares the changes between the previous and current frames.
  • static: Static detection mode, after execution, it will take and save a base picture. Subsequent frames will be continuously compared with this picture. If you need to take a new base picture, you need to switch back to dynamic detection mode first, and then execute static detection mode setting again.

• Get Frame Change Rate

  • Frame change rate: the change amount of pixels between the previous and current frames. Suppose there are 2 pixels that have changed, pixel A has changed by 27, pixel B has changed by 10, then this value is 27+10=37. The value is the sum of the difference in the R.G.B components of the two pixels.

• Get Maximum Change Rate

  • Maximum change rate: the change amount of the most intense pixel.

• Set Scan Interval X Axis Y Axis

  • Set the scan interval on the x-axis and y-axis.

• Get Boundary Box Number

  • Get the number of boundary boxes generated by pixel changes.

• Get X Number Boundary Box Information

  • Return detailed information of the Xth boundary box as a list, including the number of changed pixels in the boundary box, the x-axis coordinate

of the boundary box, the y-axis coordinate of the boundary box, the width of the boundary box, and the height of the boundary box.

• Click here to download the UIFlow example

Motion Detection - Data Packet Format

Return JSON

{
    "FUNC": "MOTION DETECT V1.0",
    "DIFF TOTAL": 10000, // Frame change rate
    "DIFF MAX": 75, // Maximum change rate
    "TOTAL": 3, // Number of boundary boxes
    "0": {
        "x": 45,
        "y": 18,
        "w": 126,
        "h": 72,
        "area": 342 // Number of changed pixels in this boundary box
    },
    "1": {
        "x": 0,
        "y": 169,
        "w": 130,
        "h": 24,
        "area": 173
    },
    "2": {
        "x": 39,
        "y": 204,
        "w": 276,
        "h": 34,
        "area": 141
    }
}

Setting JSON

{
    "MOTION DETECT": 1.0, // Function flag, cannot be omitted
    "mode": "COMPUTE_MODE_STATIC", // Optional "COMPUTE_MODE_STATIC" static detection mode or "COMPUTE_MODE_DYNAMIC" dynamic detection mode
    "thr_w": 20, // Optional Width threshold of the boundary box,[3,200]
    "thr_h": 20, // Optional Height threshold of the boundary box,[3,200]
    "stepx": 1, // Optional X-axis scan interval,[0, 40], set to 0 to disable boundary box detection
    "stepy": 2, // Optional Y-axis scan interval,[0, 40], set to 0 to disable boundary box detection
    "delta": 20, // Optional Change rate threshold,[0, 99]
     "merge": 10 // Optional Boundary box merge threshold,[0, 40]
}

Object Tracking

• Initialize

  • Initialize

• Set Tracking Box Coordinates x y Tracking Box Width Height

  • Set the target selection box, parameters are the position of the current target on the image (select a target with significant color features as much as possible)

• Get Tracking Box Trajectory Details

  • Read the coordinates of the selected target on the image, and the return value is in the form of a list, which includes the x, y coordinates of the upper left corner of the selection box, and the width and height of the selection box.

• Click here to download the UIFlow example

Object Tracking - Data Packet Format

Return JSON

{
    "FUNC": "TARGET TRACKER V1.0",
    "x": 282,
    "y": 165,
    "w": 13,
    "h": 15
}

Setting JSON

{
    "TARGET TRACKER": " V1.0",
    "x": 282, //xywh cannot be omitted
    "y": 165,
    "w": 13,
    "h": 15
}

Color Tracking

• Initialize

  • Initialize.

• Set Target Color L Threshold Min 0 L Threshold Max 0 A Threshold Min A Threshold Max B Threshold Min B Threshold Max

  • Set the LAB threshold for tracking (LAB color space values, colors outside this range will be filtered).

• Set Scan Interval X Axis Y Axis

  • Set the scan interval on the X-axis and Y-axis, [0, 40]. Set to 0 to disable boundary box detection.

• Set Boundary Box Merge Threshold

  • Set the boundary box merge threshold.

• Set Boundary Box Width Threshold 0 Height Threshold 0

  • Set the boundary box width threshold 0 and height threshold 0.

• Get Boundary Box Number

  • Get the number of boundary boxes.

• Get Boundary Box Details

  • Get the details of the boundary box, including the number of changed pixels in the boundary box, the x-axis coordinate of the boundary box, the y-axis coordinate of the boundary box, the width of the boundary box, and the height of the boundary box.

Setting LAB Thresholds

Download LAB Color Picking Tool

• Click here to download the UIFlow example

Color Tracking - Data Packet Format

Return JSON

{
    "FUNC": "COLOR TRACKER V1.0",
    "TOTAL": 3, // Number of boundary boxes
    "0": {
        "x": 45,
        "y": 18,
        "w": 126,
        "h": 72,
        "area": 342 // Number of changed pixels in this boundary box
    },
    "1": {
        "x": 0,
        "y": 169,
        "w": 130,
        "h": 24,
        "area": 173
    },
    "2": {
        "x": 39,
        "y": 204,
        "w": 276,
        "h": 34,
        "area": 141
    }
}

Setting JSON

{
    "COLOR TRACKER": 1.0, // Function flag, cannot be omitted
    "thr_w": 20, // Optional Width threshold of the boundary box,[3,200]
    "thr_h": 20, // Optional Height threshold of the boundary box,[3,200]
    "stepx": 1, // Optional X-axis scan interval,[0, 40], set to 0 to disable boundary box detection
    "stepy": 2, // Optional Y-axis scan interval,[0, 40], set to 0 to disable boundary box detection
    "merge": 10, // Optional Boundary box merge threshold,[0, 40]
    "Lmin": 0, // Optional L threshold lower limit [0, 100]
    "Lmax": 0, // Optional L threshold upper limit [0, 100]
    "Amin": 0, // Optional A threshold lower limit [0, 255]
    "Amax": 0, // Optional A threshold upper limit [0, 255]
    "Bmin": 0, // Optional B threshold lower limit [0, 255]
    "Bmax": 0, // Optional B threshold upper limit [0, 255]
}

Face Detection

• Initialize

  • Initialize

• Get Number of Faces

  • Read the number of faces recognized.

• Get Details of the xth Face

  • Read the details data of the specified number of faces, returned in list format, including the coordinates of the face box, length, width, and confidence level.

• Click here to download the UIFlow example

人脸识别-数据包格式

回传JSON

{
   "FUNC": "FACE DETECT",  // 功能说明
   "count": 3,   // 识别到的人脸数量
   "2": {  // 人脸编号
      "x": 97,    // ROI
      "y": 26,
      "w": 64,
      "h": 86,
      "value": 0.859508,  // 置信率
       "classid": 0,  
       "index": 2,
       "objnum": 3
        },
    "1": {
       "x": 70,
       "y": 157,
       "w": 38,
       "h": 63,
       "value": 0.712100,
       "classid": 0,
       "index": 1,
       "objnum": 3
       },
    "0": {
       "x": 199,
       "y": 145,
       "w": 31,
       "h": 40,
       "value": 0.859508,
       "classid": 0,
       "index": 0,
       "objnum": 3
       }
    }

Face Recognition - Data Packet Format

JSON Response

{
   "FUNC": "FACE DETECT",
   "count": 3,
   "2": {
      "x": 97,
      "y": 26,
      "w": 64,
      "h": 86,
      "value": 0.859508,
      "classid": 0,
      "index": 2,
      "objnum": 3
   },
   "1": {
      "x": 70,
      "y": 157,
      "w": 38,
      "h": 63,
      "value": 0.712100,
      "classid": 0,
      "index": 1,
      "objnum": 3
   },
   "0": {
      "x": 199,
      "y": 145,
      "w": 31,
      "h": 40,
      "value": 0.859508,
      "classid": 0,
      "index": 0,
      "objnum": 3
   }
}

QR Code Recognition

• Initialize

  • Initialization

• Get QR Code Information

  • Read the content of the recognized QR code

• Get QR Code Version

  • Read the version of the recognized QR code

• Click here to download UIFlow example

JSON Response

{
   "count": 1,
   "FUNC": "FIND QRCODE",
   "0": {
      "x": 57,
      "y": 16,
      "w": 197,
      "h": 198,
      "payload": "m5stack",
      "version": 1,
      "ecc_level": 1,
      "mask": 2,
      "data_type": 4,
      "eci": 0
   }
}

Barcode Recognition

• Initialize

  • Initialization

• Get Recognized Barcode Content

  • Read the content of the recognized barcode

• Get Recognized Barcode Rotation Angle

  • Read the rotation angle of the recognized barcode

• Get Recognized Barcode Type

  • Read the type of the recognized barcode

• Get Recognized Barcode Position Information

  • Read the selected coordinates, length, and width of the recognized barcode, returning a list

• Click here to download UIFlow example

JSON Response

{
    "0": {
        "x": 62,
        "y": 90,
        "w": 100,
        "h": 45,
        "payload": "123",
        "type": 15,
        "rotation": 0.000000,
        "quality": 28
    },
    "count": 1,
    "FUNC": "FIND BARCODE"
}

DataMatrix Code Recognition

• Initialize

  • Initialization

• Get Data Matrix Code Information

  • Read the content of the recognized DataMatrix code

• Get Data Matrix Code Rotation Angle

  • Read the rotation angle of the recognized DataMatrix code

• Get Data Matrix Code Position Information

  • Read the selected coordinates, length, and width of the recognized DataMatrix code, returning a list

• Click here to download UIFlow example

JSON Response

{
    "0": {
        "x": 20,
        "y": 116,
        "w": 96,
        "h": 96,
        "payload": "m5stack",
        "rotation": 1.588250,
        "rows": 16,
        "columns": 16,
        "capacity": 12,
        "padding": 1
    },
    "count": 1,
    "FUNC": "FIND DATAMATRIX"
}

AprilTag Code Recognition

• Initialize

  • Initialize

• Get Rotation Angle of AprilTag Code

  • Returns the rotation angle of the AprilTag in radians (int)

• Get Coordinates of AprilTag Code

  • Reads the selected coordinates, center coordinates, length, and width of the recognized AprilTag code, the return value is a list.

• Get Movement Units of AprilTag Code

  • Reads the position offset of the AprilTag code.

• Click here to download UIFlow example

JSON Feedback

{
    "0": {
        "x": 71,
        "y": 5,
        "w": 85,
        "h": 88,
        "id": 1,
        "family": 16,// AprilTag category
        "cx": 115,
        "cy": 49,
        "rotation": 6.219228,// Rotation angle of AprilTag in radians (int).
        "decision_margin": 0.451959,// Color saturation of AprilTag matching (values from 0.0 to 1.0), where 1.0 is optimal.
        "hamming": 0,// Acceptable number of bit errors for AprilTag
        "goodness": 0.000000, // Color saturation of AprilTag image
        "x_translation": 0.868200, // Number of units to move the image left or right after rotation
        "y_translation": 0.245313,// Number of units to move the image up or down after rotation
        "z_translation": -2.725188,// Amount scaled by the image. Default is 1.0
        "x_rotation": 3.093776,// Degrees to rotate the image around the x-axis in the frame buffer
        "y_rotation": 0.065489,// Degrees to rotate the image around the y-axis in the frame buffer
        "z_rotation": 6.219228 // Degrees to rotate the image around the z-axis in the frame buffer
    },
    "count": 1,
    "FUNC": "FIND APRILTAG"
}

JSON for Setting Recognition Mode

{
    "FIND CODE": 1.0,
    "mode":"DATAMATRIX" // Recognition mode, options: QRCODE, APRILTAG, DATAMATRIX, BARCODE
}

Custom Tag Recognition

• Initialize

  • Initialize

• Get Current Number of Recognized Tag Cards

  • Number of tag cards recognized in the current image

• Get Binary String of Recognition Result

  • Binary data string of recognition result, when there are multiple cards, pass the index to select different card contents.

• Get uint64_t Type Code Content

  • uint64_t type content binary code, maximum encoding of 64 bits (8 x 8) TAG.

• Get Positional Information of Tags

  • Coordinates and length-width information of tag cards

00000000      00000000              
00111100      00@@@@00        @@@@  
01000010      0@0000@0       @    @ 
01000010  ->  0@0000@0  ->   @    @ 
01011010      0@0@@0@0       @ @@ @ 
01000010      0@0000@0       @    @ 
01000010      0@0000@0       @    @ 
00000000      00000000              

Tag Reader Example

• Download UIFlow Example Here

Custom Tag Recognition - Data Packet Format

JSON Feedback

{
    "FUNC": "TAG READER V2.0",
    "TOTAL": 1,
    "0": {
        "x": 113,
        "y": 65,
        "w": 117,
        "h": 105,
        "p0x": 113, // Coordinates of the 4 vertices of the TAG
        "p0y": 77,
        "p1x": 211,
        "p1y": 65,
        "p2x": 230,
        "p2y": 156,
        "p3x": 127,
        "p3y": 170,
        "rotation": 8, // Relative rotation angle of the TAG
        "rows": 8, // Number of rows of the TAG (excluding the locator frame)
        "columns": 8, // Number of columns of the TAG (excluding the locator frame)
        "size": 64, // Length of the actual content of the TAG, this value = number of rows * number of columns = (rows) * (columns)
        "code": "0x003C42425A424200", // uint64_t type content binary code, maximum encoding of 64 bits (8 x 8) TAG
        "binstr": "0000000000111100010000100100001001011010010000100100001000000000" // Binary data string, this value can encode TAGs of any length and width
    }
}

Line Tracking

• Initialize

  • Initialize

• Get Line Offset Angle

  • Obtain the angle offset of the line

• Set Tracking Target Color L Threshold Min L Threshold Max A Threshold Min A Threshold Max B Threshold Min B Threshold Max

  • Set LAB thresholds for tracking (LAB color space values, colors outside this range will be filtered)

• Set Line Weight 0 Region Weight, Line Weight 1 Region Weight, Line Weight 2 Region Weight

  • Set line region weights: the three weights correspond to the contribution values of the angles in the three regions in the image. For example, setting a larger value for weight_2 will make the angle change more pronounced when turning.

Setting LAB Thresholds

• Download UIFlow Example Here

Line Tracking - Data Packet Format

JSON Feedback

{
    "FUNC": "LINE TRACKER V1.0",
    "angle": 3.8593475818634033 // Angle of turning the car
}

Setting JSON

{
    "LINE  TRACKER": 1.0, // Function flag, cannot be omitted
    "thr_w": 20, // Optional, width threshold of the boundary box, [3,200]
    "thr_h": 20, // Optional, length threshold of the boundary box, [3,200]
    "stepx": 1, // Optional, X scan interval, [0, 40], set to 0 to disable boundary box detection
    "stepy": 2, // Optional, Y scan interval, [0, 40], set to 0 to disable boundary box detection
    "merge": 10, // Optional, boundary box merge threshold, [0, 40]
    "Lmin": 0, // Optional, L threshold lower limit [0, 100]
    "Lmax": 0, // Optional, L threshold upper limit [0, 100]
    "Amin": 0, // Optional, A threshold lower limit [0, 255]
    "Amax": 0, // Optional, A threshold upper limit [0, 255]
    "Bmin": 0, // Optional, B threshold lower limit [0, 255]
    "Bmax": 0, // Optional, B threshold upper limit [0, 255]
    "weight_0": 0.1, // Optional, weight
    "weight_1": 0.3, // Optional, weight
    "weight_2": 0.7  // Optional, weight
}

More Content

GitHub