Hand Recognition Example

1.Hand Recognition

Before You Start:

• Overview And Setup
• Hand Recognition Function Guide

1.1 Hand Gesture Recognition - Output Relevant Data

It can detect hand gestures within the HuskyLens 2 field of view and acquire gesture-related data. Readable data includes: the total number of gestures detected in the frame, the ID of the gesture closest to the center of the HuskyLens 2 camera frame, and the ID of the first detected gesture.

Example Program:

from unihiker import GUI
from pinpong.board import Board
from dfrobot_huskylensv2 import *


u_gui=GUI()
Board().begin()
huskylens = HuskylensV2_I2C()
huskylens.knock()
huskylens.switchAlgorithm(ALGORITHM_HAND_RECOGNITION)
line1=u_gui.draw_text(text="Hand ID1 count:",x=0,y=0,font_size=15, color="#0000FF")
line2=u_gui.draw_text(text="ID1 name:",x=0,y=40,font_size=15, color="#0000FF")
line3=u_gui.draw_text(text="First ID1 hand",x=0,y=80,font_size=15, color="#0000FF")
line4=u_gui.draw_text(text="Coords:",x=0,y=120,font_size=15, color="#0000FF")
while True:
    huskylens.getResult(ALGORITHM_HAND_RECOGNITION)
    if (huskylens.available(ALGORITHM_HAND_RECOGNITION)):
        line1.config(text=(str("Hand ID1 count:") + str((huskylens.getCachedResultNumByID(ALGORITHM_HAND_RECOGNITION, 1)))))
        line2.config(text=(str("ID1 name:") + str((huskylens.getCachedResultByID(ALGORITHM_HAND_RECOGNITION, 1).name if huskylens.getCachedResultByID(ALGORITHM_HAND_RECOGNITION, 1) else -1))))
        line4.config(text=(str("Coords:") + str((str((huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1).xCenter if huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1) else -1)) + str((str(",") + str((huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1).yCenter if huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1) else -1))))))))

Click 'Run' in Mind+ and wait for the program to upload. HuskyLens 2 will enter the Hand Gesture Recognition function. Point the HuskyLens 2 camera at the gesture in the frame to learn it. For detailed instructions on how to learn gestures, refer to: Hand Recognition.

Once learning is complete, aim at the gesture and observe the output results on the screen.

Output Result: it can output the total number of detected gestures. All detected gestures will be counted (i.e., framed by a box), whether they have been learned or not. It can output corresponding data as required, such as the second line, which outputs the total number of learned gestures. The third and fourth lines output the specified gesture ID. Learned gestures are assigned IDs in learning order, while unlearned gestures have an ID of 0.

1.2 Get Data for a Specific Gesture in the Frame

It can acquire keypoint data for a specific gesture, such as its ID, Name, fingers, and wrist. Detailed data includes: Gesture ID, Gesture Name, X and Y coordinates of the gesture's center point, Width, Height, Wrist X/Y coordinates, and the X/Y coordinates for the base, joint, and tip of each finger. For details, please see the Hand Gesture Recognition Blocks Description.

The example program below shows how to get the X/Y coordinate data for the wrist and the tips of all five fingers for the gesture closest to the center of the camera frame. This data can also be obtained for unlearned gestures.

from unihiker import GUI
from pinpong.board import Board
from dfrobot_huskylensv2 import *


u_gui=GUI()
Board().begin()
huskylens = HuskylensV2_I2C()
huskylens.knock()
huskylens.switchAlgorithm(ALGORITHM_HAND_RECOGNITION)
line1=u_gui.draw_text(text="Center hand ID:",x=0,y=0,font_size=10, color="#0000FF")
line2=u_gui.draw_text(text="Wrist coords:",x=0,y=40,font_size=10, color="#0000FF")
line3=u_gui.draw_text(text="Thumb tip coords:",x=0,y=80,font_size=10, color="#0000FF")
line4=u_gui.draw_text(text="Index finger tip coords:",x=0,y=120,font_size=10, color="#0000FF")
line5=u_gui.draw_text(text="Middle finger tips coords:",x=0,y=160,font_size=10, color="#0000FF")
line6=u_gui.draw_text(text="Ring finger tips coords",x=0,y=200,font_size=10, color="#0000FF")
line7=u_gui.draw_text(text="Pinky finger tip coords:",x=0,y=240,font_size=10, color="#0000FF")
while True:
    huskylens.getResult(ALGORITHM_HAND_RECOGNITION)
    if (huskylens.available(ALGORITHM_HAND_RECOGNITION)):
        line1.config(text=(str("Center hand ID:") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).ID if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))
        line2.config(text=(str("Wrist coords:") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).wrist_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).wrist_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))
        line3.config(text=(str("Thumb tip coords:") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).thumb_tip_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).thumb_tip_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))
        line4.config(text=(str("Index finger tip coords:") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).index_finger_tip_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).index_finger_tip_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))
        line5.config(text=(str("Middle finger tips coords:") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).middle_finger_tip_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).middle_finger_tip_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))
        line6.config(text=(str("Ring finger tips coords") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).ring_finger_tip_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).ring_finger_tip_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))
        line7.config(text=(str("Pinky finger tip coords:") + str((str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).pinky_finger_tip_x if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1)) + str((str("，") + str((huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION).pinky_finger_tip_y if huskylens.getCachedCenterResult(ALGORITHM_HAND_RECOGNITION) else -1))))))))

Output Result: As shown below, after running the program, the screen displays the Gesture ID and the keypoint data for that gesture. Since this gesture has not been learned, the Gesture ID is 0.

In addition to the data above, more data for a specific gesture can be acquired. For example, determining if a specific gesture is in the frame, the name of a specific gesture, or the count of identical gestures in the frame. When multiple identical gestures appear, it's possible to specify and acquire parameters for one of them, including Name, gesture center X/Y coordinates, Width, Height, fingertip coordinates, wrist coordinates, and more.

Example Program:

from unihiker import GUI
from pinpong.board import Board
from dfrobot_huskylensv2 import *


u_gui=GUI()
Board().begin()
huskylens = HuskylensV2_I2C()
huskylens.knock()
huskylens.switchAlgorithm(ALGORITHM_HAND_RECOGNITION)
line1=u_gui.draw_text(text="Hand ID1 count:",x=0,y=0,font_size=10, color="#0000FF")
line2=u_gui.draw_text(text="ID1 name:",x=0,y=40,font_size=10, color="#0000FF")
line3=u_gui.draw_text(text="First ID1 hand:",x=0,y=80,font_size=10, color="#0000FF")
line4=u_gui.draw_text(text="Coords:",x=0,y=120,font_size=10, color="#0000FF")
while True:
    huskylens.getResult(ALGORITHM_HAND_RECOGNITION)
    if (huskylens.available(ALGORITHM_HAND_RECOGNITION)):
        line1.config(text=(str("Hand ID1 count:") + str((huskylens.getCachedResultNumByID(ALGORITHM_HAND_RECOGNITION, 1)))))
        line2.config(text=(str("ID1 name:") + str((str((huskylens.getCachedResultByID(ALGORITHM_HAND_RECOGNITION, 1).name if huskylens.getCachedResultByID(ALGORITHM_HAND_RECOGNITION, 1) else -1)) + str("banana")))))
        line4.config(text=(str("Coords:") + str((str((huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1).xCenter if huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1) else -1)) + str((str("，") + str((huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1).yCenter if huskylens.getCachedIndexResultByID(ALGORITHM_HAND_RECOGNITION, 1, 1-1) else -1))))))))

Output Result: As shown, it can acquire the count and name of Gesture ID 1 in the frame, and the coordinate position of the first detected Gesture ID 1.