Computer Vision Powered Robotic Hand

About This Project

I designed this project to be my introduction to computer vision and robotics. I had never worked with either of them directly, but I felt I had relevant experience in adjacent fields which would translate over well. And although it was a difficult project in terms of my time constraints, I learned a ton about OpenCV, MediaPipe, robotic joints, and DFA. It was a 3 week sprint to build this from ideation to completion as I wanted to produce a working prototype ideally within the first week of classes beginning.

Summary

Objective

Design a robotic hand with individually actuated digits and 8 total DoF
Utilize computer vision to track hand and produce skeleton
Control all motors and run image collection/processing on one machine

Engineering Decisions

Designed an actuated quaternion joint to produce +/- 90° ROM in all directions
Developed biomimetic joint mechanism with high tensile strength tendons (fishing line)
Utilized OpenCV for image processing and MediaPipe for hand landmark recognition
Selected PCA9685 PWM driver and Raspberry Pi over I2C bus to allow for accurate individual servo control and simultaneous image processing
Performed hand calculations to ensure torque requirements did not exceed servo stall torque

Results

All digits and wrist are actuated with individual servos (8)
Robot mimics hand with <2 second latency
All components FDM 3D printed and fastened with heat-set inserts and M3 bolts

Wrist Design

To satisfy my requirements of mimicking a human wrist, I needed a joint that facilitated 144° in flexion/extension (up/down), 52° in radial/ulnar deviation (waving), and 200° in supination/pronation (rotation). As this was a minimum and I was working with electro-mechanical components, my goal was to exceed this ROM.

While researching methods of actuating a wrist-mimicking joint, I came across several designs. As the wrist is akin to an ovular ball and socket joint, it's difficult to perfectly mimic its ROM in the same form factor. One design, however, that piqued my interest was the quaternion joint from the LIMS2-AMBIDEX humanoid robot developed at the Korea University of Technology. It allows for a signifcant ROM while also allowing a hollow center for components to travel through. Through this article, I was able to design my own version of it with a slightly different method of actuation. With this joint, I was able to produce +/- 90° ROM in all directions.

Final CAD, Quaternion Joint Proof of Concept, Quaternion Joint Testing, Minimum Wrist ROM

Hand, Forearm, & Base

Full CAD, Hand & Forearm, Forearm Section View, Forearm Top View, Base

The palm housed the servos controlling the digits as I wanted the assembly to be as compact as possible. Although it made the palm thicker, it allowed me to make the forearm much smaller. In addition to this, the wrist mechanism would create slack in the tendons every time it was actuated if the servos were housed in the forearm.

Each finger is based off a parametric finger model that I developed and then customized by lengthening or shortening them as compared to my own fingers. The tendons in the fingers are made of fishing line due to its high tensile strength. They are tied to each end of a servo which allows the finger to actuate when the servo turns. A pocket was designed for each servo to fit in in addition to a plate that bolted onto the back to hold them in place.

Assembly Progression

The forearm housed two of the three wrist servos in addition to the PCA9685 servo driver and voltage regulator. The fishing line tied to the wrist joint threaded into an opening on the forearm and was tied to the servos.

The base was originally designed to house the PiCam, but due to the ribbon cable being too short, it ended up being relocated. However, the servo which facilitated supination-pronation of the hand was located there. This servo was bolted into the base and rotated a plate. Due to the large circumference of the servo plate in comparison to the servo creating an unstable mechanism, three wheels were installed on the plate to stabilize the hand.

Software

To 'see' the hand and process it, I used OpenCV which is a computer vision library. And although there are good hand detection models using only OpenCV, I wanted a great model. To do this, I paired OpenCV with MediaPipe, which is a machine learning framework with very accurate landmark detection for everything from face and hand detection to pose estimation. With this data, I was able to detect my hand and use the various landmarks to send PWM signals to the servos.

To actuate the digits, I used a binary open/close system. If, for example, node 8's Y coordinates were below node 6's, the servo would be sent a signal to rotate. This was repeated for every digit except the thumb which used X coordinates instead of Y.

To facilitate radial/ulnar deviation, I took the inverse tangent of the 0 and 9 node to find the angle between them. This angle then drove the servo's rotation. For flexion/extension, I used the Y coordinate of node 9 and related it to the frame height and supination/pronation was driven by the X coordinate of node 9.

Hand Landmark Model, Video Output

Hardware

I used a Raspberry Pi 3 to run everything as it's more powerful than the Arduino and easily supports Python programs. To actuate everything, I used 8 separate MG90S 9G servos due to their minimal weight and sufficient stall torque. To run them, however, I needed the PCA9685 PWM servo driver board as it makes running multiple servos much easier. The PCA9685 communicates with the Pi over its I2C bus which provides quick servo control.

To power it all, I had a 12V, 7A power supply from a previous project. The PCA9685 only takes in 6V max, however, so I had to purchase a buck voltage regulator. As the total current draw of the servos was only 3.2A (400mA stall current/servo), this was more than sufficient.

Results

Issues & Next Steps

Significant latency in servo commands
- I've looked into multithreading to allow for frame input and image processing to be on separate threads. In addition to this, at the moment, the program only seems to be using ~35% of each of the four cores on the Pi.
- Switching to EMG sensors would bypass any image processing and significantly lighten the load on the Pi and speed it up
MG90S servos have very low stall torque, limiting hand size and width
- Utilizing hydraulics instead would make it much stronger but also comes with changes in material selection to keep up with the added strength (just to name one thing)

Page updated

Google Sites

Report abuse