A computer vision project for Eye (pupil) tracking
Background/Goal
Current eye tracking systems available in the market are either uncomfortable to wear, expensive, slow, or often not customizable. The goal is to research, architect, develop/build a low cost wearable (head mounted) goggles like system that can track a user's pupil movement and detect the user's gaze in real time while being least pervasive. This project solves that problem by using low cost and high performance MIPI cameras as well as various other parts and systems. The use cases for this application are tremendous: Assistive technologies, law enforcement eye tracking, robotics, driver monitoring in self driving vehicles, behavioral research, retail customer gaze tracking, video games and so much more.
Parts List
| Quantity | Description | Seller | Price |
|---|---|---|---|
| 1 | Jetson TK-1 | NVIDIA | $192.00 |
| 1 | Raspberry Pi 3 | MCM Electronics | $35.00 |
| 1 | MIPI Camera | Amazon | $32.99 |
| 1 | Pi Camera HDMI Cable Extension | Tindie | $14.95 |
| 1 | Flat HDMI Cable | Frys | $12.99 |
| 1 | Infrared Pass Filter | B & H Photo Video | $16.00 |
| 1 | NETGEAR MaxRange Wireless Router | Amazon | $21.99 |
| 1 | Mediabridge Cat5e Ethernet | Amazon | $5.89 |
| 1 | Sugru Moldable Glue | Amazon | $12.50 |
| 1 | Bolle Mojo Snow Goggles | Amazon | $16.46 |
| 2 | Infrared LED | Mouser | $1.02 each |
| 1 | Ribbon Cable | DigiKey | $14.95 |
| 1 | 220 Ohm Resistor | DigiKey | $0.20 |
| 2 | 50 Pin 2mm Connector | DigiKey | $4.75 each |
| 1 | 2mm x 2mm Prototype Board | Allied Electronics | $23.63 |
| 1 | 22 AWG Wire | RadioShack | $9.99 |
| 1 | 2mm Connector (SQW-125-01-F-D) | Samtec | Free Sample |
| 1 | 3D Printed Camera Mount | UTA FabLab | $0.50 |
| Quantity | Description | Part Number | Seller | Price |
|---|---|---|---|---|
| 3 | 10uF Capacitor | 81-ZRB18AD71A106KE1L | Mouser | $0.41 each |
| 4 | 0.1uF Capacitor | 80-C0603C104K8R | Mouser | $0.10 each |
| 1 | 1uF Capacitor | 80-C0603C105K8P | Mouser | $0.10 |
| 1 | 22 Ohm Resistor | 71-CRCW0603J-22-E3 | Mouser | $0.10 |
| 2 | 2.2k Ohm Resistor | 71-CRCW0603-2.2K-E3 | Mouser | $0.10 each |
| 1 | 10k Ohm Resistor | 71-CRCW0603-10K-E3 | Mouser | $0.10 |
| 1 | 82k Ohm Resistor | 71-CRCW0603-82K-E3 | Mouser | $0.10 |
| 1 | 100k Ohm Resistor | 71-CRCW0603-100K-E3 | Mouser | $0.10 |
| 1 | Schottky Diode 30V 200mA | BAT54XV2T1GOSCT-ND | Digi-Key | $0.31 |
| 1 | MIPI Camera Connector | 255-2528-1-ND | Digi-Key | $2.29 |
| 1 | IC LDO 3.3V 0.5A | 296-13424-1-ND | Digi-Key | $0.58 |
| 1 | 600mA CMOS LDO | 621-AP2112K-3.3TRG1 | Mouser | $0.46 |
| 1 | pcDuino MIPI Camera | SEN-13249 | SparkFun Electronics | $15.95 |
| 1 | Jetson TK-1 75 Pin GPIO Connector | SQW-125-01-F-T | Samtec | Free Samples |
Architectural Details
Here is a very high level system diagram
Nvidia Jetson TK-1 Embedded Development Board
- Development board used to implement the eye tracker algorithm.
- Uses OpenCV4Tegra, CUDA 6.5 Toolkit, Cmake, and a g++ compiler.
- Connects via Ethernet with the Raspberry Pi 3.
- Infrared LEDs are powered using the Jetson TK-1 GPIO.
Raspberry pi 3 Embedded Development Board
- Single-board computer used to stream video via Ethernet to the Jetson TK-1.
- An infrared MIPI Camera is connected to the Raspberry Pi 3 with an HDMI Extension Board.
- A static IP has been set on the device for easier streaming connectivity.
- The Raspberry Pi 3 is powered by the Jetson TK-1.
High Level Software Architecture
- Read live video stream from the camera with OpenCV
- Process each video frame using computer vision algorithms through various stages: greyscale, apply masking kernels, remove eye lashes, RANSAC algorithm, edge detection, prune unwanted edges using a threshold, connect edges, use a 3D ellipse fitter
- Calculate pupil(ellipse's) angle, pupil center, pupil size and determine user's gaze vector
- Display the resulting video frame with eye tracking
The following images show various stages of the algorithmic process:
