A computer vision project for Eye (pupil) tracking


Current eye tracking systems available in the market are either uncomfortable to wear, expensive, slow, or often not customizable. The goal is to research, architect, develop/build a low cost wearable (head mounted) goggles like system that can track a user's pupil movement and detect the user's gaze in real time while being least pervasive. This project solves that problem by using low cost and high performance MIPI cameras as well as various other parts and systems. The use cases for this application are tremendous: Assistive technologies, law enforcement eye tracking, robotics, driver monitoring in self driving vehicles, behavioral research, retail customer gaze tracking, video games and so much more.

Parts List

Eye Tracker System
Quantity Description Seller Price
1 Jetson TK-1 NVIDIA $192.00
1 Raspberry Pi 3 MCM Electronics $35.00
1 MIPI Camera Amazon $32.99
1 Pi Camera HDMI Cable Extension Tindie $14.95
1 Flat HDMI Cable Frys $12.99
1 Infrared Pass Filter B & H Photo Video $16.00
1 NETGEAR MaxRange Wireless Router Amazon $21.99
1 Mediabridge Cat5e Ethernet Amazon $5.89
1 Sugru Moldable Glue Amazon $12.50
1 Bolle Mojo Snow Goggles Amazon $16.46
2 Infrared LED Mouser $1.02 each
1 Ribbon Cable DigiKey $14.95
1 220 Ohm Resistor DigiKey $0.20
2 50 Pin 2mm Connector DigiKey $4.75 each
1 2mm x 2mm Prototype Board Allied Electronics $23.63
1 22 AWG Wire RadioShack $9.99
1 2mm Connector (SQW-125-01-F-D) Samtec Free Sample
1 3D Printed Camera Mount UTA FabLab $0.50

MIPI Camera PCB Parts List
Quantity Description Part Number Seller Price
3 10uF Capacitor 81-ZRB18AD71A106KE1L Mouser $0.41 each
4 0.1uF Capacitor 80-C0603C104K8R Mouser $0.10 each
1 1uF Capacitor 80-C0603C105K8P Mouser $0.10
1 22 Ohm Resistor 71-CRCW0603J-22-E3 Mouser $0.10
2 2.2k Ohm Resistor 71-CRCW0603-2.2K-E3 Mouser $0.10 each
1 10k Ohm Resistor 71-CRCW0603-10K-E3 Mouser $0.10
1 82k Ohm Resistor 71-CRCW0603-82K-E3 Mouser $0.10
1 100k Ohm Resistor 71-CRCW0603-100K-E3 Mouser $0.10
1 Schottky Diode 30V 200mA BAT54XV2T1GOSCT-ND Digi-Key $0.31
1 MIPI Camera Connector 255-2528-1-ND Digi-Key $2.29
1 IC LDO 3.3V 0.5A 296-13424-1-ND Digi-Key $0.58
1 600mA CMOS LDO 621-AP2112K-3.3TRG1 Mouser $0.46
1 pcDuino MIPI Camera SEN-13249 SparkFun Electronics $15.95
1 Jetson TK-1 75 Pin GPIO Connector SQW-125-01-F-T Samtec Free Samples

Architectural Details

Here is a very high level system diagram

Nvidia Jetson TK-1 Embedded Development Board

  • Development board used to implement the eye tracker algorithm.
  • Uses OpenCV4Tegra, CUDA 6.5 Toolkit, Cmake, and a g++ compiler.
  • Connects via Ethernet with the Raspberry Pi 3.
  • Infrared LEDs are powered using the Jetson TK-1 GPIO.

Raspberry pi 3 Embedded Development Board

  • Single-board computer used to stream video via Ethernet to the Jetson TK-1.
  • An infrared MIPI Camera is connected to the Raspberry Pi 3 with an HDMI Extension Board.
  • A static IP has been set on the device for easier streaming connectivity.
  • The Raspberry Pi 3 is powered by the Jetson TK-1.

High Level Software Architecture

  • Read live video stream from the camera with OpenCV
  • Process each video frame using computer vision algorithms through various stages: greyscale, apply masking kernels, remove eye lashes, RANSAC algorithm, edge detection, prune unwanted edges using a threshold, connect edges, use a 3D ellipse fitter
  • Calculate pupil(ellipse's) angle, pupil center, pupil size and determine user's gaze vector
  • Display the resulting video frame with eye tracking

The following images show various stages of the algorithmic process: