Coming in strong on our list of cool projects made with our boards is another interesting project that uses the Zybo board:
“An object tracking vision system using a moving camera implemented in a Zynq heterogeneous device”, by Marcin Kowalczyk, from AGH University of Science and Technology in Krakow, Poland.
Autonomous object tracking with use of a moving camera is used for examples in issues connected with security, surveillance or in military applications. This particular project involved creating a video system demonstration for tracking objects, assuming that the camera is mounted on a moving stand.
The system consists of:
– Camera – Xiaomi Yi Action YDXJ01XY sports camera
– Computing platform – the Digilent Zybo board
– Sensors – PmodNAV, PmodGYRO, PmodACL
– Bluetooth module – PmodBT2
– Servo controller
– Power supply
– Pointer – laser pointer
The output of tracking module is the basis for the positioning of moving stand. The aim of the positioning is to keep a tracked object in the center of frame and mark it with a pointer. Below are the modules of the project can be visualized by the connection between them.
- Object tracking – This contains elements responsible for receiving and decoding data from camera. Except
that it contains tracking algorithm. Outputs of this module are properly delayed input signals,
tracked object coordinates and signal indicating end of frame processing.
- Object marking – The marks received coordinates in frame. It may be marked with two perpendicular
lines or rectangle.
- Visualization – This is responsible for coding data from RGB form and synchronization signals to VGA
form and sending it to output.
- Center coordinates to angles – converts the input coordinates to the error of pan and tilt angles.
- Kalman filter – This receives data from sensors (accelerometer, magnetometer and gyroscope) and calculates
position using sensory fusion.
- Regulator – Based on any angle errors received on input it calculates a new set of positions for servomechanisms,
converts it to pulse widths and passes them to output.
- Communication – This is responsible for receiving commands and data from PC and sending appropriate signals
to the servo controller. If autonomous mode is enabled, received pulse width from regulator is passed
It was decided by the team to use Xiaomi Yi Action YDXJ01XY sports cameraCommunication: it was implemented between all elements of the system. It can be divided into PC-Zynq part and Zynq-servo controller ’Maestro’.
- PC – ZYNQ – via bluetooth
- ZYNQ – Maestro – via UART interface which was imposed, because it is the only protocol supported by Maestro.
Suitable pins (MIO 14 and 15) of ZYBO’s MIO PMOD connector were connected to Maestro pins
responsible for serial communication. Then pins were connected to another processor’s peripheral for
serial communication – UART0
In the designed system, camera is a sensor of regulation error. Regulator was created for system without
sensors of moving stand position, thus control algorithm is currently based only on data from camera. For constructing a model, simulations were performed in MATLAB and Simulink, below is the block diagram of the servomechanism:
and the block diagram of entire system:
- Tracking by detection algorithm, output can be seen:
2. Mean Shift algorithm:
Calibration of the sensors was needed in order to obtain proper results, and their integration and fusion as well.
– Parallel realization of algorithm (hardware implementation) requires a lot more work than software
implementation (in processor).
– Mean-shift algorithm has good effects, when tracked object’s color is much different from the
background’s (if H component of HSV color space is used).
– Tracking by detection algorithm usually gives correct coordinates of object, but it does not use
information from previous frame. In result, there are possible situations, when in two consecutive
frames object is detected in other sides of image (e.g. in result of noises), what may result in
mechanical damage, related to sharp movement of servomechanisms.
– Selection of proper regulator and it’s parameters is much easier based on mathematical model of
– Mathematical model does not have to perfectly reproduce operation of the real system. Effects
such as very short delays may be neglected.
– Breakdown of project into independent modules greatly simplifies work.
– Testing each module separately, after realization, makes it easier to connect them into a complete