← All projects

Kilobot Pheromone Project

A camera-and-projector system that let Kilobot robots communicate indirectly by leaving and following fading virtual pheromone trails.

Python OpenCV Tkinter AVR C Kilolib Kilobots Multithreading HSV color detection Computer vision Swarm robotics

Overview

This project explored a follow-the-leader swarm behavior using virtual pheromones made from light projected onto the arena. One Kilobot acted as the leader and left a fading light trail behind it, which the other robots attempted to detect and follow. An overhead camera tracked the leader’s movement, while a projector translated that movement into the virtual pheromone trail on the arena. The follower robots sensed the projected light with their built-in light sensors and used it to guide their movement, creating a feedback loop between the physical swarm and a digital environment. The system brought together computer vision, embedded programming, multi-threading and swarm robotics in a setup that worked somewhat like a lightweight digital twin of the arena. It was developed together with Jonas Scharin as our bachelor's project at Örebro University.

Architecture & Implementation

The system was built around a continuous feedback loop between the physical arena and a digital environment. An overhead camera captured the Kilobots as they moved, and OpenCV was used to locate the leader by detecting its red LED. Each frame was converted to HSV color space, filtered using configurable color thresholds, and processed with masking, dilation and contour detection. Small contours were discarded, while valid detections were converted into positional data used to build the virtual pheromone trail.

Physical setup of the Kilobot environment: (1) arena, (2) overhead controller, (3) webcam, (4) projector, and (5) adjustable steel frame.
Physical setup of the Kilobot environment: (1) arena, (2) overhead controller, (3) webcam, (4) projector, and (5) adjustable steel frame.

Camera processing and pheromone rendering ran on separate threads so that image capture would not be blocked while the environment was being updated and rendered. The threads shared a history of detected leader positions protected by a mutex. Each position was stored together with its creation time, allowing points to be removed once they exceeded the configured pheromone lifetime. The remaining positions were drawn as circles whose opacity decreased with age, producing the fading trail before the image was resized and displayed through the projector. A small control interface allowed parameters such as pheromone size, lifetime and HSV detection ranges to be adjusted while the system was running.

flowchart LR
    camera([Overhead camera]) --> environment[Virtual environment<br/>Track leader + create trail]
    environment --> projector([Projector])

    subgraph Arena[Physical arena]
        kilobots([Kilobots])
    end

    projector -->|Projected pheromones| Arena
    Arena -->|Robot movement| camera

The Kilobots ran their own firmware in AVR C using Kilolib and reacted to what they could sense in the physical environment. Their behavior was organized as a state machine containing states for leading, searching, following a trail, returning after losing it, and an experimental state for reacting to changes in pheromone intensity. The leader followed a randomized movement pattern with a strong preference for moving forward, while the followers searched for projected light and switched to a following state once they found a trail. If a follower lost the trail, it used a widening zig-zag recovery movement to try to find it again.

The Kilobots' possible states depending on the detected level of present pheromones.
The Kilobots' possible states depending on the detected level of present pheromones.

The Kilobots' ambient light sensors produced noisy readings, so the firmware filtered the input instead of reacting to individual measurements. A sample of readings was collected, sorted, and reduced to its median value, making it more reliable to determine whether a robot was inside or outside a projected pheromone trail.

Physical behavior also required calibration. Kilobots move using two vibration motors, and the resulting movement varied noticeably between individual robots. Their forward and turning motion was therefore calibrated to make the swarm more predictable. The camera and projector were mounted above the arena on an adjustable three-legged steel frame, allowing their position, height and projection area to be aligned with the physical environment.

A simple demonstration of the digital twin system with a leader and follower Kilobot. It shows how the leader's red LED is being detected by the camera and projects a green pheromone onto that area. It also shows the projected view with all the pheromone settings.

There were several additional implementation challenges behind the system, including compensating for limitations in the Kilobots' light sensors, calibrating their vibration-based movement and refining the behavior of the individual robot states. These details, together with the design and evaluation of the system, are documented in the full thesis report.

Project source code Thesis report