This challenge is much more open-ended than the augmented reality challenge:
Given a somehow-designated object in a scene, track that object as it moves about the scene.
The object could be designated a number of different ways:
1. The largest object moving in the foreground.
2. The object is a different color than the rest of the scene.
3. Designated with some kind of GUI (e.g. click on the object to track).
4. Your idea here.
Similarly, “tracking” can mean a number of different things:
1. Overlay some kind of marker over the designated object in the scene, and move that marker as the object moves.
2. Move a camera to keep the object centered in the field of view.
3. Move a robot so that it follows the designated object without letting it get too close or too far away.
I suggest you start with either tracking the largest moving object in the foreground, or a uniquely colored object using a marker and a fixed camera.