Vision

The sr.robot3 library contains support for detecting fiducial markers with the provided webcam. Markers are attached to various items in the Student Robotics arena. Each marker encodes a number in a machine-readable way, which means that robots can identify these objects. For information on which markers codes are which, see the markers page.

Using knowledge of the physical size of the different markers and the characteristics of the webcam, your robot can calculate the position of markers in 3D space relative to the camera. Therefore, if the robot can see a marker that is at a fixed location in the arena, a robot can calculate its exact position in the arena.

Under the hood, the vision system is based on AprilTag and OpenCV, using the 36H11 marker set.

Camera

The interface to the vision system is through the camera, accessible through R.camera.

see: Take a photo through the webcam, and return a list of Marker instances, each of which describes one of the markers that were found in the image.

Here’s an example that will repeatedly print out the distance to each arena marker that the robot can see:

from sr.robot3 import *
R = Robot()

while True:
    markers = R.camera.see()
    print("I can see", len(markers), "markers:")

    for m in markers:
        print(" - Marker #{0} is {1} metres away".format(m.id, m.distance / 1000))

see_ids: Take a photo through the webcam, and return a list of marker ids (not full Marker objects). This doesn’t do the same pose estimation calculations as see, and so is much faster to run.

from sr.robot3 import *
R = Robot()

while True:
    marker_ids = R.camera.see_ids()

    if 0 in marker_ids:
        print("I can see marker 0!")
    else:
        print("I cannot see marker 0!")

The simulated camera does not have capture or save methods.

save: Take a photo through the webcam, and save it to the provided location.

from sr.robot3 import *
R = Robot()

# `R.usbkey` is the path to your USB drive
R.camera.save(R.usbkey / "initial-view.png")

capture: Take a photo through the webcam, and return the image data as an OpenCV array.

import cv2
from sr.robot3 import *
R = Robot()

frame = R.camera.capture()

# Flip the image with OpenCV
flipped = cv2.flip(frame, 0)

Field of View

The Logitech C500 has a field of view of 72° and the C270 has a field of view of 60°.

Definition of Axes

The vision system describes the markers it can see using three coordinate systems. These are intended to be complementary to each other and contain the same information in different forms.

The axis definitions match those in common use, as follows:

x-axis: The horizontal axis running left-to-right in front of the camera. Rotation about this axis is equivalent to leaning towards or away from the camera.
y-axis: The vertical axis running top-to-bottom in front of the camera. Rotation about this axis is equivalent to turning on the spot, to the left or right.
z-axis: The axis leading away from the camera to infinity. Rotation about this axis is equivalent to being rolled sideways.

Note that the axes are all defined relative to the camera. Since we have no way to know how you've mounted your camera, you may need to account for that in your usage of the vision system's data.

Objects of the Vision System

The vision system is made up of a number of objects, the primary of which is the Marker:

`Marker`

A Marker object contains information about a detected marker. It has the following attributes:

id: The id of the marker.
size: The physical size of the marker, as the vision system expects it.

Pixel coordinate information is not available in the simulator.

pixel_centre: A PixelCoordinates describing the position of the centre of the marker.
pixel_corners: A list of 4 PixelCoordinates instances, each representing the position of the corners of the marker.
distance: The distance between the camera and the centre of the marker, in millimetres.
orientation: An Orientation instance describing the orientation of the marker.
spherical: A SphericalCoordinate instance describing the position of the marker relative to the camera.
cartesian: A CartesianCoordinates instance describing the position of the marker relative to the camera.

`PixelCoordinates`

Pixel coordinate information is not available in the simulator.

A named tuple of x and y coordinates for the point, in pixels relative to the top left of the image.

# Print the x and y coordinates of the pixel location
print(marker.pixel_centre.x, marker.pixel_centre.y)

`CartesianCoordinates`

A named tuple of x, y and z coordinates for the point, in millimeters relative to the camera. Increasing values are to the right, below and away from the camera respectively.

# Print the x, y and z coordinates of the marker's location
print(marker.cartesian.x, marker.cartesian.y, marker.cartesian.z)

`Orientation`

Orientation information is returned in different formats between the simulator and the physical robot kits. One (possibly both) of them may change to resolve this.

In the simulator the roll, pitch and yaw properties are strict aliases for the rot_x, rot_y and rot_z properties. Additionally the rotation_matrix and quaternion properties are not present.

There is a bug in the simulator such that the rot_x and rot_z values are not reliable – specifically they depend on the orientation of the robot itself within the simulated environment in addition to the orientation of the marker. This is believed to be a bug in Webots simulation software.

An Orientation object describes the orientation of a marker.

rot_x: Rotation of the marker about the cartesian x-axis, in radians. This is a pitch-like rotation.
Leaning a marker towards the camera increases the value of rot_x, while leaning it away from the camera decreases it. A value of either π or -π indicates that the marker is upright (there is a discontinuity in the value at π and -π, as both values represent the same position).
rot_y: Rotation of the marker about the cartesian y-axis, in radians. This is a yaw-like rotation.
Turning a marker clockwise (as viewed from above) decreases the value of rot_y, while turning it anticlockwise increases it. A value of 0 means that the marker is perpendicular to the line of sight of the camera.
rot_z: Rotation of the marker about the cartesian z-axis, in radians. This is a roll-like rotation.
Turning a marker anticlockwise (as viewed from the camera) increases the value of rot_z, while turning it clockwise decreases it. A value of 0 indicates that the marker is upright.

There are additional attributes for the principal axis rotations of the marker.

yaw: A rotation about the about the vertical axis, in radians (an axis top to bottom through the token). Turning a marker clockwise (as viewed from above) increases the value of yaw, while turning it anticlockwise decreases it. A value of 0 means that the marker is perpendicular to the line of sight of the camera.
This differs from rot_y in the direction that increases the value.
pitch: A rotation about the transverse axis, in radians (an axis right to left across the token). Tilting the marker backward increases the value of pitch, while tilting it forwards decreases it. A value of 0 indicates that the marker is facing the camera square-on.
This differs from rot_x in the zero point and direction to increase the value.
roll: A rotation about the longitudinal axis, in radians (an axis normal from the apparent front to the back of the token, normal to the marker). Rotating the marker anti-clockwise (from the perspective of the camera) increases the value of roll, while rotating it clockwise decreases it. A value of 0 indicates that the marker is upright.
This differs from rot_x in the zero point and direction to increase the value.

Finally there are attributes which express the orientation in other forms:

rotation_matrix: The rotation matrix represented by this orientation. This 3×3 matrix is represented by a list of 3 lists, each with 3 values, in an arrangement compatible with tools such as numpy.
quaternion: The quaternion represented by this orientation. On the physical kits this is implemented as a pyquaternion.Quaternion instance.

`SphericalCoordinate`

Spherical coordinate information is returned in different formats between the simulator and the physical robot kits.

The simulator does not provide the theta or phi values and the values for rot_x and rot_y may be slightly different for equivalent positions.

The spherical coordinates system has three values to specify a specific point in space.

distance: The radial distance, the distance from the origin to the point, in millimetres.
theta: This is the angle from directly in front of the camera to the vector which points to the location in the horizontal plane, in radians. A positive value indicates a counter-clockwise rotation. Zero is at the centre of the image.
phi: The polar angle from the y-axis of the camera “down” to the vector which points to the location, in radians. Zero is directly upward.

Also available are two computed angles which express the same location slightly differently:

rot_x: Approximate rotation around the X-axis, in radians. This is the angle from the camera’s horizontal plane to the vector which points to the location. Zero is at the centre of the image. Values increase towards the bottom of the image.
rot_y: Rotation around the Y-axis, in radians. This is similar to theta, however values increase towards the right of the image. Zero is at the centre of the image.

The camera is located at the origin, where the coordinates are (0, 0, 0).

Documentation

Vision