This tutorial demonstrates how to implement a real-time gaze follower for the TRACKPixx3 on the console display. This feature is available with M16 and RB3D video modes. Both of these modes have been described, with code examples, in other VOCAL tutorials:
M16 mode is a high bit depth mode that shows up to 12 bits per colour depending on the display. If you are using M16 with a DATAPixx3, TRACKPixx3 and a third-party monitor, the output will be 8 bits per colour. RB3D mode can only be used with a PROPixx.
The Console
The console monitor is an optional secondary display connected to video out 2 on the DATAPixx3, which acts as a control unit for the TRACKPixx3.
The console display shows a copy of the stimulus display. If a TRACKPixx3 is connected, the console can also show a live feed of the TRACKPixx3 camera view in the top left corner.
The console allows the experimenter to monitor study progress, without adding any load to the PC graphics card, which could affect the stability of the stimulus display video. However, because the console is a duplicate video signal, it is impossible to present unique information on it.
M16 and RB3D modes allow the user to draw content to both displays such that the content is only visible on the console. They do this by using colour lookup tables (CLUTs) and transparencies.
Colour lookup tables (CLUTs) and transparencies
A CLUT is a 256 x 3 table of R, G and B values. This table is specified by the user and loaded onto VPixx hardware. When a CLUT row is invoked as the colour value for a given stimulus, the stimulus is drawn in the RGB value stored in that row.
The user can create two different CLUTs for the stimulus display and console. If we draw a square on our display and assign its colour as CLUT row 2, the stimulus display will show the square in the colour in row 2 of the stimulus display CLUT, and the console will show the square in the colour stored in row 2 of the console CLUT.
We can also designate a specific RGB colour value as a transparency. If a CLUT row contains this designated value, any content we draw that invokes that row of the CLUT will be transparent.
Using this method, we can create a stimulus display full of transparencies, and a console CLUT full of colours, and invoke the CLUT whenever we want to draw something exclusively on our console (i.e., our real-time gaze data).
Adding gaze data
We can use the console to visualize the participant’s gaze in real time. To do this, we will need to routinely poll the participant's current eye position using GetEyePosition.
Datapixx('RegWrRd'); %get most recent data from hardware
[xR, yR, xL, yL, ~,~,~,~,~] = Datapixx('GetEyePosition');
We can use this position to visualize gaze location on the screen.
GetEyePosition is convenient, but slow. Polling the tracker from your PC takes time. We only recommend using GetEyePosition for real-time visualization and online gaze monitoring. For post-processing and analysis of eye movements, use the eye tracking buffer data recorded and stored on our hardware.
Coordinate systems for drawing
Depending on the software you are using, you may need to convert the x, y position information provided by GetEyePosition into the coordinates your software uses for drawing. Consider the conversions below:
|
Program |
Location of Origin (0,0) |
(x, y) value of bottom right corner |
Conversion |
|---|---|---|---|
|
VPixx Screen Coordinates (used by TRACKPixx3) |
Center |
960, -540 |
None |
|
Psychtoolbox |
Top left corner |
1920, 1080 |
PTBx = x+960; PTBy = 540-y |
|
PsychoPy |
Center |
1, -1 |
Use pixel units for drawing gaze trace |
These conversions are used in our demonstrations below.
Examples
Gaze overlay in M16 mode
In this example, we present a simple 9-dot grid and show the left and right gaze in cyan and magenta, exclusively on the console.
Psychtoolbox has a useful overlay window feature, which we will use in conjunction with their utilities for drawing in M16 mode.
Gaze overlay and scrolling flag status indicators in M16 mode
This example extends the previous example by adding a pseudo-scope trace with the event flags for the left and right eyes. These flags indicate if the participant is currently making a saccade or a fixation, and are stored in columns 12-15 of the TRACKPixx3 buffer. The parameters for these flags can be modified by the user with the following commands:
Datapixx('SetFixationThresholds', maxSpeed, numOfConsecutiveSamples );
Datapixx('SetSaccadeThresholds', minSpeed, numOfConsecutiveSamples);
Datapixx('RegWr');
These are simple rolling window averages. For more complex analyses, you must manually apply your algorithm to the raw data.
Here is an example of the scrolling statuses:
When the trace is high, it indicates the fixation flag is raised for that eye; a low trace indicates a saccade flag is raised. An absence of a trace indicates a blink or tracking loss. An intermediate trace value suggests the eye is tracked, but neither fixation nor saccade criteria have been met.
This demo code could easily be modified to show other trial metadata in the scrolling format, such as:
-
x or y coordinates of either eye
-
when fixation is within a certain region of the display
-
eye movement velocity
-
pupil diameter
Below is a short video showing a demonstration of the scrolling trace:
The demo code format is largely the same as the first example. The helper function includeFlagTrace() handles the acquisition and drawing of flag data. It is invoked on every iteration of the drawing loop. A separate helper function allows for easy changes to the displayed data without modifying the main experiment script.
Gaze overlay in RB3D mode
It is possible to track the eyes while the participant wears passive 3D glasses. Consider using search windows to omit glare from the image processing and improve tracking quality. See this guide for more details.
In this example, we present a grid similar to our first example, but in 3D. The left and right eye images are drawn in blue and red, respectively, and are offset slightly from one another.
Psychtoolbox does not have an overlay window feature for this mode, so to draw our gaze follower we will need to index our CLUT using the green colour channel, which is reserved for this purpose. For example, to index row three of the CLUT, we would pass [0,3,0] as the colour value of the stimulus. This stimulus will only appear on the console.
Note that this mode does not require transparencies, as RB3D mode will ignore all CLUT content for the stimulus display. However, we still need to populate a stimulus display CLUT, append the console CLUT to it, and load both onto our hardware to ensure the CLUTs are formatted correctly.
The lack of a dedicated overlay window means that any image or texture with a green component value > 1 will invoke the colour lookup table. This can make images look strange on the console display. To avoid this, do not use green values in your generated stimuli and consider explicitly setting the green channel of any imported stimuli to 0. For example:
image = imread(imagePath); % Load the image
image(:,:,2) = 0; % Set green channel to 0
imageTexture = Screen('MakeTexture', window, image); % Make the image texture