Skip to main content
Skip table of contents

Implementing a Real-Time Gaze Follower on the Console

This tutorial demonstrates how to implement a real-time gaze follower for the TRACKPixx3 on the console display. This feature is available with M16 and RB3D video modes. Both of these modes have been described, with code examples, in other VOCAL tutorials:

M16 mode is a high bit depth mode that shows up to 12 bits per colour depending on the display. If you are using M16 with a DATAPixx3, TRACKPixx3 and a third-party monitor, the output will be 8 bits per colour. RB3D mode can only be used with a PROPixx.

The Console

The console monitor is an optional secondary display connected to video out 2 on the DATAPixx3, which acts as a control unit for the TRACKPixx3.

2024-06-27_15-00-05.png

Back of the DATAPixx3 with video out 2 port identified

The console display shows a copy of the stimulus display. If a TRACKPixx3 is connected, the console can also show a live feed of the TRACKPixx3 camera view in the top left corner.

The console allows the experimenter to monitor study progress, without adding any load to the PC graphics card, which could affect the stability of the stimulus display video. However, because the console is a duplicate video signal, it is impossible to present unique information on it.

2024-06-20_10-36-02.png

Example TRACKPixx3 layout with Console Display shown

M16 and RB3D modes allow the user to draw content to both displays such that the content is only visible on the console. They do this by using colour lookup tables (CLUTs) and transparencies.

Colour lookup tables (CLUTs) and transparencies

A CLUT is a 256 x 3 table of R, G and B values. This table is specified by the user and loaded onto VPixx hardware. When a CLUT row is invoked as the colour value for a given stimulus, the stimulus is drawn in the RGB value stored in that row.

The user can create two different CLUTs for the stimulus display and console. If we draw a square on our display and assign its colour as CLUT row 2, the stimulus display will show the square in the colour in row 2 of the stimulus display CLUT, and the console will show the square in the colour stored in row 2 of the console CLUT.

We can also designate a specific RGB colour value as a transparency. If a CLUT row contains this designated value, any content we draw that invokes that row of the CLUT will be transparent.

Using this method, we can create a stimulus display full of transparencies, and a console CLUT full of colours, and invoke the CLUT whenever we want to draw something exclusively on our console (i.e., our real-time gaze data).

Real-time gaze follower in M16 mode. See demo below for how to implement this example.

Adding gaze data

We can use the console to visualize the participant’s gaze in real time. To do this, we will need to routinely poll the participant's current eye position using GetEyePosition.

MATLAB
Datapixx('RegWrRd'); %get most recent data from hardware
[xR, yR, xL, yL, ~,~,~,~,~] = Datapixx('GetEyePosition');

We can use this position to visualize gaze location on the screen.

GetEyePosition is convenient, but slow. Polling the tracker from your PC takes time. We only recommend using GetEyePosition for real-time visualization and online gaze monitoring. For post-processing and analysis of eye movements, use the eye tracking buffer data recorded and stored on our hardware.

Coordinate systems for drawing

Depending on the software you are using, you may need to convert the x, y position information provided by GetEyePosition into the coordinates your software uses for drawing. Consider the conversions below:

Program

Location of Origin (0,0)

(x, y) value of bottom right corner

Conversion

VPixx Screen Coordinates (used by TRACKPixx3)

Center

960, -540

None

Psychtoolbox

Top left corner

1920, 1080

PTBx = x+960; PTBy = 540-y

PsychoPy

Center

1, -1

Use pixel units for drawing gaze trace

These conversions are used in our demonstrations below.

Examples

Gaze overlay in M16 mode

In this example, we present a simple 9-dot grid and show the left and right gaze in cyan and magenta, exclusively on the console.

Psychtoolbox has a useful overlay window feature, which we will use in conjunction with their utilities for drawing in M16 mode.

MATLAB example
MATLAB
%Some PTB setup handling
AssertOpenGL;
KbName('UnifyKeyNames');
Screen('Preference', 'TextAntiAliasing', 0);
Screen('Preference', 'SkipSyncTests', 1);

% Configure PsychToolbox imaging pipeline to use 32-bit floating point numbers
% and M16 output with overlay window.
PsychImaging('PrepareConfiguration');
PsychImaging('AddTask', 'General', 'FloatingPoint32Bit');
PsychImaging('AddTask', 'General', 'EnableDataPixxM16OutputWithOverlay');

%Set up VPixx system & commence eye tracking
Datapixx('Open');
Datapixx('SetTPxAwake');
Datapixx('SetupTPxSchedule');
Datapixx('StartTPxSchedule');
Datapixx('RegWr');

% Open our window and overlay window
screenNumber=max(Screen('Screens'));
[win, winRect] = PsychImaging('OpenWindow', screenNumber, [0,0,0]);
overlay = PsychImaging('GetOverlayWindow', win);

% Ensure that the graphics board's gamma table does not transform our pixels
% Colour corrections can have disastrous effects on CLUTs
Screen('LoadNormalizedGammaTable', win, linspace(0, 1, 256)' * [1, 1, 1]); 

%% CREATE OUR CLUTS
% We'll arbitrarily use full green as our transparency colour
transparencyColor = [0, 1, 0];
Datapixx('SetVideoClutTransparencyColor', transparencyColor);
Datapixx('EnableVideoClutTransparencyColorMode');
Datapixx('RegWr');

% Populate our CLUTs with transparency to start
clutTestDisplay = repmat(transparencyColor, [256,1]);   
clutConsoleDisplay = repmat(transparencyColor, [256,1]);  

% ! ON WINDOWS, DrawFormattedText scales the color by 255/256, therefore
% the color is off by 1 for the upper half of the CLUT 
% On OS-X, DrawFormattedText seems to apply a grossly non-linear mapping
% between the argument intensity and the actual draw intensity.
% Other draw commands like DrawRect do not seem to show this bug.

%Since we only need two colours, we will assign a block of rows to each and pick
%a row in the middle to index. This should avoid any OS-related issues.

%Keep in mind that right and left eye refer to the right and left eyes as they 
%appear in the tracker window. They are not necessarily the left and right eyes 
%from the participant's perspective.
clutConsoleDisplay(1:10,:) = repmat([0, 1, 1], [10,1]);   % Left eye Cyan
clutConsoleDisplay(11:20,:) = repmat([1, 0, 1], [10,1]);   % Right eye Magenta

Datapixx('SetVideoClut', [clutTestDisplay;clutConsoleDisplay]);
Datapixx('RegWr');

%STIMULUS PARAMETERS
% Set some parameters for our 3x3 grid of dots
[centerX, centerY] = RectCenter(winRect);
numDots = 9;
gridSize = 3; % 3x3 grid
dotSpacing = 100; % Spacing between dots in pixels
dotSize = 20; % Diameter of each dot

% Calculate the starting position (top-left corner of the grid)
startX = centerX - (dotSpacing * (gridSize - 1)) / 2;
startY = centerY - (dotSpacing * (gridSize - 1)) / 2;

% Initialize the positions array
positions = zeros(2, numDots);

% Fill the positions array with the coordinates of the 9 dots
index = 1;
for row = 0:(gridSize - 1)
    for col = 0:(gridSize - 1)
        positions(1, index) = startX + col * dotSpacing;
        positions(2, index) = startY + row * dotSpacing;
        index = index + 1;
    end
end

%Specify the radius of our gaze follower markers
gazeRadius = 10;

%Start a while loop that exists on pressing "Escape"
while 1
    % Draw the dots
    Screen('DrawDots', win, positions, dotSize, [1,1,1], [], 2);

    %The overlay plane is not automatically cleared
    %to background (or transparent) color after a flip. Instead its content
    %persists across flips. You need to clear it out manually via a
    %Screen(%FillRect’) command.
    Screen('FillRect', overlay, 255, winRect); %Transparency color
  
    %Get eye position
    Datapixx('RegWrRd');
    [xR, yR, xL, yL, ~,~,~,~,~] = Datapixx('GetEyePosition');

    %Convert camera coordinates (top left is -960, 540) to 
    %screen position in PTB coordinates (top left is 0,0)
    eyePos = [xL+960, 540-yL, xR+960, 540-yR];

    %Draw left and right eyes on overlay
    Screen('FillOval', overlay, 5, [eyePos(1) - gazeRadius, eyePos(2) - gazeRadius, eyePos(1) + gazeRadius, eyePos(2) + gazeRadius]);
    Screen('FillOval', overlay, 15, [eyePos(3) - gazeRadius, eyePos(4) - gazeRadius, eyePos(3) + gazeRadius, eyePos(4) + gazeRadius]);

    %Flip
    Screen('Flip', win);

    %Check for escape
    [~, ~, keyCode] = KbCheck; 
    if keyCode(KbName('Escape'))
        break;
    end
end

Datapixx('StopTPxSchedule');
Datapixx('DisableVideoClutTransparencyColorMode');
Datapixx('RegWr');
Screen('CloseAll');
Python example

Coming soon

Gaze overlay in RB3D mode

It is possible to track the eyes while the participant wears passive 3D glasses. Consider using search windows to omit glare from the image processing and improve tracking quality. See this guide for more details.

In this example, we present a grid similar to our first example, but in 3D. The left and right eye images are drawn in blue and red, respectively, and are offset slightly from one another.

Psychtoolbox does not have an overlay window feature for this mode, so to draw our gaze follower we will need to index our CLUT using the green colour channel, which is reserved for this purpose. For example, to index row three of the CLUT, we would pass [0,3,0] as the colour value of the stimulus. This stimulus will only appear on the console.

Note that this mode does not require transparencies, as RB3D mode will ignore all CLUT content for the stimulus display. However, we still need to populate a stimulus display CLUT, append the console CLUT to it, and load both onto our hardware to ensure the CLUTs are formatted correctly.

MATLAB example
MATLAB
%% SET UP
%Some parameters
AssertOpenGL;
KbName('UnifyKeyNames');
Screen('Preference', 'TextAntiAliasing', 0);
Screen('Preference', 'SkipSyncTests', 1);

%Set up VPixx system
Datapixx('Open');
Datapixx('SetTPxAwake');
Datapixx('SetupTPxSchedule');
Datapixx('StartTPxSchedule');
Datapixx('RegWr');

%Set our PROPixx up for RB3D with overlay
% Enable PROPixx RB3D Sequencer
Datapixx('Open');
Datapixx('SetPropixxDlpSequenceProgram', 1);
Datapixx('RegWr');

%Enable video mode that supports RB3D with overlay
Datapixx('SetVideoMode', 9);
Datapixx('RegWr');

% You can modify the per eye crosstalk here.
Datapixx('SetPropixx3DCrosstalkLR', 0);
Datapixx('SetPropixx3DCrosstalkRL', 0);
Datapixx('RegWrRd')

% Open our window
screenNumber=max(Screen('Screens'));
[win, winRect] = PsychImaging('OpenWindow', screenNumber, [0,0,0]);

% Ensure that the graphics board's gamma table does not transform our pixels
Screen('LoadNormalizedGammaTable', win, linspace(0, 1, 256)' * [1, 1, 1]); 

%% CREATE OUR CLUTS
clutTestDisplay = repmat([0,0,0], [256,1]);  
clutConsoleDisplay = repmat([0,0,0], [256,1]);  

% ! ON WINDOWS, DrawFormattedText scales the color by 255/256, therefore
% the color is off by 1 for the upper half of the CLUT 
% On OS-X, DrawFormattedText seems to apply a grossly non-linear mapping
% between the argument intensity and the actual draw intensity.
% Other draw commands like DrawRect do not seem to show this bug.

%Since we only need two colours, we will assign a block of rows to each and pick
%a row in the middle to index. This should avoid any OS-related issues.

%Keep in mind that right and left eye refer to the right and left eyes as they 
%appear in the tracker window. They are not necessarily the left and right eyes 
%from the participant's perspective.
clutConsoleDisplay(1:10,:) = repmat([0, 1, 1], [10,1]);   % Left eye Cyan
clutConsoleDisplay(11:20,:) = repmat([1, 0, 1], [10,1]);   % Right eye Magenta

Datapixx('SetVideoClut', [clutTestDisplay;clutConsoleDisplay]);
Datapixx('RegWr');

%% STIMULUS PARAMETERS
% Calculate the positions of the 9 dots in a 3x3 grid
[centerX, centerY] = RectCenter(winRect);
numDots = 9;
gridSize = 3; % 3x3 grid
dotSpacing = 100; % Spacing between dots in pixels
dotSize = 20; % Diameter of each dot

% Calculate the starting position (top-left corner of the grid)
startX = centerX - (dotSpacing * (gridSize - 1)) / 2;
startY = centerY - (dotSpacing * (gridSize - 1)) / 2;

% Initialize the positions array
positions = zeros(2, numDots);

% Fill the positions array with the coordinates of the 9 dots
index = 1;
for row = 0:(gridSize - 1)
    for col = 0:(gridSize - 1)
        positions(1, index) = startX + col * dotSpacing;
        positions(2, index) = startY + row * dotSpacing;
        index = index + 1;
    end
end

%Let's add an offset for our other eye image
positionsRight = positions-5;

%And our gaze follower radius
gazeRadius = 10;

%% PRESENTATION LOOP
%Start a while loop that draws our grid and the gaze position on the
%console. Hitting 'escape' ends the loop.
while 1
    % Draw the dots for our left eye in blue
    Screen('DrawDots', win, positions, dotSize, [0,0,255], [], 2);

    % Draw the dots for our right eye in red
    Screen('DrawDots', win, positionsRight, dotSize, [255,0,0], [], 2);
  
    %Get eye position
    Datapixx('RegWrRd');
    [xR, yR, xL, yL, ~,~,~,~,~] = Datapixx('GetEyePosition');

    %Convert to camera coordinates (top left is -960, 540) to 
    % screen position in PTB coordinates (top left is 0,0)
    eyePos = [xL+960, 540-yL, xR+960, 540-yR];
    disp(eyePos)

    %Draw left and right eyes on console. Console CLUT is indexed via the green
    %colour channel.
    Screen('FillOval', win, [0,5,0], [eyePos(1) - gazeRadius, eyePos(2) - gazeRadius, eyePos(1) + gazeRadius, eyePos(2) + gazeRadius]);
    Screen('FillOval', win, [0,15,0], [eyePos(3) - gazeRadius, eyePos(4) - gazeRadius, eyePos(3) + gazeRadius, eyePos(4) + gazeRadius]);

    %Flip
    Screen('Flip', win);

    %Check for escape
    [~, ~, keyCode] = KbCheck; 
    if keyCode(KbName('Escape'))
        break;
    end
end

%% SHUT DOWN
% Set the PROPixx back to normal sequencer
Datapixx('SetPropixxDlpSequenceProgram', 0);
Datapixx('SetVideoMode', 0);
Datapixx('StopTPxSchedule');
Datapixx('RegWrRd');

% Done. Close the onscreen window.
Screen('CloseAll')
Datapixx('Close');
Python example

Coming soon

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.