- A joint scanpath-trajectory diffusion framework that couples both modalities via channel-level concatenation, enabling end-to-end training that exploits their complementarity without additional architectural complexity.
- A CRPS-based metrics family that generalizes any existing sequence similarity measure into a proper probabilistic scoring rule for evaluating the distributional accuracy and diversity of generative gaze models.
- State-of-the-art performance on free-viewing and task-driven benchmarks, with ablations confirming the contribution of joint training and the informativeness of the proposed metrics in distinguishing models that capture human gaze variability from those that do not.
Contributions
Figure 1: Complementary representations of human gaze.
Prior work models gaze using either (a) scanpaths for fixation-level structure or (b) continuous trajectories for fine-grained motion dynamics. Our approach jointly leverages (c) both, aligning fixation anchors with the underlying trajectory along a shared time axis.
Motivation: Scanpaths provide a coarse structural summary of where attention dwells, while eye-tracking trajectories provide a fine-grained signal for how the eye moves between fixations. Existing generative gaze models supervise on only one of these modalities, discarding complementary cues. ST-DiffEye trains a single diffusion model jointly on both, so fixation-level structure and continuous dynamics inform a shared latent representation.
Figure 2: Overview of ST-DiffEye.
(1) The visual stimulus I and task condition c are encoded and fused into a joint representation Vjoint. (2) Each trajectory point is assigned a continuous fixation index by anchoring fixations to their nearest trajectory points and linearly interpolating between anchors. (3) The augmented trajectory tokens are denoised by a DiT conditioned on Vjoint, then decoded into a trajectory with a predicted valid length. (4) At sampling time, the generated trajectory is post-processed via fixation extraction to produce the final scanpath.
Figure 3: Why CRPS — failure modes of mean, best, and KLD protocols.
Four generators are evaluated against the same ground-truth Gaussian: zero-variance, low-variance, mid-variance, and realistic. Each non-CRPS protocol is won by a different degenerate generator (mean → zero-variance; best → low-variance; KLD → mid-variance), while only CRPS correctly identifies the realistic generator. CRPS jointly penalizes inaccuracy and insufficient diversity in a single interpretable number.
Figure 4: Qualitative comparison with baselines.
Generated scanpaths on COCO-Search18 (top: target-present and target-absent rows) and the free-viewing benchmarks COCO-FreeView and MIT1003 (bottom). ST-DiffEye produces accurate, task-consistent scanpaths in target-present search and exploratory, human-like coverage in free-viewing, while several baselines collapse to low-diversity outputs or drift from salient regions.