Back to all blogs
A research thesis on distinguishing genuinely present participants from replayed, prerecorded, or generated participants using temporal, behavioral, and causal interaction signals.
Abhishek Kaushik
In series of experiments at Sherlock.sh, our team found, a convincing video frame does not prove that a participant is live.
A prerecorded video can look perfectly authentic. A replayed stream can contain a real person. A sufficiently strong generated avatar may eventually produce pixels that are nearly indistinguishable from a webcam feed. If liveness is defined only as a visual classification problem, the detector is always competing against improvements in image generation.
We think the stronger formulation is different:
Liveness is a property of interaction, not merely video appearance.
A genuinely present participant exists inside a causal loop with the conversation. They hear a question, process it, react, interrupt, hesitate, shift gaze, change facial expression, respond to unexpected events, and adapt to the behavior of other participants. These events create temporal structure. A prerecorded or replayed participant may reproduce realistic pixels, but it must also reproduce the correct relationship between what just happened in the interaction and what happens next.
That creates a different research problem. Consider an video interview where an interviewer asks: “Can you explain the tradeoff in the approach you just described?”. A live participant may begin responding after 700 milliseconds, interrupt the interviewer slightly, look upward while thinking, revise the previous answer, and react when the interviewer asked challenges an assumption.
The important signal is not any one of these behaviors. It is the causal chain: question → perception → cognitive delay → behavioral response → conversational adaptation
A replay attack must somehow satisfy that chain despite the future question being unknown. A generated participant must synthesize not only plausible speech and video, but also produce them with appropriate timing, coordination, and responsiveness.
Our hypothesis is that this interaction structure contains measurable liveness information. So, the question we probed was what could constitute interaction-level liveness?
Several signals may become meaningful when considered jointly. Response latency that tells - does the participant react within a plausible distribution after unpredictable prompts? Turn-taking that tells - do interruptions, overlaps, pauses, and conversational handoffs behave like a participant actively tracking the dialogue? Audiovisual causality that tells - does facial movement, gaze, posture, and expression change in ways that are temporally consistent with what was just heard or said? Contextual adaptation - does the participant respond correctly when a question references something that happened several minutes earlier? Unexpected interaction events - can the participant adapt when another speaker interrupts, corrects them, changes direction, or introduces an unanticipated request? Challenge-response - can the system occasionally introduce a low-friction unpredictable action whose correct response would be difficult to prepare in advance?
None of these signals individually proves liveness.
The research problem is whether their joint temporal structure can. Why this is hard: Human interaction is noisy. People pause. Network latency varies. Some speakers respond slowly. Others interrupt constantly. Neurodiversity, language fluency, cultural norms, poor connectivity, and interview format can all change timing behavior.
A useful system cannot simply learn: “fast response = live” or: “unusual gaze = suspicious.” It must distinguish legitimate variability from broken causal structure. There is also an adversarial problem. Once attackers understand which signals are measured, generated systems can begin optimizing for them. A robust detector therefore cannot depend on a small fixed set of behavioral heuristics. The harder goal is to test whether the participant is embedded in the same causal timeline as everyone else in the meeting.
Our central research question was:
Can interaction dynamics reliably distinguish a genuinely present participant from prerecorded, replayed, or generated participation, even when individual frames and audio samples appear authentic?
For Sherlock, this moves liveness detection away from pixel forensics alone. The system would not merely ask: “Does this person look real?”. It would ask: “Is this participant continuously perceiving, responding to, and causally participating in this interaction as it unfolds?”
That is a much stronger definition of being live.
