Face Alignment Errors Explained
Key findings
"Face Alignment" is the mathematical process of warping a source face to match the geometry of a target face. This process typically relies on just 5 key landmarks: the left eye, right eye, nose tip, left mouth corner, and right mouth corner. An "Alignment Error" occurs when the AI misidentifies one of these points—usually due to occlusion, extreme angles, or shadow—causing the swapped face to appear twisted, stretched, or sliding off the head. Because the entire face geometry hangs on these 5 anchors, a single pixel of error in landmark detection can rotate the entire jawline by several degrees.
Applicable Scope
This explanation applies to landmark-based one-shot swappers (using insightface / 2dfan4). It explains why faces sometimes look "Picasso-esque" or detached.
What the phenomenon looks like
- "The mouth is printed on the cheek."
- "One eye is higher than the other, like a Picasso painting."
- "The face seems to be sliding off the head during a turn."
- "The jawline is vibrating or detached from the neck."
- "Green dots (debug landmarks) are appearing in the final video."
These are the sorts of search queries and troubleshooting posts that should route to this page.
When this problem appears most often
Alignment fails when the "5 Anchors" are hard to find:
- Extreme Side Profile (>60 degrees): The "far eye" and "far mouth corner" are hidden behind the nose. The AI has to guess where they are. If it guesses wrong, it stretches the face across the nose bridge.
- Occlusion (Hand/Hair): If a hand covers the mouth, the detector might mistake a knuckle or shadow for the "mouth corner." The AI then warps the lips to attach to the hand.
- Looking Up/Down: Extreme pitch angles compress the vertical distance between eyes and mouth. The aligner might flatten the face, making it look squashed.
- Low Resolution: If the eyes are blurry blobs, the landmark detector jitters, causing the whole face to shake (see Flicker and Drift).
Why this happens
The swapper is trying to solve a 3D problem with a 2D tool.
1. The 5-Point Trap
Most one-shot swappers don't build a 3D mesh of the head. They perform a 2D Affine Transformation (scale, rotate, shear) based on 5 points.
- The Fragility: It's like hanging a poster with 5 pins. If you move one pin (e.g., the right eye landmark is 2 pixels too low), the whole poster wrinkles and twists.
2. The "Mean Face" Assumption
The identity embedding (inswapper) is essentially a texture map of a front-facing face.
- The Stretch: To fit a side profile, the AI has to stretch the "far side" of this texture. It doesn't know what the side of the nose looks like; it just stretches the front-nose pixels. This causes the "smear" effect on profiles.
3. Debug Artifacts (The "Green Dots")
Sometimes, users see green or red dots on the face in the final video.
- The Cause: This is a user error—leaving the
face_debugger(preview tool) active during the final render. The software literally renders the landmark points onto the frame.
Trade-offs implied
- Landmark Score Threshold:
- High Score (Strict): Rejects frames where landmarks are uncertain. Result: No twisted faces, but lots of flickering (face disappears during turns).
- Low Score (Loose): Swaps everything. Result: Face stays on during turns, but looks monstrous/twisted when angles get extreme.
- 3D vs 2D Models:
- 2D Swappers (FaceFusion): Fast, works on any GPU. Prone to profile errors.
- 3D Swappers (DeepFaceLab): Can handle profiles perfectly by building a 3D mesh. Cost: Requires weeks of training and massive VRAM.
Frequently asked questions (short answers)
Q: Can I manually fix the alignment points? A: Not in one-shot automated tools like FaceFusion. You can do this in manual tools like DeepFaceLab (XSeg), but it requires frame-by-frame editing.
Q: Why is the mouth open when the actor is closed? A: The landmarks for the "inner mouth" were detected open (perhaps due to shadow). The AI warped the open-mouth texture to fit those points.
Q: Why does the face look "flat"? A: The alignment process flattens the 3D curvature of the face into a 2D plane. It loses the "depth" of the nose and cheekbones, especially in lighting that relies on 3D shadows.
Q: How do I remove the green dots?
A: Turn off the face_debugger or "Preview" overlay before hitting "Start" or "Render."
Related phenomena (links)
- Why Video Face Swaps Flicker and Drift – When unstable landmarks cause shaking.
- Why Some Faces Can't Be Swapped by AI – When landmarks are totally lost.
- Why Face Swap Often Looks Unnatural – The visual result of slight misalignment.
Final perspective
Alignment errors are the "cracks" in the illusion of 2D face swapping. Until one-shot tools adopt real-time 3D mesh fitting (which is computationally heavy), side profiles and extreme angles will always remain the "danger zone" for artifacts. The best fix is to avoid source footage with >60 degree head turns.
