Here is the continuation from earlier this morning, but sadly, it omits an important case near the end. I'll post it nonetheless, maybe the ideas help you or others.

**Input**Point C = our camera's location.

Triangle A = a textured element to draw.

Triangle B = another textured element to draw.

A and B share no inner points. (= All points in A ∩ B, if any, are in the boundary of A or in the boundary of B or in both boundaries.)

We're working in ℝ³, so triangles, planes, points, ... are subsets or elements of ℝ³.

A triangle is the convex hull of three pairwise-different points. Thus, triangles contain their corners and their edges, which together constitute the boundary of the triangle. An inner point of a triangle is a point of the triangle that is not in that triangle's boundary.

Distance means the Euclidean distance.

The distance between two points X and Y is

d(X, Y) = sqrt((x1-y1)² + (x2-y2)² + (x3-y3)²).

The distance between two subsets S1 and S2 of ℝ³ is

d(S1, S2) = inf(d(X, Y) : X ∈ S1, Y ∈ S2).

A distance vector from S1 to S2 of ℝ³ is a vector V with |V| = d(S1, S2) such that there is a point X ∈ S1 with X + V ∈ S2.

For arbitrary subsets S1 and S2 of ℝ³, distance vectors don't necessarily exist and are not necessarily unique. But all of our subsets of ℝ³ will be sufficiently nice (they're points, lines, triangles, planes, half-planes) so that the distance vector will always exist and will always be unique.

**Task:** Determine which of the two triangles, A or B, to paint first.

Plane EA = the infinite 2D plane that contains A.

Plane EB = the infinite 2D plane that contains B.

Vector VA = the distance vector from C to EA.

Vector VA = the distance vector from C to EB.

**Case: VA = 0 or VB = 0 (zero visible area)**If VA = 0, then the camera C is contained in EA, and the camera would only see at a 1D degeneration of A with zero 2D area. Don't paint A altogether. Likewise, if VB = 0, don't paint B. (With software, maybe we should treat small nonzero |VA| < 0.001 as this edge case, too? I worry about artifacts from floating-point math in other cases otherwise. Experiment.)

For the remaining cases, we may assume that C is contained in neither EA or EB, therefore |VA| > 0 and |VB| > 0.

**Case: VA/|VA| = VB/|VB| (Parallel Planes)**Equivalently: VA and VB are linearly dependent; one is a multiple of the other. In this case, we have parallel planes EA and EB. Paint the farther plane's triangle first: If |VA| > |VB|, paint A first. If |VA| < |VB|, paint B first. If |VA| = |VB| for the parallel planes EA and EB, you can paint either A first or B first, it doesn't matter.

Given parallel planes EA and EB, the edge case |VA| = |VB| can only happen in two ways.

Either we have VA = VB and thus EA = EB, then A and B won't overlap during rendering because we assume the points in A ∩ B are negligible boundary points.

Or we have VA = −VB, then the camera sandwiched exactly halfway between the parallel EA and EB. Here, the camera wouldn't render any overlap of the entire planes EA and EB whatsoever, thus we won't see any overlap between the particular subsets A of EA and B of EB.

**Case: Nonparallel Planes**Line X = EA ∩ EB.

Vector VX = the distance vector from C to X.

This line X will halve the planes EA and EB into two half-planes each, making four half-planes in total. The main idea will be that, for at least one plane out of EA or EB, we can clearly label one of its two resulting half-planes as "the closer half-plane" and "the farther half-plane".

Theorem: Either |VX| > |VA| or |VX| > |VB| or both.

We have |VX| ≥ |VA| because X is a subset of EA. (|VX| < |VA| is not possible.)

If |VX| = |VA|, then the point of EA closest to C must lie in X, and this point is then also the point of X closest to C, again because X is a subset of EA. This closest point is C + VX = C + VA. Thus, if |VX| = |VA|, then VX = VA. Likewise, |VX| ≥ |VB|, and if |VX| = |VB|, then VX = VB.

It remains to show that |VX| = |VA| and |VX| = |VB| can't both be true.

Assume both are true. Then VX = VA = VB. The two planes EA and EB then have identical normal vectors because a distance vector to a plane is necessarily normal to the plane. Thus, EA and EB are parallel, even identical. But we're in the case of nonparallel planes, a contradiction. End of proof.

Without loss of generality, let |VX| > |VA|. (Otherwise, swap A and B.) Then the point in EA closest to C, i.e., C + VA, lies outside X. We can now let X divide EA into two half-planes, with exactly one of them containing C + VA.

Since we don't care about boundary points of A or B, we don't care if X is part of either half-plane. The argument will be the same either way.

For sake of a sound definition, let's include X in both half-planes. Then the two half-planes aren't disjoint, their intersection will be X. Formally:

A half-plane H of a plane E is a subset of E so that there exists a bijection f: ℝ² -> E with H = f({(x, y) : y ≥ 0}) and that is affine (preserves straight lines, preserves parallel lines, is a translation plus a linear map).

Half-plane EAN ("EA near") = the half-plane of EA that contains C + VA.

Half-plane EAF ("EA far") = the half-plane of EA that does not contain C + VA.

If A is contained entirely in EAF, paint A first.

If A is contained entirely in EAN, paint B first.

Look again at the image from earlier:

X is the red line. The two planes EA and EB divide ℝ³ into four disjoint open 3D subspaces -- the air around the blue planes.

We know |VA| > 0 and |VB| > 0 from rejecting earlier cases, therefore the camera point C lies in one of the four open subspaces. What does the camera see from here? It sees a blue region that's part of one plane, then X, and then a different blue region that's part of the second plane. This holds no matter in which of the four subspaces C lies.

Now assume that EB is opaque. What part of EA does C see? It will see EAN -- all of it, even -- and it will not see EAF because the opaque EB blocks that completely.

Thus, if our triangle A is contained in EAN, paint B first, then paint A over B to show all of A. If A is contained in EAF, paint A first.

If A is neither fully contained in EAF nor in EAN, some inner points of A must intersect X.

Now it gets ugly, and I'll stop being exact. The idea is: If inner points of A intersect X, then inner points of B can still intersect X, but elsewhere in X at best. We have a nasty case analysis where we have to see if B appears above, below, left, or right of A. And by now, I doubt whether the approach so far (with EAN and EAF) is so fruitful afterall. If we can't kill this ugly case with my approach, we should probably find a new theory where this ugly case vanishes as just another standard case. I speculate that we can make progress by looking at the 3 corners of the triangle A and comparing them with the 3 corners of the triangle B.

-- Simon