All computer vision stuff is like
Step 1. Do some linear algebra that makes logical sense, uses a rational scientific model of how we believe the world works, looks reasonable
Step 2. The result of Step 1 is basically garbage. Feed it into a nonlinear optimizer to make it actually good