Remove some unnecessary includes, replace some with forward
declarations.

This also exposed a few places that were missing some includes.