Fix runs on Ascent with FP32

Glaser, Jensrequested to merge
fix_ascent into clip_wciscc2024

By removing the second model in the forward pass, we can halve the memory usage

Merge request reports