Get rid of unnecessary fences

Created by: dalg24

We have been very conservative in the past when we were forced to use UVM and added fences all over the place. This need to be re-evaluated.

NB: Make sure to add UVM to the CI before starting to work on this (addressed in #46)