Keep permutation on the device

Arndt, Danielrequested to merge
github/fork/masterleinad/keep_permute_on_device into master

Created by: masterleinad

The changes here should only affect a CUDA-aware MPI implementation.

  • If the DeviceType's memory space is Kokkos::Host we, of course, do everything in host memory space.
  • If the DeviceType's memory space is Kokkos::Cuda and we enable KOKKOS_USE_CUDA_AWARE_MPI, the permutation array now is stored on the device and Distributor::doPostsAndWaits doesn't allow memory on the device anymore, but just uses the permutation array directly since the input array is stored on the device.
  • If the DeviceType's memory space is Kokkos::Cuda and KOKKOS_USE_CUDA_AWARE_MPI is disabled, the input array for Distributor::doPostsAndWaits is stored on the host so we should also create the permutation array on the host.

Merge request reports