More parallel+serial discussion

Created by: germasch

This is follow-up from #1443 (closed) and PR #1466. The PR addresses the fundamentals of the issue, but the work isn't really complete.

Open issues:

  • python bindings support (I think @pnorbert is working on this, should be relatively straightforward).
  • which parts of adios2 make sense to support in serial mode? (InsituMPI: probably not, but maybe all / most of the rest)? Right now, BP3/BP4 and HDF5 are supported, if lightly tested.
  • SST should be supported, but is harder because it's a separate lib, and uses C.
  • corner cases with the current implementation.

The current approach is to replace MPI_* functions by their SMPI_* wrapper equivalents. These wrappers will call through to the real MPI if it's initialized (and not finalized), otherwise it'll do the MPI dummy stuff. I somewhat deliberately only changed selected pieces to use those wrappers, since I'm not too convinced that it's a real clean approach, though I'm torn about that. It could however, be considered to do a global search and replace and use the wrappers everywhere. Some things still wouldn't work (e.g., no easy way to implement MPI_Send/MPI_Recv properly on a single proc). The current state isn't horrible, ie., if someone were to try to use some part which hasn't been converted to SMPI_* wrappers w/o calling MPI_Init, they'd get MPI complaining about just that, rather than some mysterious subtle problms.

What I think would be cleaner, but potentially involve some code duplication, would be to have separate code paths for serial/parallel case where both are supported. e.g., in the serial case, AggregateMetadata probably shouldn't even be called. This, however, requires being able to distinguish whether a function is meant to execute serially. I think there is one straightforward way to make that distinction, ie., define MPI_COMM_NULL as having the meaning "don't use MPI, this executes serially". That's clearer than the current reliance on MPI_Initialized/MPI_Finalized; that status, one should note, might actually change along the way. So it's possible to have SMPI_Comm_dup return the communicator itself when called before MPI_Init (because that's what mpidummy) does, then initialize MPI, and then eventually have SMPI_Comm_free call the real MPI_Comm_free, which will rightfully complain that the communicator hadn't been dup'd in the first place. This actually happens in real life, which is why I didn't change the MPI_Comm_dup handling inside of core::ADIOS for now.

Switching between real MPI and mpidummy based on the communicator (MPI_COMM_NULL vs not) resolves a lot of ambiguity -- it's also cheaper than having to check MPI_Initialized and MPI_Finalized on every MPI call, so I'd be in favor of going that way. That's also a pretty small/localized change.

On the bigger issue (wrap everything with SMPI vs separate code paths), I think one might have try and look at it to decide what's better.