TransportMan mods essentially disable background Opens
So, while PR #1984 does resolve TSAN's concerns, it looks like it does so at the cost of nearly completely eliminating the optimization for which the async was introduced. Specifically, we're waiting for the async thread at the beginning of every TransportMan call. BP3/4 call OpenFilesAsync inside InitTransports(), and in each the very next subroutine call is InitBPBuffers(), which calls TransportManager::GetTransportsTypes(), which now waits for the async task so that it can put the transport type in the BP buffers that it's initializing. The goal of letting substantial other work get done before we wait for the Open() is not fulfilled in any meaningful way.
However, this current arrangement seems unnecessary and I'm hoping @pnorbert can confirm. GetTransportsTypes returns a vector of strings (because what's not a string?) which match the vector of filenames that the OpenFilesAsync call is opening and which get passed to PetProcessGroupIndex. Currently these file type values are set only after the files are opened, hence the required wait to get them. But it seems like the file types should be predictable. I.E. don't we know the type of the file we're trying to create before we create it? (Types here are "File_NULL", "File_POSIX", "File_fstream", "File_stdio", and "WAN_zmq".). So presumably this could be refactored so that we actually achieved some asynchrony here.. Norbert?