buffers in adios2

Created by: germasch

@chuckatkins @pnorbert: Let me open this issue to offer my thoughts the issue mentioned in #1366. What came up there is on the use of std::vector<char> as a buffer in adios2, and associated performance implications.

To reiterate some background, which is probably already well known, std::vector default-constructs new members of the vector when a non-zero size vector is created, or on vector.resize(). In particular for primitive C++ types like char, that means the memory will b zero-initialized.

The advantage to that is some amount of safety it provides, one will not end up with uninitialized contents by mistake. The disadvantage is that in many uses, zero-initialization is not needed because the buffer will be filled anyway, e.g., reading from a file or copying user provided data to it. Zero-initialization has two costs: The actual cost of what's essentially a loop setting all the values to zero, and more indirectly the cost associated with touching the memory, e.g., page faults and filling up the cache, evicting potentially more useful information.

A second, related issue is growing the buffer using resize(), which will occasionally lead to entirely new memory being allocated , in which case all of the old contents have to be copied, again having the same two kinds of performance costs.

As far as just avoiding the cost of zero-initializing when one knows it's not needed, this can actually be done with std::vector by providing a custom allocator. So that's kind of a simple fix, except that one needs to make sure that zero-initialization really isn't needed. (In BP3Serializer there just a few small uses that rely on zero-initialization).

But I think there is a point for a buffer abstraction that is more flexible and more powerful. In fact, one kinda exists: BufferSTL. It essentially provides std::vector semantics but maintains its own backing store (which happens to be std::vector). It also provides absolutePosition, but that's orthogonal, so I'll ignore that here.

The BufferSTL interface really does have a one-to-one correspondence to std::vector, but the naming is quite different. E.g.:

• bufferSTL.m_Position <-> vector.size()/resize()/end()
• bufferSTL.Resize <-> vector.reserve

I actually have changes which adds the standard-named interface to BufferSTL. One advantage to that is code gets less confusing. See #1320 (closed). That would become newblock->data.DataSize = m_BP3Serializer->m_Data.size().

It also reduces code duplication, because in BP3Serializer there are a number of functions that exist twice, once taking BufferSTL, once taking std::vector, but essentially doing the same thing. They can be consolidated after the interface renaming.

At that point BufferSTL is basically a custom buffer type, which happens to use std::vector as backing store. I've made it so that other backing stores can be substituted. (My motivation for doing that is so that I can use a mmap'd file as backing store, but that's not what this is about.)

An easy thing to do would be to use malloc and realloc for the backing store. I know that's not C++, but it'd be hidden inside of BufferSTL's vector-like C++ interface, so it wouldn't be visible to code using the buffer. A motivation for this is that on Linux, realloc can take advantage of mremap, which allows it to play virtual memory tricks to avoid actually copying any data even if there is not enough room to grow the existing buffer. That means that realloc can be orders of magnitude faster than vector.resize().

Obviously, if one provides a custom backing store, one also has control over whether to zero-initialize memory. In particular, it'd be possible to use anonymous mmap to provide he backing store (that's what malloc/new does, too, for large allocations). A great thing about that is that the memory provided will actually be zero, without having to zero it. If anyone cares, I can explain how that works in more detail -- essentially, the kernel maps the same zeroed page over and over again, marked copy-on-write, so only when actually written too, it'll be replaced by a new (zeroed) page.