Commit ec8f4a38 authored by Atmn Patel's avatar Atmn Patel
Browse files

[OpenMP][Libomptarget] Introduce Remote Offloading Plugin 

This introduces a remote offloading plugin for libomptarget. This
implementation relies on gRPC and protobuf, so this library will only
build if both libraries are available on the system. The corresponding
server is compiled to `openmp-offloading-server`.

This is a large change, but the only way to split this up is into RTL/server
but I fear that could introduce an inconsistency amongst them.

Ideally, tests for this should be added to the current ones that but that is
problematic for at least one reason. Given that libomptarget registers plugin
on a first-come-first-serve basis, if we wanted to offload onto a local x86
through a different process, then we'd have to either re-order the plugin list
in `rtl.cpp` (which is what I did locally for testing) or find a better
solution for runtime plugin registration in libomptarget.

Differential Revision: https://reviews.llvm.org/D95314
parent 15313f64

openmp/docs/SupportAndFAQ.rst

+9 −0
Original line number Diff line number Diff line
@@ -109,3 +109,12 @@ through a similar mechanism. It is worth noting that this support requires 
`extensions to the OpenMP begin/end declare variant context selector

<https://clang.llvm.org/docs/AttributeReference.html#pragma-omp-declare-variant>`__

that are exposed through LLVM/Clang to the user as well.



Q: What is a way to debug errors from mapping memory to a target device?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^



An experimental way to debug these errors is to use :ref:`remote process 

offloading <remote_offloading_plugin>`.

By using ``libomptarget.rtl.rpc.so`` and ``openmp-offloading-server``, it is

possible to explicitly perform memory transfers between processes on the host

CPU and run sanitizers while doing so in order to catch these errors.
 
 No newline at end of file

openmp/docs/design/Runtimes.rst

+56 −0
Original line number Diff line number Diff line
@@ -273,6 +273,62 @@ LLVM/OpenMP Target Host Runtime Plugins (``libomptarget.rtl.XXXX``) 


.. _device_runtime:





.. _remote_offloading_plugin:



Remote Offloading Plugin:

^^^^^^^^^^^^^^^^^^^^^^^^^



The remote offloading plugin permits the execution of OpenMP target regions

on devices in remote hosts in addition to the devices connected to the local

host. All target devices on the remote host will be exposed to the

application as if they were local devices, that is, the remote host CPU or

its GPUs can be offloaded to with the appropriate device number. If the

server is running on the same host, each device may be identified twice:

once through the device plugins and once through the device plugins that the

server application has access to.



This plugin consists of ``libomptarget.rtl.rpc.so`` and

``openmp-offloading-server`` which should be running on the (remote) host. The

server application does not have to be running on a remote host, and can

instead be used on the same host in order to debug memory mapping during offloading.

These are implemented via gRPC/protobuf so these libraries are required to

build and use this plugin. The server must also have access to the necessary

target-specific plugins in order to perform the offloading.



Due to the experimental nature of this plugin, the CMake variable

``LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN`` must be set in order to

build this plugin. For example, the rpc plugin is not designed to be

thread-safe, the server cannot concurrently handle offloading from multiple

applications at once (it is synchronous) and will terminate after a single

execution. Note that ``openmp-offloading-server`` is unable to

remote offload onto a remote host itself and will error out if this is attempted.



Remote offloading is configured via environment variables at runtime of the OpenMP application:

    * ``LIBOMPTARGET_RPC_ADDRESS=<Address>:<Port>``

    * ``LIBOMPTARGET_RPC_ALLOCATOR_MAX=<NumBytes>``

    * ``LIBOMPTARGET_BLOCK_SIZE=<NumBytes>``

    * ``LIBOMPTARGET_RPC_LATENCY=<Seconds>``



LIBOMPTARGET_RPC_ADDRESS

""""""""""""""""""""""""

The address and port at which the server is running. This needs to be set for

the server and the application, the default is ``0.0.0.0:50051``. A single

OpenMP executable can offload onto multiple remote hosts by setting this to

comma-seperated values of the addresses.



LIBOMPTARGET_RPC_ALLOCATOR_MAX

""""""""""""""""""""""""""""""

After allocating this size, the protobuf allocator will clear. This can be set for both endpoints.



LIBOMPTARGET_BLOCK_SIZE

"""""""""""""""""""""""

This is the maximum size of a single message while streaming data transfers between the two endpoints and can be set for both endpoints.



LIBOMPTARGET_RPC_LATENCY

""""""""""""""""""""""""

This is the maximum amount of time the client will wait for a response from the server.



LLVM/OpenMP Target Device Runtime (``libomptarget-ARCH-SUBARCH.bc``)

--------------------------------------------------------------------

openmp/libomptarget/plugins/CMakeLists.txt

+1 −0
Original line number Diff line number Diff line
@@ -74,6 +74,7 @@ add_subdirectory(ppc64) 
add_subdirectory(ppc64le)

add_subdirectory(ve)

add_subdirectory(x86_64)

add_subdirectory(remote)



# Make sure the parent scope can see the plugins that will be created.

set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS}" PARENT_SCOPE)

openmp/libomptarget/plugins/exports

+2 −0
Original line number Diff line number Diff line
@@ -19,6 +19,8 @@ VERS1.0 { 
    __tgt_rtl_run_target_region;

    __tgt_rtl_run_target_region_async;

    __tgt_rtl_synchronize;

    __tgt_rtl_register_lib;

    __tgt_rtl_unregister_lib;

  local:

    *;

};

openmp/libomptarget/plugins/remote/CMakeLists.txt

0 → 100644
+55 −0
Original line number Diff line number Diff line 
##===----------------------------------------------------------------------===##

# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.

# See https://llvm.org/LICENSE.txt for license information.

# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

#

##===----------------------------------------------------------------------===##

#

# Build a plugin (client) and server for remote offloading.

#

##===----------------------------------------------------------------------===#



cmake_minimum_required(VERSION 3.15)



if (NOT(CMAKE_SYSTEM_NAME MATCHES "Linux"))

  libomptarget_say("Not building remote offloading plugin: only support Linux hosts.")

  return()

endif()



if (NOT(LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN))

  return()

endif()



find_package(Protobuf)

find_package(gRPC CONFIG)



find_program(PROTOC protoc)

find_program(GRPC_CPP_PLUGIN grpc_cpp_plugin)



if (Protobuf_FOUND AND gRPC_FOUND AND PROTOC AND GRPC_CPP_PLUGIN)

  libomptarget_say("Building remote offloading plugin.")

  set(directory "${CMAKE_BINARY_DIR}/include/openmp/libomptarget/plugins/remote/")

  file(MAKE_DIRECTORY ${directory})

  execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory ${directory})

  execute_process(

          COMMAND protoc --cpp_out=${directory} -I ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/openmp.proto

          COMMAND protoc --grpc_out=${directory} -I ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/openmp.proto --plugin=protoc-gen-grpc=${GRPC_CPP_PLUGIN}

  )



  set(GRPC_SRC_FILES

      ${directory}/openmp.grpc.pb.cc

      ${directory}/openmp.pb.cc

  )



  set(GRPC_INCLUDE_DIR

      ${directory}

  )

else()

  libomptarget_say("Not building remote offloading plugin: required libraries were not found.")

endif()



set(RPC_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/include/)

set(RPC_SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR}/lib/)



add_subdirectory(src)

add_subdirectory(server)