Allow driving LWR via message queue.
New abstraction for creating client managers (build_client_manager function) - if this is passed a URL - it will be treated as an AMQP URL and clients will stage by sending a single message via a queue using that connection. By default it will assume communication with a remote manager named '_default_' (the out of the box one for the LWR) - different LWR managers can be targetted by adding 'manager' as an argument to the client manager. The upshot of this is only one LWR (or one LWR with any given manager name) should target a given AMQP endpoint right now. On the backend, message_queue_url should be defined in main section of the server.ini properties and all manaagers will bind to this URL. Again, it will be nice to allow more customization and allowing different managers to bind to different URLs, etc.... Couple more notes, right now all file actions must be executable by the LWR (instead of on client side) so "none" or "remote_copy" are the only valid file actions if the message queue variant is going to work and a shared file system is needed between Galaxy data and remote LWR server. The staging directory does not need to be shared however, so this should real immediate utility in some environments (if the LWR server can mount the Galaxy data and then a staging/scratch area used by the compute cluster the solution should be workable - and it should be much easier technically and politically to just mount Galaxy's data on one LWR server then a whole compute cluster). Added a full integration test of this form of LWR as well as smaller test of AMQP abstraction shared between client and server for consuming and publishing messages.
Showing
- lwr/app.py 13 additions, 1 deletionlwr/app.py
- lwr/lwr_client/__init__.py 2 additions, 2 deletionslwr/lwr_client/__init__.py
- lwr/lwr_client/amqp_exchange.py 70 additions, 0 deletionslwr/lwr_client/amqp_exchange.py
- lwr/lwr_client/client.py 78 additions, 40 deletionslwr/lwr_client/client.py
- lwr/lwr_client/manager.py 61 additions, 9 deletionslwr/lwr_client/manager.py
- lwr/lwr_client/staging/up.py 0 additions, 2 deletionslwr/lwr_client/staging/up.py
- lwr/managers/staging/postprocess.py 8 additions, 1 deletionlwr/managers/staging/postprocess.py
- lwr/managers/stateful.py 29 additions, 11 deletionslwr/managers/stateful.py
- lwr/managers/status.py 2 additions, 0 deletionslwr/managers/status.py
- lwr/managers/unqueued.py 5 additions, 4 deletionslwr/managers/unqueued.py
- lwr/messaging/__init__.py 29 additions, 0 deletionslwr/messaging/__init__.py
- lwr/messaging/bind_amqp.py 38 additions, 0 deletionslwr/messaging/bind_amqp.py
- lwr/routes.py 1 addition, 1 deletionlwr/routes.py
- test/amqp_test.py 48 additions, 0 deletionstest/amqp_test.py
- test/check.py 29 additions, 6 deletionstest/check.py
- test/integration_test.py 3 additions, 1 deletiontest/integration_test.py
Loading
Please register or sign in to comment