Commit f079029d authored by wslink Upstream's avatar wslink Upstream Committed by Aron Helser
Browse files

wslink 2017-07-11 (d97e30e7)

Code extracted from:

    https://github.com/Kitware/wslink.git

at commit d97e30e725d810075da41b42d7df4e2f4cd1e61a (master).
parents
Copyright (c) 2017, Kitware Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# wslink
[![Build Status](https://travis-ci.org/Kitware/wslink.svg?branch=master)](https://travis-ci.org/Kitware/wslink)
Wslink allows easy, bi-directional communication between a python server and a
javascript client over a [websocket]. The client can make remote procedure
calls (RPC) to the server, and the server can publish messages to topics that
the client can subscribe to. The server can include binary attachments in
these messages, which are communicated as a binary websocket message, avoiding
the overhead of encoding and decoding.
## RPC and publish/subscribe
The initial users of wslink driving its development are [VTK] and [ParaViewWeb].
ParaViewWeb and vtkWeb require:
* RPC - a remote procedure call that can be fired by the client and return
sometime later with a response from the server, possibly an error.
* Publish/subscribe - client can subscribe to a topic provided by the server,
possibly with a filter on the parts of interest. When the topic has updated
results, the server publishes them to the client, without further action on
the client's part.
Wslink is replacing a communication layer based on Autobahn WAMP, and so one
of the goals is to be fairly compatible with WAMP, but simplify the interface
to the point-to-point communication we actually use.
## Examples
* Set up a Python (2.7 or 3.5+) [virtualenv] using requirements.txt
* Install node.js 6+ for the javascript client
* `cd wslink/js`
* `npm run build:example`
* `cd ../python`
* `python examples/webserver.py`
- starts a webserver at [localhost](http://localhost:8080/) with buttons to test RPC and pub/sub methods
* `python examples/simple.py --content client/www`
- starts the same example using the configurable server
## Testing
* additional testing dependencies are in requirements-dev.txt
* `cd python/src`
* `python tests/testWSProtocol.py`
* Uses a 'mock' of the WslinkWebSocketServerProtocol's sendMessage, to check that the expected messages or errors are generated.
## Existing API
Existing ParaViewWeb applications use these code patterns:
* @exportRPC decorator in Python server code to register a method as being remotely callable
* session.call("method.uri", [args]) in the JavaScript client to make an RPC call. Usually wrapped as an object method so it appears to be a normal class method.
* session.subscribe("method.uri", callback) in JS client to initiate a pub/sub relationship.
* server calls self.publish("method.uri", result) to push info back to the client
We don't support introspection or initial handshake about which methods are
supported - the client and server must be in sync.
Message format:
```javascript
{
const request = {
wslink: 1.0,
id: `rpc:${clientId}:${count}`,
method: 'myapp.render.window.image',
args: [],
kwargs: { w: 512, h: 512 }
};
const response = {
wslink: 1.0,
id: `rpc:${clientId}:${count}`,
result: {}, // either result or error, not both
error: {}
};
// types used as prefix for id.
const types = ['rpc', 'publish', 'system'];
}
```
```python
// add a binary attachment
def getImage(self):
return {
"size": [512, 512],
"blob": session.addAttachment(memoryview(dataArray)),
"mtime": dataArray.getMTime()
}
```
### Binary attachments
session.addAttachment() takes binary data and stores it, returning a string key
that will be associated with the attachment. When a message is sent that uses
the attachment key, a text header message and a binary message is sent
beforehand with each attachment. The client will then substitute the binary
buffer for the string key when it receives the final message.
### Subscribe
The client tracks subscriptions - the server currently blindly sends out
messages for any data it produces which might be subscribed to. This is not
very efficient - if the client notifies the server of a subscription, it can
send the data only when someone is listening. The ParaViewWeb app Visualizer
makes an RPC call after subscribing to tell the server to start publishing.
### Handshake
When the client initially connects, it sends a 'hello' to authenticate with
the server, so the server knows this client can handle the messages it sends,
and the server can provide the client with a unique client ID - which the
client must embed in the rpc "id" field of its messages to the server.
* The first message the client sends should be hello, with the secret key provided by its launcher.
* Server authenicates the key, and responds with the client ID.
* If the client sends the wrong key or no key, the server responds with an authentication error message.
### Design
More extensive discussion in the [design](design.md) document.
[ParaViewWeb]: https://www.paraview.org/web/
[virtualenv]: https://virtualenv.pypa.io/
[VTK]: http://www.vtk.org/
[websocket]: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket
r"""
Wslink allows easy, bi-directional communication between a python server and a
javascript client over a websocket.
wslink.server creates the python server
wslink.websocket handles the communication
"""
__version__ = '0.1.1'
__license__ = 'BSD-3-Clause'
from .uri import checkURI
# name is chosen to match Autobahn RPC decorator.
def register(uri):
"""
Decorator for RPC procedure endpoints.
"""
def decorate(f):
# called once when method is decorated, because we return 'f'.
assert(callable(f))
if not hasattr(f, '_wslinkuris'):
f._wslinkuris = []
f._wslinkuris.append({ "uri": checkURI(uri) })
return f
return decorate
This diff is collapsed.
r"""server is a module that enables using python through a web-server.
This module can be used as the entry point to the application. In that case, it
sets up a Twisted web-server.
web-pages are determines by the command line arguments passed in.
Use "--help" to list the supported arguments.
"""
from __future__ import absolute_import, division, print_function
import logging
# from vtk.web import testing
from wslink import upload
from wslink import websocket as wsl
from autobahn.twisted.resource import WebSocketResource
from autobahn.twisted.websocket import listenWS, WebSocketServerFactory
from twisted.web import resource
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.internet.endpoints import serverFromString
from twisted.python import log
# =============================================================================
# Setup default arguments to be parsed
# -s, --nosignalhandlers
# -d, --debug
# -p, --port 8080
# -t, --timeout 300 (seconds)
# -c, --content '/www' (No content means WebSocket only)
# -a, --authKey vtkweb-secret
# =============================================================================
def add_arguments(parser):
"""
Add arguments known to this module. parser must be
argparse.ArgumentParser instance.
"""
import os
parser.add_argument("-d", "--debug",
help="log debugging messages to stdout",
action="store_true")
parser.add_argument("-s", "--nosignalhandlers",
help="Prevent Twisted to install the signal handlers so it can be started inside a thread.",
action="store_true")
parser.add_argument("-i", "--host", type=str, default='localhost',
help="the interface for the web-server to listen on (default: localhost)")
parser.add_argument("-p", "--port", type=int, default=8080,
help="port number for the web-server to listen on (default: 8080)")
parser.add_argument("-t", "--timeout", type=int, default=300,
help="timeout for reaping process on idle in seconds (default: 300s)")
parser.add_argument("-c", "--content", default='',
help="root for web-pages to serve (default: none)")
parser.add_argument("-a", "--authKey", default='wslink-secret',
help="Authentication key for clients to connect to the WebSocket.")
parser.add_argument("-f", "--force-flush", default=False, help="If provided, this option will force additional padding content to the output. Useful when application is triggered by a session manager.", dest="forceFlush", action='store_true')
parser.add_argument("-k", "--sslKey", type=str, default="",
help="SSL key. Use this and --sslCert to start the server on https.")
parser.add_argument("-j", "--sslCert", type=str, default="",
help="SSL certificate. Use this and --sslKey to start the server on https.")
parser.add_argument("-ws", "--ws-endpoint", type=str, default="ws", dest='ws',
help="Specify WebSocket endpoint. (e.g. foo/bar/ws, Default: ws)")
parser.add_argument("--no-ws-endpoint", action="store_true", dest='nows',
help="If provided, disables the websocket endpoint")
parser.add_argument("--fs-endpoints", default='', dest='fsEndpoints',
help="add another fs location to a specific endpoint (i.e: data=/Users/seb/Download|images=/Users/seb/Pictures)")
# Hook to extract any testing arguments we need
# testing.add_arguments(parser)
# Extract any necessary upload server arguments
upload.add_arguments(parser)
return parser
# =============================================================================
# Parse arguments and start webserver
# =============================================================================
def start(argv=None,
protocol=wsl.ServerProtocol,
description="wslink web-server based on Twisted."):
"""
Sets up the web-server using with __name__ == '__main__'. This can also be
called directly. Pass the optional protocol to override the protocol used.
Default is ServerProtocol.
"""
try:
import argparse
except ImportError:
# since Python 2.6 and earlier don't have argparse, we simply provide
# the source for the same as _argparse and we use it instead.
from vtk.util import _argparse as argparse
parser = argparse.ArgumentParser(description=description)
add_arguments(parser)
args = parser.parse_args(argv)
# configure protocol, if available
try:
protocol.configure(args)
except AttributeError:
pass
start_webserver(options=args, protocol=protocol)
# =============================================================================
# Stop webserver
# =============================================================================
def stop_webserver() :
reactor.callFromThread(reactor.stop)
# =============================================================================
# Convenience method to build a resource sub tree to reflect a desired path.
# =============================================================================
def handle_complex_resource_path(path, root, resource):
# Handle complex endpoint. Twisted expects byte-type URIs.
fullpath = path.encode('utf-8').split(b'/')
parent_path_item_resource = root
for path_item in fullpath:
if path_item == fullpath[-1]:
parent_path_item_resource.putChild(path_item, resource)
else:
new_resource = Resource()
parent_path_item_resource.putChild(path_item, new_resource)
parent_path_item_resource = new_resource
# =============================================================================
# Start webserver
# =============================================================================
def start_webserver(options, protocol=wsl.ServerProtocol, disableLogging=False):
"""
Starts the web-server with the given protocol. Options must be an object
with the following members:
options.host : the interface for the web-server to listen on
options.port : port number for the web-server to listen on
options.timeout : timeout for reaping process on idle in seconds
options.content : root for web-pages to serve.
"""
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.static import File
import sys
if not disableLogging:
# redirect twisted logs to python standard logging.
observer = log.PythonLoggingObserver()
observer.start()
# log.startLogging(sys.stdout)
# Set logging level.
if (options.debug): logging.basicConfig(level=logging.DEBUG)
contextFactory = None
use_SSL = False
if options.sslKey and options.sslCert:
use_SSL = True
wsProtocol = "wss"
from twisted.internet import ssl
contextFactory = ssl.DefaultOpenSSLContextFactory(options.sslKey, options.sslCert)
else:
wsProtocol = "ws"
# Create default or custom ServerProtocol
wslinkServer = protocol()
# create a wslink-over-WebSocket transport server factory
transport_factory = wsl.TimeoutWebSocketServerFactory(\
url = "%s://%s:%d" % (wsProtocol, options.host, options.port), \
timeout = options.timeout )
transport_factory.protocol = wsl.WslinkWebSocketServerProtocol
transport_factory.setServerProtocol(wslinkServer)
root = Resource()
# Do we serve static content or just websocket ?
if len(options.content) > 0:
# Static HTTP + WebSocket
root = File(options.content)
# Handle possibly complex ws endpoint
if not options.nows:
wsResource = WebSocketResource(transport_factory)
handle_complex_resource_path(options.ws, root, wsResource)
if options.uploadPath != None :
from wslink.upload import UploadPage
uploadResource = UploadPage(options.uploadPath)
root.putChild("upload", uploadResource)
if len(options.fsEndpoints) > 3:
for fsResourceInfo in options.fsEndpoints.split('|'):
infoSplit = fsResourceInfo.split('=')
handle_complex_resource_path(infoSplit[0], root, File(infoSplit[1]))
site = Site(root)
if use_SSL:
reactor.listenSSL(options.port, site, contextFactory)
else:
reactor.listenTCP(options.port, site)
# flush ready line
sys.stdout.flush()
# Work around to force the output buffer to be flushed
# This allow the process launcher to parse the output and
# wait for "Start factory" to know that the WebServer
# is running.
if options.forceFlush :
for i in range(200):
log.msg("+"*80, logLevel=logging.CRITICAL)
# Initialize testing: checks if we're doing a test and sets it up
# testing.initialize(options, reactor, stop_webserver)
# Start the reactor
if options.nosignalhandlers:
reactor.run(installSignalHandlers=0)
else:
reactor.run()
# Give the testing module a chance to finalize, if necessary
# testing.finalize()
if __name__ == "__main__":
start()
r"""
This module provides a server side mechanism to support uploading files
to the server.
$ python file_upload_handler.py --upload-directory .../path-to-upload-directory --upload-port port
--upload-directory
Path to directory where uploaded files should go. If this argument is not
specified, then uploaded files will end up in the directory from which python
is run.
--port
Port on which upload server should listen.
"""
from __future__ import absolute_import, division, print_function
# import os to concatenate paths in a system independent way
import os
# import twisted as the webserver
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor
# import to process command line arguments
try:
import argparse
except ImportError:
# since Python 2.6 and earlier don't have argparse, we simply provide
# the source for the same as _argparse and we use it instead.
from vtk.util import _argparse as argparse
def add_arguments(parser) :
"""
Extract arguments needed by the upload server.
--upload-directory => Full path to location where files should be uploaded
"""
parser.add_argument("--upload-directory",
default=os.getcwd(),
help="path to root upload directory",
dest="uploadPath")
class UploadPage(Resource) :
isLeaf = True
uploadDirectory = os.getcwd()
def __init__(self, uploadDir) :
self.uploadDirectory = uploadDir
Resource.__init__(self)
def render_POST(self, request):
for key in request.args :
filename = os.path.join(self.uploadDirectory, key)
with open(filename, 'w') as nfd:
for lineText in request.args[key] :
nfd.write(lineText)
request.setHeader('Access-Control-Allow-Origin', '*')
return '<html><body>Your post was received</body></html>'
if __name__ == "__main__":
# Create argument parser
parser = argparse.ArgumentParser(description="File Upload Server")
add_arguments(parser)
parser.add_argument("--port",
default=8081,
help="port for upload server to listen on",
dest="port")
args = parser.parse_args()
# Now start the twisted server
resource = UploadPage(args.uploadPath)
factory = Site(resource)
reactor.listenTCP(args.port, factory)
reactor.run()
import re
componentRegex = re.compile(r"^[a-z][a-z0-9_]*$")
def checkURI(uri):
"""
uri: lowercase, dot separated string.
throws exception if invalid.
returns: uri
"""
components = uri.split('.')
for component in components:
match = componentRegex.match(component)
if not match:
raise Exception("invalid URI")
return uri
r"""
This module implements the core RPC and publish APIs. Developers can extend
LinkProtocol to provide additional RPC callbacks for their web-applications. Then extend
ServerProtocol to hook all the needed LinkProtocols together.
"""
from __future__ import absolute_import, division, print_function
import inspect, logging, json, sys, traceback
from twisted.web import resource
from twisted.python import log
from twisted.internet import reactor
from twisted.internet import defer
from twisted.internet.defer import Deferred, returnValue
from . import register as exportRpc
from autobahn.twisted.websocket import WebSocketServerFactory
from autobahn.twisted.websocket import WebSocketServerProtocol
# =============================================================================
#
# Base class for objects that can accept RPC calls or publish over wslink
#
# =============================================================================
class LinkProtocol(object):
"""
Subclass this to communicate with wslink clients. LinkProtocol
objects provide rpc and pub/sub actions.
"""
def __init__(self):
self.publish = None
self.addAttachment = None
self.coreServer = None
def init(self, publish, addAttachment):
self.publish = publish
self.addAttachment = addAttachment
def getSharedObject(self, key):
assert(self.coreServer)
return self.coreServer.getSharedObject(key)
# =============================================================================
#
# Base class for wslink ServerProtocol objects
#
# =============================================================================
class ServerProtocol(object):
"""
Defines the core server protocol for wslink. Gathers a list of LinkProtocol
objects that provide rpc and publish functionality.
"""
def __init__(self):
self.linkProtocols = []
self.secret = None
self.initialize()
def init(self, publish, addAttachment):
self.publish = publish
self.addAttachment = addAttachment
def initialize(self):
"""
Let sub classes define what they need to do to properly initialize
themselves.
"""
pass
def setSharedObject(self, key, shared):
if not hasattr(self, "sharedObjects"):
self.sharedObjects = {}
if (shared == None and key in self.sharedObjects):
del self.sharedObjects[key]
else:
self.sharedObjects[key] = shared
def getSharedObject(self, key):
if (key in self.sharedObjects):
return self.sharedObjects[key]
else:
return None
def registerLinkProtocol(self, protocol):
assert( isinstance(protocol, LinkProtocol))
protocol.coreServer = self
self.linkProtocols.append(protocol)
# Note: this can only be used _before_ a connection is made -
# otherwise the WslinkWebSocketServerProtocol will already have stored references to
# the RPC methods in the protocol.
def unregisterLinkProtocol(self, protocol):
assert( isinstance(protocol, LinkProtocol))
protocol.coreServer = None
try:
self.linkProtocols.remove(protocol)
except ValueError as e:
log.error("Link protocol missing from registered list.")
def getLinkProtocols(self):
return self.linkProtocols
def updateSecret(self, newSecret):
self.secret = newSecret
@exportRpc("application.exit")
def exit(self):