External data sources (preview)

This feature is in preview and is available by request.

Onboarding external data sources is currently in preview mode. If you don't see External Data Source in menu this means that feature is not enabled for your setup.

For more information, contact Xapix.

Data from an external data source that is not available as usual Xapix data sources can be polled using an External Executor.

Examples of external data sources in this category include:

  • SQL databases

  • File systems of remote machines

  • ssh command executors

An external data source is different from usual REST and SOAP data sources conceptually but shares the same interface for convenience purposes.

About the External Executor

The External Executor uses a third-party service, namely, an External Executor service, as an interface to get data.

On startup, this service connects to a Clojud instance via a websocket, listens to the request, does any necessary tasks and then sends back a reply.

See External Executor service example below.

This means that a user can put data from any kind of data source into a pipeline but with the cost of developing and managing third-party service.

External executor service example

An External Executor service must follow Clojud external executor protocol and be able to connect to Clojud instances via a websocket.

The following shows an example of a simple python script that increments an input value.

#!/usr/bin/env python3
import os
import asyncio
import json
import time
import websockets
endpoint = os.environ.get('CLOJUD_ENDPOINT', "ws://localhost:4020/api/v1/executor?name=incrementer")
print("Starting incrementer loop... ")
async def executor():
while True:
try:
async with websockets.connect(endpoint) as websocket:
print(f"Connected to {endpoint}... ")
while True:
req = json.loads(await websocket.recv())
print(f"< {req}")
reply = {
'type': 'reply',
'id': req['id'],
'result': 'success',
'response': [{'callback': req['callbacks']['success'],
'args': [req['payload']['value'] + 1]}]
}
print(f"> {reply}")
await websocket.send(json.dumps(reply))
except websockets.exceptions.ConnectionClosed:
print("Disconnected, reconnecting in 1 second")
time.sleep(1)
except OSError:
print("Connection refused, reconnecting in 1 second")
time.sleep(1)
asyncio.get_event_loop().run_until_complete(executor())

This python script has the following features:

  • Connects to Clojud via websocket using CLOJUD_ENDPOINT (line 9) and waits for requests.

  • Request has id (line 23) and callbacks (line 25) fields.

  • Same id should be put in reply (line 28).

  • Response has callback (line 25) and args (line 26) fields.

Adding an external data source

Steps
Steps
  1. In a project, select Home > Data Sources.

    If your project does not have any data sources, click Create a new Data Source. Otherwise, Click Add Data Source

  2. Choose External Data Source.

  3. Provide a Name for the External executor.

  4. Enter the payload parameters.

  5. Click Preview Data Source (you may need to provide values for detected path parameters). You can view the response in the Preview area.

  6. Click Save Data Source.

Example
Example
External data source added
Parameters for an external data source added

External executor name should be equal to a registered name. For example, if executor used ws://clojud-host:4020/api/v1/executor?name=external-name as an endpoint to connect to Clojud, then external-name must be used as External executor name.