2019-08-21

Clojud

With this release we are replacing our previous data processing engine (Firebird) with Clojud.
Clojud uses Kafka to balance workload, store pipeline configurations and intermediate state. It also leverages Kafka Connect to implement Event Streams and to push Analytics to Elasticsearch. All data in Clojud Kafka topics are in JSON format and immediately available for audit and analysis.
Due to this change, this upgrade cannot be installed without downtime. All projects have to be published to Clojud after updating. This can be done either through the Phoenix Web UI for each project, or you can run the following command after updating:
1
NAMESPACE=xapix
2
kubectl exec -n $NAMESPACE $(kubectl get pods --selector=app=phoenix -n $NAMESPACE --output=jsonpath="{.items..metadata.name}") -- bundle exec rake phoenix:project:publish_all
Copied!

New Features

    Clojud is the new data processing backend, leveraging the power of Kafka and Kafka Connect to deliver improved scalability, stability and traceability.
    Map Unit is now in preview. The Map Unit allows you to use sub-pipelines to easily process collections. After enabling the via the 'map_unit' feature flag, you can add Map Units to your pipelines, define the collection to process and then create a pipeline which will be executed for each element of the collection. The result of each sub-pipeline will then be used to create the resulting collection which then can be used as input in other units. Known limitations:
      Export of projects using map units isn't available at the moment
      In sub-pipelines, when adding a data source unit, selecting 'Get Credentials from Request Unit' will not automatically connect those credentials to the given data source unit (it must be done manually).

Other changes

    UI cleanups
    Resolved an issue that prevented adding units in Safari
    improved line drawing for data mappings
    Our helm charts now respect username, password and TLS settings for external elasticsearch clusters
    Clojud exposes more metrics to Prometheus: number/duration of incoming http requests, number/duration of outgoing http requests
Last modified 1yr ago
Export as PDF
Copy link