v0.16.0

Clojud

With this release we are replacing our previous data processing engine (Firebird) with Clojud.

Clojud uses Kafka to balance workload, store pipeline configurations and intermediate state. It also leverages Kafka Connect to implement Event Streams and to push Analytics to Elasticsearch. All data in Clojud Kafka topics are in JSON format and immediately available for audit and analysis.

Due to this change, this upgrade cannot be installed without downtime. All projects have to be published to Clojud after updating. This can be done either through the Phoenix Web UI for each project, or you can run the following command after updating:

NAMESPACE=xapix
kubectl exec -n $NAMESPACE $(kubectl get pods --selector=app=phoenix -n $NAMESPACE --output=jsonpath="{.items..metadata.name}") -- bundle exec rake phoenix:project:publish_all

New Features

  • Clojud is the new data processing backend, leveraging the power of Kafka and Kafka Connect to deliver improved scalability, stability and traceability.

  • Map Unit is now in preview. The Map Unit allows you to use sub-pipelines to easily process collections. After enabling the via the 'map_unit' feature flag, you can add Map Units to your pipelines, define the collection to process and then create a pipeline which will be executed for each element of the collection. The result of each sub-pipeline will then be used to create the resulting collection which then can be used as input in other units. Known limitations:

    • Export of projects using map units isn't available at the moment

    • In sub-pipelines, when adding a data source unit, selecting 'Get Credentials from Request Unit' will not automatically connect those credentials to the given data source unit (it must be done manually).

Other changes

  • UI cleanups

  • Resolved an issue that prevented adding units in Safari

  • improved line drawing for data mappings

  • Our helm charts now respect username, password and TLS settings for external elasticsearch clusters

  • Clojud exposes more metrics to Prometheus: number/duration of incoming http requests, number/duration of outgoing http requests