Event streams

Xapix processes streams of data (Kafka event streams) in real-time. Kafka event streams consume data from sources such as IoT sensors that generate data traffic on a regular time schedule or in random events.

For example, you can use Xapix to transform data from these sources and send to a webhook service for consumption by applications or to a different Kafka topic.

About Kafka event streams

Kafka event streams are a continuously updated data set "of unknown or unlimited size". They are a reliable published streams of data records (or messages).

Topic

A Topic is the key abstraction of a Kafka event stream.

Kafka is run on Kafka Servers as a cluster. In a Kafka cluster, streams of records (data) are stored in categories called Topics. Each Topic contains a key, a value and a timestamp. Each Kafka event stream in Xapix contains a single topic.

Topics in a Kafka event stream

Producer and Consumer

Kafka event streams have one or many Event Producers and one or many Event Consumers.

Event Producers publish data events (messages) to topics. Event Consumers read data published by topics. For a Consumer to read published data, they must be subscribed to a topic.

Xapix Consumer reading published data from a Kafka event stream

Each Consumer is part of a Consumer Group, even if the group contains only one Consumer. Consequently, the consumer group receives topics to which a consumer belonging to the group is subscribed.

In Xapix, you define a single Consumer Group with one Consumer.

Sink

In a Kafka event stream, a Sink exports data to any external system. However, in Xapix, the Sink provides a termination of the pipeline.

Pipeline

Kafka event stream transformed/orchestrated in Xapix pipeline

In a Xapix pipeline, the Event unit connects to a Kafka event stream. The Topic data is orchestrated/transformed based on the design of the pipeline. The transformed data is made available via webhooks.

The following screenshot shows a completed pipeline using a Kafka event stream. It contains the usual elements of a pipeline built for a REST endpoint and you use the same techniques to build it.

Pipeline using Kafka endpoints and event streams
Section topics
Section topics