Apache Kafka is an open source streaming platform which is used for integrating distributed systems in the form of data streams. Originally developed by LinkedIn in 2011, Kafka became open source in 2012. Since then, Kafka has evolved from a simple message queue to a full streaming platform for large amounts of data, where you can combine events for everything from old, traditional (legacy) systems to IoT sensors in order to develop data-driven analysis solutions.
In addition to publishing and subscribing to streams of data or events, the platform can analyse and act on combinations of events in a time window. Apache Kafka can also be used as a database for storing events/data.
Many companies now base their technology and data processing on Apache Kafka, such as Netflix, LinkedIn, and Airbnb.
At Cegal, we work with development/architecture on Kafka and the operation of Kafka. We are a partner with Confluent, which comes from the group that originally developed Kafka in LinkedIn. Confluent is now the largest contributor to the open source project Kafka. The company has established a wide range of solutions based around Kafka, which are offered in an enterprise package or as a SaaS solution with cloud providers Amazon, Google and Azure.
We work with a number of Norwegian clients and are pivotal to their use of Kafka.