Kafka
Search results
Apache Kafka is a scalable, high-performance, and fault-tolerant platform for data pipelines, streaming analytics, and mission-critical applications. Learn how Kafka is used by thousands of companies across various industries and how to connect, process, and store streams of events.
- Introduction
To process streams of events as they occur or...
- Powered By
Apache Kafka is the most popular open-source...
- Introduction
- Introduction
- Why Kafka? Benefits and Use Cases
- Kafka Architecture – Fundamental Concepts
- Kafka Components and Ecosystem
Apache Kafka is an event streaming platform used to collect, process, store, and integrate data at scale. It has numerous use cases including distributed streaming, stream processing, data integration, and pub/sub messaging. In order to make complete sense of what Kafka does, we'll delve into what an event streaming platform is and how it works. So...
Kafka is used by over 100,000 organizations across the world and is backed by a thriving community of professional developers, who are constantly advancing the state of the art in stream processing together. Due to Kafka's high throughput, fault tolerance, resilience, and scalability, there are numerous use cases across almost every industry - from...
Kafka Topics
Events have a tendency to proliferate—just think of the events that happened to you this morning—so we’ll need a system for organizing them. Kafka’s most fundamental unit of organization is the topic, which is something like a table in a relational database. As a developer using Kafka, the topic is the abstraction you probably think the most about. You create different topics to hold different kinds of events and different topics to hold filtered and transformed versions of the same kind of e...
Kafka Partitioning
If a topic were constrained to live entirely on one machine, that would place a pretty radical limit on the ability of Kafka to scale. It could manage many topics across many machines—Kafka is a distributed system, after all—but no one topic could ever get too big or aspire to accommodate too many reads and writes. Fortunately, Kafka does not leave us without options here: It gives us the ability to partitiontopics. Partitioning takes the single topic log and breaks it into multiple logs, eac...
How Kafka Partitioning Works
Having broken a topic up into partitions, we need a way of deciding which messages to write to which partitions. Typically, if a message has no key, subsequent messages will be distributed round-robin among all the topic’s partitions. In this case, all partitions get an even share of the data, but we don’t preserve any kind of ordering of the input messages. If the message does have a key, then the destination partition will be computed from a hash of the key. This allows Kafka to guarantee t...
If all you had were brokers managing partitioned, replicated topics with an ever-growing collection of producers and consumers writing and reading events, you would actually have a pretty useful system. However, the experience of the Kafka community is that certain patterns will emerge that will encourage you and your fellow developers to build the...
Franz Kafka [b] (3 July 1883 – 3 June 1924) was a German-language novelist and writer from Prague. He is widely regarded as one of the major figures of 20th-century literature. His work fuses elements of realism and the fantastic. [4]
The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems (for data import/export) via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications.
To process streams of events as they occur or retrospectively. And all this functionality is provided in a distributed, highly scalable, elastic, fault-tolerant, and secure manner. Kafka can be deployed on bare-metal hardware, virtual machines, and containers, and on-premises as well as in the cloud.
Learn what Apache Kafka is, how it works, and what it is used for. Kafka is a distributed event streaming platform that can handle large volumes of data in a scalable and fault-tolerant manner.
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
