News
Building a Real-Time Weather Streaming Pipeline with Kafka, Docker & Python
17+ hour, 50+ min ago (17+ words) Introduction In modern data engineering, handling high-velocity, real-time streams. .. Tagged with luxdev, dataengineering, kafka, docker....
Real-Time APIs Are Simpler Than You Think: Redis, Lua, and 4k Updates/sec
1+ day, 18+ hour ago (882+ words) Building real-time systems is often presented as a distributed systems problem. Kafka, stream processors, event buses, fanout pipelines, multiple caches " the architecture diagrams usually become complicated very quickly. But the problem we were trying to solve was actually much simpler....
Building a Mission-Critical Integration Blind, Part I: XML, Queues, and Guesswork
3+ day, 13+ hour ago (72+ words) Hacker Noon Building a Mission-Critical Integration Blind, Part I: XML, Queues, and Guesswork Software engineer with 12+ years of experience in high-load systems, search, and data platforms Advanced Rabbit MQ Support Part II: Deeper Insight into Queues Basics of Rabbit MQ Broker Building…...
Getting started with Change Data Capture in Amazon Aurora DSQL
3+ day, 18+ hour ago (1531+ words) With the introduction of CDC, Aurora DSQL now supports native streaming of database changes to downstream services. CDC captures row-level modifications and delivers them to external systems in near real time. In this post, we demonstrate how to configure Aurora…...
When (and when not) to use Apache Kafka" Diskless Topics
3+ day, 23+ hour ago (1420+ words) May 14, 2026 | By Paul Brebner I recently wrote a Visual Guide to Apache Kafka Diskless Topics, which introduces the main ideas behind Kafka Diskless Topics and links to the relevant Kafka KIPs. At present, the only accepted KIP is the high-level…...
Kafka ile Event Driven Architecture (EDA): Event ve Channel Tasar'm'n' Do'ru Yapmak
4+ day, 19+ hour ago (730+ words) Bu yaz, Event Driven Architecture ve Kafka tabanl event/channel tasarmn T'rk'e teknik kaynak ihtiyacn da g'zeterek hazrlanm" bir yaz dizisinin ilk b'l'm'd'r. Bir event, sistemde ger'ekle"mi" anlaml bir i" olayn temsil eder: Buradaki 'nemli nokta "udur: Event bir…...
Faster Queries in Local Laptop Mode for Apache Spark
5+ day, 14+ hour ago (148+ words) Databricks Faster Queries in Local Laptop Mode for Apache Spark" Apache Spark" is optimized for distributed large-scale processing, but this architecture introduces significant overhead for small, local datasets " often making sub-100 MB queries take several seconds. In this talk, we…...
SASL-OAuthbearer with AWS Lambda: How I Stopped Fighting Kafka Auth at 2am
6+ day, 3+ hour ago (1156+ words) TL; DR: The thing that caught me off guard was how silent the failure was. My Lambda function was trying to connect to an MSK cluster, the connection timed out, and the only thing in Cloud Watch was `org. " Reading…...
Kafka with Go Part 1 " Understanding Async Systems, Distributed Architecture, and Your First Kafka Publisher/Subscriber
6+ day, 11+ hour ago (451+ words) Modern web applications are no longer just simple static websites. Years ago, many websites were mostly: But modern software systems are very different. Today's applications do things like: Modern backend systems are now closer to continuously running software systems rather…...
Say goodbye to Zoo Keeper: A production-ready path to KRaft designed for service continuity at no added cost
6+ day, 23+ hour ago (703+ words) May 11, 2026 | By Phil Yang For over a decade, Apache Zoo Keeper was inseparable from Apache Kafka. It coordinated Kafka's distributed metadata'tracking brokers and topics, elected controllers and leaders, and managed cluster configuration. Every Kafka cluster was dependent on a Zoo…...