12+ hour, 7+ min ago  (1245+ words) What I thought was a scheduling problem turned out to be a portability problem first In my last article, I mentioned that scheduling is the next wall I'll be walking toward. So I guess, here I am, walking towards it…...

13+ hour, 38+ min ago  (349+ words) Like many teams we didn't choose event-driven architecture in the first place. At first, the application was relatively simple. A user performed an action, the backend processed it, updated the database, and returned a response. But as the product evolved…...

1+ day, 11+ hour ago  (540+ words) To follow along, you will need: Before writing any code, you need a broker to talk to. Running Kafka by hand involves Zookeeper, broker configuration, and a fair amount of setup, so you will use Docker Compose to bring up…...

1+ day, 15+ hour ago  (886+ words) TL; DR Kafka handles event streaming: producers write to topics, consumers read. JSON. .. Tagged with dataengineering, apachekafkaschema, karapace, protobuf....

2+ day, 5+ hour ago  (281+ words) Modern full-stack applications are frequently tasked with writing transactional data to an operational database while simultaneously streaming event telemetry to downstream business intelligence (BI) engines like Power BI. Directly executing these cloud network requests within the synchronous execution path of…...

2+ day, 5+ hour ago  (555+ words) Quando iniciamos um processo Node. js, o runtime carrega o arquivo de entrada da aplica'o e executa todo o c'digo s'ncrono dispon'vel na Call Stack. Somente ap's essa etapa o Event Loop passa a assumir o controle do fluxo da…...

3+ day, 5+ hour ago  (283+ words) Understand the core layers, common patterns like ELT and Medallion, and best practices for building robust data pipelines. Data pipeline architecture is the blueprint detailing how data is collected, processed, stored, and delivered. It's not the pipeline itself, but the…...

3+ day, 3+ hour ago  (174+ words) Scan your Kafka producers, consumers, and broker configurations for performance issues, data loss risks, and operational problems. Get auto-fix PRs. Saincheisteanna coitianta a ghlacann "r AI go huathoibr'och Consuming messages without committing offsets causes the consumer group to reprocess all…...

3+ day, 8+ hour ago  (345+ words) Managing high-volume message traffic in distributed architectures is crucial. Efficient use of database and CPU resources is also very important. There are structures that allow us to receive messages in batches. The default Spring Kafka "Batch Message Listener" structure addresses…...

3+ day, 20+ hour ago  (797+ words) Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I…...