Data Engineering - hughevans.dev

hughevans.dev

Sign in Subscribe

Data Engineering

A collection of 3 posts

Getting Started with Iceberg Topics - A Beginner's Guide

Data Engineering

Getting Started with Iceberg Topics - A Beginner's Guide

Understand how Kafka integrates with Apache Iceberg™ and experiment locally with Docker and Spark The streaming data landscape is evolving rapidly, and one of the most exciting developments is the integration between Apache Kafka and Apache Iceberg. While Kafka excels at real-time data streaming, organizations often struggle with the complexity

Getting Started with Diskless Kafka - A Beginner's Guide

Data Engineering

Getting Started with Diskless Kafka - A Beginner's Guide

Diskless topics are proposed in KIP-1150, which is currently under community review. The examples in this article use "Inkless", Aiven's implementation of KIP-1150 that lets you run it in production. I joined Aiven as a Developer Advocate in May, shortly after the Kafka Improvement Proposal KIP-1150:

From Radio Waves to Kafka Topics - Building a Real-Time Aircraft Data Pipeline

Data Engineering

From Radio Waves to Kafka Topics - Building a Real-Time Aircraft Data Pipeline

If you want to showcase real-time data architectures you need a data source that's live, high-volume, varied, and messy enough to showcase real-world challenges. This is an issue I've run into several times over the last year whilst giving talks about real-time analytics using Kafka, Druid,