Data Engineering - hughevans.dev

hughevans.dev

Sign in Subscribe

Data Engineering

A collection of 11 posts

Get Kafka-Nated (Episode 8) Realistic Synthetic Streaming Data with Michael Drogalis

Data Engineering

Get Kafka-Nated (Episode 8) Realistic Synthetic Streaming Data with Michael Drogalis

Check out the latest episode of Get Kafka-Nated! It's always a treat chatting with Michael Drogalis and I had a great time picking his brain about his career in streaming and his Synthetic Stream Data product ShadowTraffic. You can watch the recording below. Original release You can find

Get Kafka-Nated (Episode 7) Redpanda vs Kafka with Tristan Stevens

Data Engineering

Get Kafka-Nated (Episode 7) Redpanda vs Kafka with Tristan Stevens

Check out the latest episode of Get Kafka-Nated! I get a great conversation with Tristan about the merits of Kafka vs Kafka Compatible solutions. You can watch the recording below. Original release You can find all the past episodes of Get Kafka-Nated as well as Kafka news and technical deep

Getting Started with Iceberg Topics - A Beginner's Guide

Data Engineering

Getting Started with Iceberg Topics - A Beginner's Guide

Understand how Kafka integrates with Apache Iceberg™ and experiment locally with Docker and Spark The streaming data landscape is evolving rapidly, and one of the most exciting developments is the integration between Apache Kafka and Apache Iceberg. While Kafka excels at real-time data streaming, organizations often struggle with the complexity

Get Kafka-Nated (Episode 6) Kafka Meets Flink with Ben Gamble

Data Engineering

Get Kafka-Nated (Episode 6) Kafka Meets Flink with Ben Gamble

Check out the latest episode of Get Kafka-Nated! This one was a real treat - I had a great time chatting with my friend and field CTO at Ververica Ben about all things Flink and Kafka. I think we could probably have talked all day about this but you can

Get Kafka-Nated (Episode 5) Uber Staff Engineer and Kafka Tiered Storage Contributor Satish Duggana

Data Engineering

Get Kafka-Nated (Episode 5) Uber Staff Engineer and Kafka Tiered Storage Contributor Satish Duggana

Check out the latest episode of Get Kafka-Nated! I had a fantastic conversation with Satish D from Uber, who was a key contributor to KIP 405 that brought tiered storage to Kafka. We explored what tiered storage actually is, the specific storage challenges at Uber that led to KIP 405,

Get Kafka-Nated (Episode 4) Alex Merced Co Author of the Iceberg Definitive Guide

Data Engineering

Get Kafka-Nated (Episode 4) Alex Merced Co Author of the Iceberg Definitive Guide

Check out the latest episode of Get Kafka-Nated! I had a fantastic conversation with Alex Merced, Head of DevRel at Dremio and co-author of "Apache Iceberg: The Definitive Guide," about why Iceberg is becoming essential for streaming data architectures. We explored how Apache Iceberg is blurring the lines

Get Kafka-Nated (Episode 3) Greg Harris on Contributing to KIP 1150 diskless

Data Engineering

Get Kafka-Nated (Episode 3) Greg Harris on Contributing to KIP 1150 diskless

Check out the latest episode of Get Kafka-Nated! I had a fantastic conversation with Greg Harris about his work on KIP-1150 and life as an open source software engineer. We dove deep into the technical architecture, explored the challenges of implementing this game-changing feature, and discussed what Diskless topics mean

Get Kafka-Nated (Episode 2) Life of a Kafka contributor with Josep Prat

Data Engineering

Get Kafka-Nated (Episode 2) Life of a Kafka contributor with Josep Prat

Check out the second episode of Get Kafka - Nated! I had a great time chatting with Josep Prat about his experiences as a Kafka Contributor and Apache Kafka PMC member. We covered everything from first open source contributions to how to properly manage a major open source project. Tune

Get Kafka-Nated Podcast (Episode 1) Apache Kafka®'s Evolution - 14 Yrs of Streaming

Data Engineering

Get Kafka-Nated Podcast (Episode 1) Apache Kafka®'s Evolution - 14 Yrs of Streaming

Check out the first episode of Get Kafka - Nated! Filip Yonov and I had a great chat exploring everything from Kafka's journey from on-prem to the cloud, this years major Kafka improvement proposals, to what we're excited about for the future of Kafka. Tune in

Getting Started with Diskless Kafka - A Beginner's Guide

Data Engineering

Getting Started with Diskless Kafka - A Beginner's Guide

Diskless topics are proposed in KIP-1150, which is currently under community review. The examples in this article use "Inkless", Aiven's implementation of KIP-1150 that lets you run it in production. I joined Aiven as a Developer Advocate in May, shortly after the Kafka Improvement Proposal KIP-1150:

From Radio Waves to Kafka Topics - Building a Real-Time Aircraft Data Pipeline

Data Engineering

From Radio Waves to Kafka Topics - Building a Real-Time Aircraft Data Pipeline

If you want to showcase real-time data architectures you need a data source that's live, high-volume, varied, and messy enough to showcase real-world challenges. This is an issue I've run into several times over the last year whilst giving talks about real-time analytics using Kafka, Druid,