If you want to showcase real-time data architectures you need a data source that’s live, high-volume, varied, and messy enough to showcase real-world challenges. This is an issue I’ve run into several times over the last year whilst giving talks about real-time analytics using Kafka, Druid, ClickHouse, and Grafana in various combinations. You could use a data generator like ShadowTraffic but when trying to bring the sometimes dry topic of data engineering to life nothing beats real data. So when I’m building demos I’ve consistently turned to the same compelling dataset: ADS-B aircraft transmissions.

I was introduced to ADS-B (Automatic Dependent Surveillance–Broadcast) by my former colleague at Imply Hellmar Becker, and is one of the technologies aircraft use to relay data including their position, heading, and speed to air traffic controllers and to other aircraft. This creates a continuous stream of real-time data that’s publicly accessible and rich with analytical possibilities. The dataset perfectly illustrates the complexities of streaming analytics—it arrives at high velocity, contains mixed data types, requires deduplication and enrichment, and benefits from both real-time alerting and historical analysis.

What makes ADS-B particularly valuable for demonstrations is its combination of technical complexity and intuitive appeal. Everyone understands aircraft movement, making it easy to visualize concepts like windowing, aggregation, and anomaly detection. Yet underneath this accessibility lies genuine engineering challenges: handling bursty traffic patterns, dealing with incomplete or duplicate messages, and correlating position data with aircraft metadata.

In this article, I’ll walk through building a complete ADS-B ingestion pipeline—from setting up a simple antenna to producing clean, structured data to Kafka topics ready for real-time analysis. By the end, you’ll have both the technical foundation and a rich dataset to explore your own streaming analytics architectures.

Understanding ADS-B Data

ADS-B transmissions use a standardized message format called SBS (BaseStation format), which arrives as comma-separated text lines. Each message contains different types of aircraft information, for example:

Position Messages (MSG,3): Location, altitude, and identification data

MSG,3,1,1,40756A,1,2025/06/01,17:42:30.733,2025/06/01,17:42:30.776,,35000,,,40.1234,-74.5678,,,0,0,0,0

Velocity Messages (MSG,4): Speed, heading, and vertical rate

MSG,4,1,1,40756A,1,2025/06/01,17:42:31.233,2025/06/01,17:42:31.276,,,450,275,,,256,,,,,0

ADS-B data has a high data velocity with anywhere from 100 to 2000 messages a second produced by a receiver depending on location. There are some problems with ADS-B data that present a barrier to real time analytics with this data: the data contains duplicate messages because the same aircraft can be tracked by multiple receivers (as many as 20-30% of messages will be duplicates), the are missing fields because not all messages contain complete information, and traffic varies by time of day and geographic location.

This real-world messiness makes ADS-B data perfect for demonstrating streaming analytics challenges like de-duplication, windowing, and real-time aggregation.

Hardware Setup and Data Collection

You can be receiving live ADS-B data for around £95 (or less, if you already have some of these parts or can pick them up second hand) and have it running in 15 minutes. Here’s my exact setup that’s been reliably collecting ADS-B data for months.

Hardware shopping list

• Raspberry Pi 3 (£35) - or any similarly capable computer, you can pick Raspberry Pi 3s now for as little as £18 second hand
• NooElec NESDR Mini USB RTL-SDR (£35) - An easy to use and readily available ADS-B antenna that can be connected to your Pi via USB. The included antenna gets you 50-100km range, which is perfect for getting started.
• Micro SD Card (£10) - These get cheaper all the time, here I’ve linked to a 128GB card that’s relatively good value but really anything over 16GB will be fine as long as you can write to it properly, for more information on picking an appropriate SD card see this great article by Chris Dzombak.
• Micro USB Power Supply (£15) - make sure to pick a reliable power supply that consistently delivers 5V/2.5A, I’ve linked to the official power supply here but almost any half decent Micro USB power supply will do.

Setup

1) Install a supported OS on your Pi, I’m using a lite version (without a UI) of the official Debian Bookworm build, for details on how to do this follow the steps in the guide on the Raspberry Pi website.

2) Install Docker on your Pi and add your user to the docker group to run docker without sudo. Important: Log out and back in for group changes to take effect.

curl -sSL https://get.docker.com | sh
sudo usermod -aG docker pi
# Log out and back in for group changes to take effect

3) Create a new Docker compose called docker-compose.yml and define an ultrafeeder services as below. Note: this is a very basic ultrafeeder configuration, you may wish to consult the setup guide in the ADS-B Ultrafeeder repo for a more in depth guide to setting up this part.

services:
  ultrafeeder:
    image: ghcr.io/sdr-enthusiasts/docker-adsb-ultrafeeder
    container_name: ultrafeeder
    restart: unless-stopped
    device_cgroup_rules:
      - "c 189:* rwm"
    ports:
      - 8080:80     # Web interface
      - 30003:30003 # SBS output (for Kafka)
    environment:
      - READSB_DEVICE_TYPE=rtlsdr
      - READSB_RTLSDR_DEVICE=00000001  # Usually 00000001
      - READSB_LAT=51.4074             # Your antenna latitude
      - READSB_LON=-0.1278             # Your antenna longitude
      - READSB_ALT=52                  # The altitude of your antenna
    volumes:
      - /dev/bus/usb:/dev/bus/usb

4) Deploy your ultrafeeder services:

docker-compose up -d

5) Optional: Add FlightRadar24 Integration

Adding FR24 gives you two immediate benefits: a professional flight tracking interface and confirmation that your data quality meets commercial standards. Plus, contributing data gets you free access to FR24’s premium features. Register via the flight radar site to get your sharing key, you should then be able to find your key in your Account Settings under “My data sharing”.

Add the flight radar feed service to your docker compose to start sending data to FR24.

# Add to existing services
  fr24feed:
    image: ghcr.io/sdr-enthusiasts/docker-flightradar24:latest
    container_name: fr24feed
    restart: always
    ports:
      - 8754:8754
    environment:
      - BEASTHOST=ultrafeeder
      - FR24KEY={your flight radar 24 key}
    dns_search: . # prevents rare connection issues related to a bug in docker and fr24feed

Redeploy with:

docker-compose up -d

Once setup your station should appear on their coverage map within 10-15 minutes.

6) Validate ADS-B Data Reception

Test that you are receiving ADS-B data correctly:

nc localhost 30003

You should see continuous messages like:

MSG,8,1,1,40756A,1,2025/06/01,17:42:30.733,2025/06/01,17:42:30.776,,,,,,,,,,,,0
MSG,3,1,1,40756A,1,2025/06/01,17:42:33.003,2025/06/01,17:42:33.015,,35000,,,40.1234,-74.5678,,,0,0,0,0
MSG,4,1,1,40756A,1,2025/06/01,17:42:35.120,2025/06/01,17:42:35.156,,,450,275,,,256,,,,,0

If your antenna has a good view of the sky you can expect around 100-2000 messages/second (depending on your location) with CPU usage sitting comfortably under 20% on a Pi 3.

Quick Troubleshooting

No aircraft? Check your antenna USB connection:

lsusb | grep RTL

You should see something like:

Bus 001 Device 033: ID 0bda:2838 Realtek Semiconductor Corp. RTL2838 DVB-T

If not, your antenna may not be connected correctly. Verify your antenna connection is secure or try different USB port (preferably USB 2.0+) and try restarting ultrafeeder:

docker-compose down && docker-compose up -d

Tracking very few aircraft? Try placing your antenna higher and away from electronics, for best results try and get an unobstructed view of the sky.

Kafka Integration

Now that we have ADS-B data streaming on port 30003 let’s produce it to Kafka to allow us to work with it as an event stream. We’ll add Kafka to our Docker stack and build a producer that can handle thousands of aircraft updates per second.

Part 1 : If I never see another pizza again, it’ll be too soon.

The year is 2022, I am a know-nothing twenty-something cloud consultant. ChatGPT has just been launched and the company I work for has created a brand-new practice with the business to focus on AI, and a colleague approaches me to see if I would be interested in helping to host the AI meetup the company has just adopted.

Now three years later I’m still a know-nothing twenty-something, but now with a little more experience under my belt. I’ve organised 22 AI and Deep Learning for Enterprise events, been a committee member for three conferences, and had a hand in helping to host and organise several other tech focused events in the capital. Participation in these communities has led me to all kinds of interesting places like speaking at events, learning new skills, running AV, and doing video editing, it’s even got me a job or two. Community organising has also left me more burnt out than I’ve ever been in my life.

Organising and hosting community events (particularly solo as I have done on occasion) is exhausting work. If you are putting on an event once a month, which for a while is what we were doing at AIDLE, the work is essentially unending. As soon as we finish hosting one event we immediately move on to planning the next (if we hadn’t started already). Hosting can also be very physically demanding: moving furniture, running to grab stacks of beer and soft drinks, and lugging an AV kit on the tube on top of the late nights can leave you physically exhausted - particularly whilst also having to put on a friendly face and look after attendees and speakers.

Needless to say, I would not still be involved in organising events without a lot of support. Support from other organisers who’ve given advice, promoted and collaborated on events, from friends who have come along to help out running AV and taking pictures, and from sponsors and hosts who provide funding and venues. Co-organisers are perhaps the greatest source of support, as the difference in the feeling of doing something alone versus being part of a team cannot be overstated. Something I wish I had starting out was a way to find other organisers to collaborate with and share resources, I’ve met some great organisers over the last three years, but it took a long time.

Part 2: It was never about the Pizza (sorry)

After the last three years of being in technical communities both as an organiser and as a regular member, I’ve eaten a LOT of pizza. I used to really like Pizza, but now not so much, and I know I’m not alone in this. Not Another Pizza is a fun little name for a group for community organisers, but (and you’ll have to forgive me for this) the pizza isn’t pizza, it’s an analogy.

Pizza is a stand in for all the bare minimum requirements of hosting an event: the food yes, but also venues, AV, community event platforms (on which I have thoughts), and so on. Of course community organisers need these things, but often as an organiser you’ll find yourself reaching for something more.

As an organiser I’ve benefited greatly from meeting collaborators, being given advice on tracking community growth and retention, tools to help automate away the drudgery of managing guest lists, resources on managing codes of conduct and code of conduct violations, and maybe a sympathetic ear from someone from time to time. Think soul food, not cold pizza.

Some of the biggest challenges I ran into early on in organising were burnout, managing relationships with sponsors, and finding speakers.

I helped beat my organiser burn out by both cutting down on the frequency of events I was organising and attending, but also by avoiding solo efforts where possible. It’s not about shifting the work on to other people but about working together as part of a team - helping other people tackle their obstacles as much as they help you tackle yours. For Developer advocates this may be a familiar feeling, being the glue between different people helps make you a very useful colleague but also means people are more willing to help you on when you need help.

Finding speakers was made easier by emulating the best practices of other groups and organisers, for example: opening a call for papers so that speakers could apply to speak at AIDLE rather than having to constantly go out chasing them. This was a lesson I only learned because I was attending other peoples events, asking for advice, and shifting my perspective by becoming a speaker myself.

So, the vision for Not Another Pizza is a support network for dev advocates, community managers, and event organisers that can help provide some of these things. There are some great communities and resources out there already doing this like the Developer Marketing Alliance, but I’ve found that resources and collaboration opportunities can be very dependent on geography, so I wanted to set something up more focused on organising technical communities in London and the UK in general.

The London community landscape is highly concentrated around the City, with many smaller communities relying on access to free or subsidised event spaces rather than paying through the nose for renting event spaces. It feels like we are also rapidly approaching “peak meetup” with several events on the same topic on any given night, with 4 different AI and data meetups on the same night as the last AIDLE meetup alone. I don’t see the large number of organisers in London as a problem though - we as organisers don’t need to compete for limited resources and venues if instead organisers and groups collaborate to pool resources, shouldering the work of organising events together. London then is the perfect environment for building this kind of meta-community of community organisers.

My hope is that Not Another Pizza can foster genuine connections between organisers and serve to facilitate meaningful collaboration beyond being just another networking event.

Part 3: Something better than another pizza

It’s very early days yet, but some friends and I are working on building out our community by inviting other dev rel professionals and organisers to the Discord community and building out the website. We’ve also been building out a catalogue of projects on the site that members of the group have been working on, including tools made by community members designed to help make managing communities easier.

One project we’ve recently started working on is bridge, an API written in Python which will eventually enable the publishing and updating of events across multiple platforms simultaneously. I’ve previously spoken about how I think applying the POSSE (Post Own Site Syndicate Everywhere) content management strategy to community events could potentially help tackle the degradation of platforms like Meetup, and I hope bridge will help enable more organisers to experiment with that.

Going forward I’d love for us to create spaces for authentic collaboration by enabling organisers to share success stories of collaborations born through Not Another Pizza, providing match-making for mentorship opportunities, hosting a library of useful resources and content, and to become a really great place to get help with community organising. Perhaps in future we could even have some in person events, who knows. For now our focus will be on growing the community and continuing development on projects like bridge. I think we’ll know if this community has been successful if we can see concrete examples of members meeting collaborators through Not Another Pizza and going on to run events.

If Not Another Pizza sounds like a useful resource you can join the discord community over at discord.notanother.pizza to meet other organisers who can help your community thrive. Don’t worry if you aren’t based in London either, there are members in the group all over the world and some community organiser experiences really do feel universal at times, like wanting nothing less than another slice of cold pizza.

Last week at TurinTech we had a workshop on agentic AI with an unusual challenge: complete a work task without writing any code manually; the goal was to see if we could boost our productivity using these tools. I’m fairly skeptical of using agentic tools at work - my concern was that I would spend as much (or even more) time vetting AI generated work than I would have doing them manually - even without considering that, could I really be productive with my hands tied behind my back?. I decided to try speedrunning a common task in my day to day work to see if I could really improve my productivity.

The Speedrun Setup

Cursor%

• Use Cursor exclusively to add an optimization example to our team’s GitHub repo

• Add a fully documented example to our optimization repo using only AI-generated code

• No manual typing of code allowed, only prompting and editing AI suggestions

Any%

• Add a fully documented example to our optimization repo manually using my normal approach

I specifically chose the task of adding an example to this repo as it is both a task I have to do frequently and a task I thought it would be simple for an agentic tool to complete as it just involves copying data from one place to another and reformatting it as a basic report. It was my hope that the simplicity of the sample task would help avoid an automation rabbit hole and give the agent the best possible chance to be genuinely useful.

Usually once I complete an optmisation or someone shares a PR with an optimisation with I manually copy the template directory, write a short summary of the project, copy the data across, and where appropriate add a demo which you can use to show the difference in peformance before and after the optimisation. For my speedrun I would see if I could complete the same task as a tool assisted speedrun using only Cursor without interacting with anything other than the terminal and chat.

The specific example I was documenting in both cases was a DataStax Langflow Optimisation by Jordan Frazier (see the PR here) who used our tool Artemis to improve the performance of the Langflow Python library.

My friend Leo built this cool web app for annotating his laptop stickers to record where he got them. This tickled the sticker-goblin and developer advocate parts of my brain, so I nagged Leo until he shared the source code with me. He did so graciously on the condition I contribute to it.

Try clicking on the stickers below to see where I got them.

Go fullscreen here.

You can find the source code for this project in the GitHub repo (hassling Leo optional). You can also try creating your own sticker gallery here on my site if you don’t fancy getting stuck into the code just yet.

In exchange for this delicious code I added a couple of features: a simple button to open the annotation generate tab from the main page and an option to replace your image with a new one when you get more stickers.

I used Claude pretty heavily because I don’t pretend for one minute to know about front-end development (I’m a data guy after all) so had to check the changes Claude suggested pretty closely and do some tweaking to get the new code to match the style of Leo’s code.

I also worked on adding some instructions to the README to help people create their own sticker galleries and to contribute to the project.

My next challenge was to create my own sticker gallery and upload it here. This was fairly simple but required some minor changes to the stickers project code and to my Jekyll configuration.

I updated the next.config.ts file with the below config to change the export config to include the relative path to the location on my site I would be putting the export static code and to configure the static site to be exported to the out directory with output: 'export'.

const nextConfig: NextConfig = {
    output: 'export',
    trailingSlash: true,
    images: {
      unoptimized: true,
    },
    assetPrefix: '/projects/stickers',
    // Add basePath for deployment in a subdirectory
    basePath: '/projects/stickers', // Adjust this to where you'll host it in your Jekyll site
};

Before building, I used the project to build my own sticker gallery and took the exported annotation.json file and replaced the default one in the stickers project.

Annotating stickers

I then built the project in the usual way with npm run build. I copied the contents of the out file into my Jekyll project and updated the _config.yml file with the below so that my Jekyll build would pick up the Next.js project files properly.

include:
  - _next
  - "**/_next/**"

Finally, I could add my stickers gallery to this post with an iframe like so:

<iframe src="/projects/stickers/index.html" width="100%" height="500px"></iframe>

Building my sticker gallery was super quick and fun thanks to this awesome tool from Leo. It was fun to contribute features and to help make the project more accessible for others to contribute too. I hope this blog inspires you to try making your own sticker gallery and to maybe get involved in trying your hand at contributing to this project.

This project is one of the first from notanother.pizza a community of community organisers and developer advocacy folks mostly based in London, you can check out our other projects here.

On Saturday I had the pleasure of speaking at the codebar Festival Fringe at Newspeak House about my thoughts on the decline of Meetup and approaches for tackling platform decline in the community organising space and elsewhere.

The community came together to raise hundreds for codebar and it was a privilege to get to speak alongside such fantastic speakers with lightning talks from Willow S. on obsolesce proofing your smart home, Kourosh Simpkins on watching the implications of watching (deep-faked) politicians play Minecraft, and Keegan R. with what has to be the single most entertaining talk on box packing ever (I’m still a little in awe).

Thank you to Leo Riviera and Melissa Tranfield for hosting and organising, I’m sure you did Kimberley Cook proud!

Writing documentation is an oft maligned part of the software development process. It takes time away from solving technical problems and can feel like an unending task., new changes and versions can make your newly updated quick-start or API reference rapidly out of date. It comes as no surprise then that there are now many tools offering to take the pain out of this process by automating some or all of the work required to produce documentation with AI.

I’ve seen the importance of good docs first hand. Good documentation helps teams to communicate a vision for a project both internally and with external stakeholders which in turn can lead to dodging costly misunderstandings, and can make the actual source code of the project better by keeping the whole team aligned on what they are building. In contrast bad documentation could see you wasting hours providing support to get new devs onboarded to a project. Worse still, in the case of competing products, it could see people opt to use different products entirely, even if they are less feature rich, simply because it’s easier to understand how to use them.

Given the importance of good documentation, is it actually possible to use AI tools to make the process of producing documentation easier without degrading the overall quality? In this blog I will explore using Swimm, an AI documentation tool, to document one of my personal projects to try to answer this question.

A photo of me in front of the Golden Gate Bridge during a visit to San Francisco for Apache Druid Summit

I made a lot of things this year including mistakes - I started the year out of a job having been let go from a Senior Platform Engineer Role because I’d pursued a Senior role before I was ready. I’m now in a better place with work but it’s been a challenging year and one thing that’s helped me stay grounded and keep moving forward is making.

Life for me has always been divided into time spent working on one project or another, a cardboard city encompassed much of my childhood bedroom, and as a teen I spent the summer tinkering with 3D printers and Raspberry Pis. The process of coming up with an idea for a project, building, and iterating until I have an end product that I’m (mostly) happy with never fails to hold my attention and recharge my creative batteries.

In this blog I’d like to revisit some of the projects I worked on this year and share some of what I learned from each one.

Tool Storage from Reused Grocery Crates

This was a super scrappy project: I’ve been running a tool library out of my garage for the past couple of years and was rapidly running out of space for donated tools so I decided to cobble something together from some grocery crates I picked up on Olio and some parts of an old fitted kitchen.

I’m still using these drawers to store things so they work but they needed a bit of tweaking with shims as the drawers initially kept falling through the runners and getting stuck - should have triple checked my measurements before taking a circular saw to my offcuts of kitchen counter!

ASCII Art Photo Booth for EMF Camp

This year for EMF camp I built an exhibit that would take your picture and print it out as ASCII art on receipt paper. This has been done lots of times before but I fancied trying my hand at it because I thought it would be a fun project.

I got this working at home but once I actually got to EMF I couldn’t figure out how to light the subjects of the photos in the environment properly to get nice ASCII images which was a shame - next time around better lightning, a backdrop, and potentially a more flexible algorithm for generating ASCII art would make for better results.

Anne Fehres and Luke Conroy & AI4Media / Better Images of AI / Hidden Labour of Internet Browsing / Licensed by CC-BY 4.0

Post Own Site Syndicate Everywhere

POSSE (Post [on your] Own Site Syndicate Everywhere) is a simple concept: when you create digital content like articles, guides, or videos simply upload it to your own website before posting links back to that content on third party sites like Medium or YouTube. One key advantage of this approach is it provides a degree of indepdence from third party platforms as all your content is preserved on your site so the loss of a social media account doesn’t mean losing years of work.

I first learned about POSSE in the excellent article of the same title by Molly White and have since adopted the approach for my own work.

An issue often sited with POSSE is that posting content across multiple channels can be labour intensive either as a result of the work required in manually posting via several third party platforms or maintaining the tooling required to automate the process. There are some ongoing efforts to help simplify the process of cross posting across multiple channels including Ryan Barrett’s Bridgy Fed project which serves as a bridge between decentralized social networks.

I wanted to build POSSE into my existing CICD for my site which I have configured as GitHub Actions - I decided to (perhaps foolishly) write a few quick scripts to post links to new articles on my Website to my various social media accounts which proved tricker than I expected! In this article I’ll talk you through my attempt at automating the common POSSE task of reposting links with a hope that elements of it may be useful to your own projects. All my code is available on my GitHub.

Posting on your own site…

I’m using Jekyll a Ruby based static website generator along with the awesome Contrast theme by Niklas Buschmann for my personal site. I use GitHub Actions workflows to to build the static HTML from my Jekyll project and then copy it across to an Nginx server running on a Raspberry Pi in my Homelab.

A screenshot of a successfully completed github action for building and deploying a jekyll site

I like being able to write my posts in both HTML and Markdown because of the flexibility it provides so this approach works well for me - it also makes it easy to edit content already published to my site. I can also test what articles will look like by running Jekyll locally which is really helpful for catching mistakes prior to publishing.

…and syndicate everywhere!

Once I’ve written a new article I want to link to it from all the other places I have a presence on the internet like Bluesky, LinkedIn, and Mastodon. Frustratingly at time of writing LinkedIn doesn’t appear to provide any methods for posting to your own LinkedIn account with their API - instead only allowing you to post to a company page so for now I’m limited to (still two great options!) Bluesky and Mastodon.

Getting new posts from a PR

Before I can post any links to articles I need to get new articles from PRs merged to my repo. All the articles on my site are stored in a directory called _posts so it’s easy enough to get a list of articles by running the handy changed-files action against that directory.

- name: Get changed files
    id: changed-files
    uses: tj-actions/[email protected]
    with:
    files: |
        _posts/**

Once I have a list of all the new articles I can easily iterate over them with a Bash loop. At the moment though this list is of file paths rather than links to the actual articles but this is easy enough to fix by extracting the name of the articles from the file path and appending it to the site url.

- name: Post all new posts
    if: steps.changed-files.outputs.any_changed == 'true'
    env:
    ALL_NEW_POSTS: $
    run: |
    
    for file in ${ALL_NEW_POSTS}; do
        echo $file
        # Get new post URLs from diff
        blog_url="https://hughevans.dev/${file:18:-3}"

Now that I can get a url for each new article it’s relatively simple to just POST that url in the text field of a post to the social media platform of my choice.

Mastodon

Posting a link to Mastodon and getting a nice embedded card is really easy as Mastodon pulls out all the OpenGraph information and renders it automatically for you into a nice preview card.

All you need to POST via the Mastodon API is your user token which can be found under Preferences > Development (see the Mastodon docs for more information).

With my list of new articles I can easily use the below POST request via curl to create a post on Mastodon with a nice preview card for my articles.

# Post to Mastodon
curl -X POST -d "{\"status\":\"$blog_url\", \
\"media_ids\":null,\"poll\":null}" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $" \
"https://hachyderm.io/api/v1/statuses"

Ta-dah!

A post on mastodon with a link to a blog about me speaking at the Barcelona Aerospike Meetup

Bluesky

I found posting an embed card to Bluesky much trickier than posting via the Mastodon API (as anyone unfortunate enough to be following me on Bluesky whilst I was writing this article probably noticed!)

I found these examples shared by Felicitas Pojtinger really helpful but my main stumbling block was that I needed to manually include the OpenGraph information as fields in the body of my post requests as Bluesky won’t detect this automatically yet.

Before posting via the API you need to create a new app password under Settings > Privacy and Security > App Passwords. I learned that posting to Bluesky isn’t as simple as just making a POST request with my app password, as a result of running on the decentralized AT Protocol, posting via the API required that I find the Decentralized Identifier (or DID) for my handle and with that get a session API key - I did both of these with the pair of curl commands in the Bash snippet below.

# Post to Bluesky
export APP_PASSWORD='$'

HANDLE='hevansdev.bsky.social'
DID_URL="https://bsky.social/xrpc/com.atproto.identity.resolveHandle"
export DID=$(curl -G \
    --data-urlencode "handle=$HANDLE" \
    "$DID_URL" | jq -r .did)

# Get API key with the app password
API_KEY_URL='https://bsky.social/xrpc/com.atproto.server.createSession'
POST_DATA="{ \"identifier\": \"${DID}\", \"password\": \"${APP_PASSWORD}\" }"
export API_KEY=$(curl -X POST \
    -H 'Content-Type: application/json' \
    -d "$POST_DATA" \
    "$API_KEY_URL" | jq -r .accessJwt)

Once I had an API key posting text to Bluesky was simple, however: I wanted to create a preview card with an embedded image and an article title. I couldn’t find any neat way to do this directly in the Bluesky API - as a work around I used the snippet below to pull the Open Graph data from the blog post automatically so I can pass it in the Bluesky POST body.

# Get page og image
og_img_url=$(curl -L $blog_url | grep 'og.image' | grep -oE "(http|https)://[a-zA-Z0-9./?=_%:-]*")
curl -O $og_img_url

# Get page title
page_title=$(curl $blog_url -so - | grep -o "<title>[^<]*" | tail -c+8)

So with all that done I should be ready to POST the link right? No such luck. The embed image first needs to be uploaded as a blob and the blob link and size recorded for use in the POST.

```

Upload embed image blob

blob=$(curl -X POST
-H “Authorization: Bearer ${API_KEY}”
-H ‘Content-Type: image/jpeg’
–data-binary @$(ls *.jpg)
“https://bsky.social/xrpc/com.atproto.repo.uploadBlob”)

My last week at Imply was at Druid Summit, it was an absolute pleasure working alongside my former colleagues Reena Leone, Peter Marshall, and Dave Klein running the event and getting to see so many excellent presentations from members of the Druid community. I organized all the panels for the 2024 event and you can see the recording of the Ops and Optimization panel I hosted below.

I had an amazing time speaking at the Aerospike Barcelona Data Management Community Meetup this week about working with flight radar data in Apache Druid. The team at Criteo were amazing hosts, super welcoming and friendly and the audience were really engaged with great questions after talks wrapped up. I’m looking forward to speaking at another Aerospike event later this year in Copenhagen.

If you’d like to check out my talk you can watch the recording below.

You’re making a change to your OpenSearch managed service and it’s all going great - right up until you make a mistake, destroying your cluster and causing you to lose all your indices. If only you had a snapshot you could restore your cluster from? Too bad you didn’t create any. 

Taking OpenSearch snapshots is relatively easy but may require making some configuration changes to your IAM roles. It’s definitely worth doing because once you’ve successfully taken a snapshot you can use it to restore the indices in deleted, destroyed, or corrupted OpenSearch clusters or even create a duplicate cluster with the same data.

Prerequisites

In order to manually take snapshots you’ll need admin access to your OpenSearch service API either via curl or OpenSearch devtools, in this guide I’ll be using the latter method.

Before taking a snapshot you will need to create a role that will allow your OpenSearch service to write the snapshot to an S3 bucket and grant permission to the OpenSearch service to use that role. The Terraform for your IAM config should look something like the below, for more details see the AWS documentation.

IAM role

resource aws_iam_role" "es_snapshot" {
  name                 = "es-snapshot"
  managed_policy_arns  = [aws_iam_policy.es_snapshot.arn]
  assume_role_policy   = <<EOF
{
"Version" : "2012-10-17",
"Statement" : [{
    "Sid" : "",
    "Effect" : "Allow",
    "Principal" : {
    "Service" : "es.amazonaws.com"
    },
    "Condition" : {
    "StringEquals" : {
        "aws:SourceAccount" : "<your aws account id>"
    },
    "ArnLike" : {
        "aws:SourceArn" : "<the arn for your opensearch cluster>"
    }
    },
    "Action" : "sts:AssumeRole"
}]

}
  EOF
}

Note the condition in the above terraform statement: this limits access to this role to a specific OpenSearch service with your AWS account, without it any OpenSearch service could assume this role.

IAM policy

resource "aws_iam_policy" "es_snapshot" {
  name = "es-snapshot-policy"
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [{
      "Action" : [
        "s3:ListBucket"
      ],
      "Effect" : "Allow",
      "Resource" : [
        "<arn of the s3 bucket you want to store your snapshots in>"
      ]
      },
      {
        "Action" : [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ],
        "Effect" : "Allow",
        "Resource" : [
          "<arn of the s3 bucket you want to store your snapshots in>/*"
        ]
      }
    ]
  })
}

Register a snapshot repository

In order to take a snapshot you first need to configure a snapshot repository to store your snapshots. In this guide I’ll be covering how to do this using an S3 bucket

First, if there isn’t one already you will need to register a snapshot repository, you can use the get request below to list any existing repositories (do not use cs-automated-enc, it is reserved by OpenSearch for automated snapshots).

GET _cat/repositories

If needed, register a new snapshot repository like so (note the use of the role we created in the previous section).

PUT _snapshot/opensearch-snapshots
{
  "type": "s3",
  "settings": {
    "bucket": "<your s3 bucket name>",
    "region": "eu-west-1",
    "role_arn": "<arn of your snapshot role>",
    "server_side_encryption": true
  }
}

Manually taking a snapshot

Check for any ongoing snapshots, you cannot take a snapshot if one is already in progress and OpenSearch automatically takes snapshots periodically.

GET _snapshot/_status

Take a snapshot. Adding the data to the end of the snapshot name is optional, but I’d recommend adding the correct time here so you can easily find the snapshot if you need to restore from it later.

PUT _snapshot/opensearch-snapshots/snapshot-2023-03-13-1135

Check snapshot progress with the first get request below and then view it with the second once complete. Use of the “pretty” query is not required but helps to make the output more readable.

GET _snapshot/_status
GET _snapshot/opensearch-snapshots/_all?pretty

You should see your snapshot listed alongside any pre-existing snapshots. Congratulations, you’re now ready to restore from a snapshot should you ever need to. Don’t stop here though, I recommend that you continue with the next section to familiarise yourself with the process of restoring from a snapshot - you should also take snapshots regularly to help reduce the risk of data loss.

Restoring from a snapshot

 Keycloak is an open source Identity and Access Management tool with features such as Single-Sign-On (SSO), Identity Brokering and Social Login, User Federation, Client Adapters, an Admin Console, and an Account Management Console.

Why use Keycloak?

There are several factors to deciding whether or not to use Keycloak or a SaaS IAM Service like AWS SSO. SaaS IAM services are typically easier to implement, better supported, and do not require manual deployment but Keycloak is free to use, feature rich, and flexible.

Pre-requisites

This guide assumes you already have at least one Keycloak instance with a Postgres database configured, if this is the case your keycloak.conf should include a section that looks something like the example below.

db=postgres
db-password=<your db password>
db-userame=keycloak
db-pool-initial-size=1
db-pool-max-size=10
db-schema=public
db-url-database=keycloak
db-url-host=<url of your db>
db-url-port=5432

If you do not yet have your database configured please refer to the documentation on configuring relational databases for Keycloak.

Configuring JDBC Ping

In order for Keycloak instances to cluster they must discover each other and this can be achieved by using JDBC Ping which allows nodes to discover each other via your existing database. JDBC Ping is a convenient discovery method because it does not require the creation of additional AWS resources and is compatible with AWS unlike the default discovery method (multicast) which is not permitted by AWS.

In order to use JDBC Ping we first need to define a transport stack, this can be achieved by adding the below element to the infinispan tag in your cache-ispn.xml file and replacing the default values (these should match the db-password and db-url-host from your keycloak.conf file).

<jgroups>
    <stack name="jdbc-ping-tcp" extends="tcp">
        <JDBC_PING connection_driver="org.postgresql.Driver"
                    connection_username="keycloak"
                    connection_password="<your database password>"
                    connection_url="jdbc:postgresql://<url of your database>:5432/keycloak"
                    initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
                    info_writer_sleep_time="500"
                    remove_all_data_on_view_change="true"
                    stack.combine="REPLACE"
                    stack.position="MPING" />
    </stack>
</jgroups>

We have now defined a new JGroups stack which will create a table in your database if one doesn’t already exist which Keycloak instances can use to discover each other, when you start a new Keycloak instance it will write its name as a new record into this table. To use this stack simply amend the transport element as shown below to reference the newly defined stack.

<transport lock-timeout="60000" stack="jdbc-ping-tcp"/>

Configuring Security Groups

Keycloak uses Infinispan to cache data both locally to the Keycloak instance and for remote caches. Infinispan by default uses port 7800 so we need to configure the Security Group our Keycloak instances are deployed to in order to permit both ingress and egress via port 7800. This can be done in a number of ways such as via the AWS Console, below is an example of configuring ports for Keycloak using Terraform.

## keycloak cluster egress
resource "aws_security_group_rule" "keycloak_cluster_egress_to_keycloak" {
    description              = "keycloak cluster"
    from_port                = 7800
    protocol                 = "tcp"
    security_group_id        = aws_security_group.keycloak.id
    source_security_group_id = aws_security_group.keycloak.id
    to_port                  = 7800
    type                     = "egress"
}

## keycloak cluster ingress
resource "aws_security_group_rule" "keycloak_cluster_ingress_to_keycloak" {
    description              = "keycloak cluster"
    from_port                = 7800
    protocol                 = "tcp"
    security_group_id        = aws_security_group.keycloak.id
    source_security_group_id = aws_security_group.keycloak.id
    to_port                  = 7800
    type                     = "ingress"
}

Restarting Keycloak

Keycloak does not automatically apply changes made to its configuration so you will need to restart your Keycloak instance/instances for clustering to work. First run the following from the terminal to rebuild your Keycloak instance to register the changes we made to your configuration.

➜/bin/kc.sh build

Once you have rebuilt Keycloak restart your Keycloak service by running the following (alternatively you can restart your Keycloak instance).

systemctl restart keycloak

Your Keycloak instances should now be running in a clustered state.

Testing your Keycloak cluster

To check that your Keycloak cluster is functioning correctly check your database and see if the JGROUPSPING table both exists and includes the name of all instances currently in the cluster, your table should look something like the below.

If you terminate a Keycloak instance or start a new instance you should see the records in this table change.

Troubleshooting

Changes made to config files aren’t applied after building Keycloak

Ensure that the config files you have changed match those configured in keycloak.conf, this guide for example assumes that you have your Infinispan config file set as cache-ispn.xml in your keycloak.conf file.

cache-config-file: cache-ispn.xml

Keycloak services don’t start after changing config files

Check the Keycloak logs and ensure your database access details (password and host url) are set correctly: if these values are incorrect the Keycloak service will fail to start.

Resources

Use of JDBC_PING with Keycloak 17 (Quarkus distro)

Embedding Infinispan caches in Java applications

Keycloak Server caching

Clustered Keycloak SSO Deployment in AWS was originally published on the Daemon Insights blog

Cheap and easy IoT with AWS

3D render of DALL-E-2 making art in an open office on a red brick background, digital art

What is DALL·E 2?

You’ve done the hard part and added instrumentation to your application to gather metrics, now you just need to expose those metrics to Prometheus so you can alert on them and monitor them, easy right?