Introduction

Cloud networking can feel like an alphabet soup with all the fancy words and acronyms—VPC, NATs, firewalls, subnets—oh my!

Fear not, today we’re going to demystify the basics of cloud networking by explaining it as if it’s your own home network! I’ve found myself using this approach to teach AWS networking many times, and the relief I hear when it “clicks” for someone is always a joy. This post is geared towards beginners, so if you’re someone that already know what a route table is and how to manage PrivateLink connections, please subscribe for future content and save yourself the reading.

Although I’m focusing on AWS, the concepts are essentially the same in Azure or GCP, though names of services may vary.

What is a VPC?

Let’s start with the Virtual Private Cloud, or VPC if you like acronyms. My Azure pals will know this as a Virtual Network, or VNet, and I have to give credit to Microsoft for the better name here. Why?

Because the VPC is just a network in the cloud. But forget about the cloud for a sec, what is a network? You’re on one right now—the one you used to access this web page and read this article. Most of us have at least one network at home, with WiFi.

WiFi is just the way that you connect wirelessly to the network though, it isn’t the network itself. Alternatively you could be on your home network by physically connecting via Ethernet cable, and some other ways we won’t get into.

VPC IP Addresses

Networks have a range of IP addresses, aka a CIDR block, such as 192.168.1.0/24. When a device connects to the network, it gets an IP address assigned, such as 192.168.1.30. Every network is totally isolated, so their IP addresses are considered “private", also known as a Local Area Network (LAN).

In fact, two different networks could have the same IP address ranges, which is fine, because they don’t communicate with each other. For example, most consumer routers you buy and probably have at home right now are going to use the very same CIDR I shared above, 192.168.1.0/24, as this is just a very common private IP range.

So think of a VPC as another network, like the one you have at home. Or to make it more fun, flip it: think of your home network as the VPC going forward.

Subnets

Next is subjects. If you have an IT background or learned networking before cloud computing was all the rage, you’d know these as VLANs.

Subnets simply divide your VPC—or network—into smaller chunks. Subnets have two primary uses:

1. Organize parts of the network for logical purposes

2. Distinguish with parts of the network are public vs. private

Again, all of this can be done at home as well, using VLANs you would typically configure in your home gateway or router.

Logical Separation

Organizing chunks of the network for different logical purposes is useful for maintainability, security, and scalability.

For example, say you start using a ton of IoT devices like smart bulbs and plugs at home. IoT devices are notorious for having more security risk, so we’d ideally want to segregate those to their own subnet/VLAN so they can’t reach our important trusted devices like mobile phones, laptops, desktops.

Each of these devices connects to the network, so they each get an IP address. So we’d need to size their subnet accordingly, making sure it has enough available IP addresses to give to each IoT device and allow room for future expansion. Still, other subnets may only require a few IP addresses. Sizing subnets is important because it can be difficult to change afterwards.

Public vs. Private Subnets

The other use for subnets I mentioned was private vs. public. By default, our subnets can’t access the Internet—we’ll get to that next.

Coming from outside-in though, we sometimes need to reach a device, or IP, from outside our network. One of the ways to do this is by connecting over the public Internet. Earlier I said networks are private though? Right, so do this we need a public IP address. Unlike the private IP addresses (e.g. 192.168.1.30), a public IP is one that anyone in the world on the public Internet can see and reach.

In the cloud, public subnets are the only ones that can have public IPs. Though this doesn’t mean every IP in a public subnet is a public IP. That said, for security we should try to only use public subnets for things that must have a public IP address, such as a public-facing application load balancer, SSH and FTP servers, etc.

Gateways: Internet Access

So how does stuff in the network access the Internet? Sure, that’s pretty important for most networks, whether we’re just loading web pages at home, or having an application download its dependencies.

This is where gateways come in, and again you’re actually using one right now to read this article!

There’s two types of gateways to know about:

1. NAT Gateway: lets private subnets access the Internet, but not the other way around.

2. Internet Gateway (IGW): lets public subnets have both inbound & outbound Internet access.

NAT Gateway

NAT stands for Network Address Translation. Just a fancy way of saying that it translates private IP addresses (e.g. 192.168.1.30) to a single public IP address on the Internet.

For example, pick two different computers/phones at home right now and on each of them go to https://www.whatsmyip.org/. You’ll see the same public IP address reported, even though you used two different devices. That’s because most consumer-grade home routers also serve as the NAT Gateway in home networks.

In more technical terms, that public IP address you saw was the current public IP address assigned by your Internet Service Provider (ISP) to your Wide Area Network (WAN). Note: your ISP may change your public IP at any point, and this is quite common for dynamic IPs.

So think of it like a funnel for your private network to access the Internet, and remember this behavior of the single public IP address. It will come in handy when we talk about firewalls.

Internet Gateway (IGW)

The other is Internet Gateways. Internet Gateways are for public subnets, and they can allow inbound traffic from the Internet too.

Put simply, if your instance/machine has a public IP, you need a Internet Gateway for it to communicate with the Internet.

Security Groups

The last piece we’ll learn today is Security Groups. These are a virtual firewall for your networks or instances. That means it has ingress and egress rules that define what IP ranges and ports are allowed to receive or transmit on respectively.

By default, everything on the same network can reach one another. To lock things down, we use Security Groups and set ingress and egress rules. For example, only allowing things from a “Application” private subnet to talk to a “Databases” subnet on port 3306 for MySQL. Or opening port 443 for a public web server.

An easy way to test if you can reach a certain port at a target IP is with nc -vz <ip> <port> like: nc -vz 10.0.0.92 3306

Let’s wrap up

These are the same essentials you’ll use in cloud providers like AWS! No voodoo magic going on here, just good ol’ networking but shifted into on-demand scalable cloud computing! Now go tell your friends, family, or neighbors how simple it all is!

Want more?

If this has helped you understand cloud networking better, please consider following/subscribing to my newsletter!

You may be wondering how to secure traffic across private networks without it going over the public Internet. On that note, if I see enough interest through comments/likes, I’ll create a Part 2 to go into intermediate concepts like VPC peering, Transit Gateway, and PrivateLink!

Simpler is Better

Ok, the title may be a bit reductive, but seriously, simpler is (usually) better. Let’s talk about the underdog of interfaces: the Command Line Interface (CLI).

Let me guess, you’ve got a small team (or are part of one), with a short deadline and a brilliant product idea. What do you do?

Spin up a backend and design a REST API. Oh users need something to see and control so throw in a shiny React frontend. REST API getting complicated? Split it into backend API and Backend For Frontend (BFF) API. Great, two months go by and the product is still not ready. In fact, I’d probably spend 3 weeks just debugging CORS issues and arguing over UI layout in Figma.

Or…do you start with a CLI that solves the problem in a few days?

I know this is brazen, and one size never fits all. I’ll get to that, too. The point is that we’re so sold on the idea that “real software products” need a polished web UI and/or desktop applications that we forget the humble CLI.

In fact, CLI is making a comeback (or maybe it was never quite lost, just forgotten). **Especially** with the rise of AI agents and standards like Model Context Protocol (MCP).

Interface Trade-off: Frontend vs CLI

I love pro/con lists, so let’s do that.

Frontend (UI)

• ✅ Looks great (hopefully)

• ✅ Great for all users, even non-technical

• ✅ Web applications keep users on latest version easily

• ❌ Good UI/UX is hard and time-consuming. Most of us suck at it.

• ❌ Highest development and QA time of all. Slowest velocity.

• ❌ Yet another layer that can break/have bugs

• ❌ Compatibility can be a nightmare

• ❌ Hard for AI to integrate well with

For most teams, even highly skilled ones, UIs where ideas go to die a slow, expensive death.

CLI

• ✅ Straightforward architecture. Some don’t even need an API to talk to.

• ✅ Easy to test

• ✅ Scriptable, automatable, easy to integrate with AI

• ✅ Highest velocity to develop and maintain

• ❌ User must know their way around a terminal

• ❌ Limited visual capabilities. TUI libraries exist but are niche. Note: some of us consider this a ✅ too.

To be clear, data and security are the common thread between these. You’re (almost) always going to start with an API to secure client/server access to some data, unless your CLI is static or client-side-only. In more mature organizations, the API is usually first, and CLI and/or SDKs use code-generation to automate development or make it consistent with the API.

Why Velocity Matters?

Shipping fast matters. Not just in startups, and not just at the onset of a new project.

Every extra interface layer is a recurring tax on your design, development, QA, and release times.

I’ve seen really innovative product ideas fail because they took too long to build, or because the second a significant feature was needed, it took too long to work cross-functionally to make it a reality.

Starting with a CLI when practical can reduce your time-to-value from months to mere days, or hell, even hours. You can always add a UI later if the demand justifies it.

Agentic AI: zero-code integration with CLIs

Let’s talk about how AI is reshaping interfaces.

The world is drooling over Function Calling, Model Context Protocol (MCP), A2A, and probably more protocols by next week. They all rely on invoking structured, deterministic behavior.

That’s CLI 101.

AI agents need predictable inputs, consistent outputs, and knowledge of available tasks. This is trivial with CLIs, especially with many CLI frameworks, like clap and cobra, offering built-in shell auto-completion and robust help outputs.

Check out this ~1 minute demo by Block, giving their AI Agent (Goose) the power to integrate with Databricks simply by giving it the Databricks CLI 🤯

That’s crazy; they didn’t write a custom MCP server or have to use some janky “accessibility” AI to control the browser, keyboard, and mouse. They just used the existing CLI. Simple.

Final Thoughts: Less Chrome, More Commands

If you’re launching something soon, ask yourself:

• Could this be a CLI first?

• Will AI tools need to interact with this later?

Or maybe you already have an API you’re proud of, you might consider adding a CLI to make it easier to script or integrate with AI tools.

Not sure where to start? If you have a client/server pattern, start with the API, full stop.

When you’re ready to code a CLI, I highly recommend Go. Rust is also really popular for building CLIs, but I’m currently re-learning Rust for the 4th time, so the learning curve is steep.

A 2022 GitGuardian report revealed that, on average, 3 out of every 1,000 commits contained a credential, and the number of leaked secrets has been rising sharply each year.

C’mon folks, let’s try harder to keep our companies out of the news.

No matter your role, if you contribute code of any variety, you should be aware of best practices when it comes to using secrets in that code. Today though, let’s focus on Terraform and look at real world examples.

(Insecure) Hardcoding sensitive values

Take the following Terraform example. It’s a classic use case that deploys a Lambda Function with an environment variable containing an API Key to a third-party API or database.

resource "aws_lambda_function" "test_lambda" {
  filename      = "lambda_function_payload.zip"
  function_name = "lambda_function_name"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "index.test"
  source_code_hash = data.archive_file.lambda.output_base64sha256
  runtime = "nodejs18.x"

  environment {
    variables = {
      # ⚠️ DO NOT DO THE FOLLOWING ⚠️
      API_KEY = "390de376-a86d-4d5d-9ff7-6ca977a3f2aa"
    }
  }
}

Great, the stage is set. It’s obvious why you shouldn’t do this: the secret is in plaintext and even stored in Git. By the way, if you think it’s easy to remove secrets from Git once they’re committed, think again.

(Insecure) Mark variables as sensitive

Fortunately, Terraform has this thing called sensitive variables we can use.

variable "api_key" {
  description = "Key credential for the API"
  type        = string
  sensitive   = true
}

Great, problem solved, right!? Nope.

Marking variables as sensitive in Terraform is good practice, but it only provides basic protection against some accidental exposure in the CLI and log output. This does NOT encrypt the secret value in the state file; you’re still at risk if this is all you do.

Let’s look at options that are considered truly “secure.”

HCP Terraform & HashiCorp Vault

If you use HCP Terraform or Terraform Enterprise, you’ll rest easy knowing that it encrypts all variable values.

Likewise, HashiCorp Vault is a great solution to securing your secrets in Terraform.

However, I don’t care about paying a premium for a feature we can do pretty trivially on our own. Let’s look at DIY options.

AWS Solutions

Integrating Secrets Manager

AWS Secrets Manager is a managed service for…yep, managing secrets. Secrets Manager is one of the go-to tools for securing sensitive values for applications hosted on AWS, and we can tap into this power for Terraform too.

The process is fairly simple:

1. Create the secret in AWS Secrets Manager. Use whatever method you prefer (AWS Console, CLI, etc.), just not Terraform. Take note of the Secret ID.

2. Use Terraform’s aws_secretsmanager_secret_version data source to query the Secret by its ID (you can use a regular TF variable to pass the ID if you wish).

3. Use Terraform’s jsondecode() function on the data source’s .secret_string attribute.

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = "arn:aws:secretsmanager:us-east-1:123456789012:secret:my_api_key-123456"
}

locals {
  api_key = jsondecode(data.aws_secretsmanager_secret_version.api_key.secret_string)
}

Pros:

• Centralizes where secrets are managed.

• Access to secrets can be limited to read-only.

• Access can be revoked even if user has access to the Terraform code.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

• Requires a non-Terraform way of creating and editing secrets initially.

KMS-Encrypted Values

AWS Key Management Service (KMS) is another acceptable solution similar to Secrets Manager. With KMS, the process is nearly identical:

1. Use the AWS CLI for KMS to encrypt the plaintext secret value and get its ciphertext.

2. Create a local or variable in Terraform to store the KMS ciphertext.

3. Use Terraform’s aws_kms_secrets data source to dynamically decrypt the ciphertext to its usable value.

locals {
  api_key_ciphertext = "AQECAHgaPa0J8WaeplGCqqVAr4HNvDaFSQ+NaiwIBhmm6qDSFwAAAGIwYAYJKoZIhvcNAQcGoFMwUQIBADBMBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI+LoLdvYv8l41OhAAIBEIAfx49FFJCLeYrkfMfAw6XlnxP13MmDBdqP8dPp28OoBQ=="
}

data "aws_kms_secrets" "this" {
  secret {
    name    = "api_key"
    payload = local.api_key_ciphertext
  }

  # <more secret{} blocks here>
}

locals {
  api_key = data.aws_kms_secrets.this.plaintext["api_key"]
}

Pros:

• Access to secrets can be limited to read-only via the KMS keys used to encrypt them.

• Access can be revoked even if user has access to the Terraform code.

• Sensitive values can be safely commited to VCS like Git, as it is ciphertext. This can make development more transparent, and provide better history for auditing.

• More control over how the encryption key is managed, if using Customer-Managed Keys (CMK), such as key rotation and multi-region keys.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

  • Note: if you are using Terraform v1.10 or later, you can use Ephemeral values to prevent it from being stored in state at all.

• Requires a non-Terraform way of creating and editing secrets initially.

Cloud-Agnostic Solutions

Not using AWS? No problem, you can still do this.

Integrating SOPS

SOPS is an open-source, free, CNCF tool for encrypting and decrypting secrets. It’s cloud-agnostic so you can choose your preferred backer: AWS KMS, Azure Key Vault, GCP KMS, as well as age and GPG for generic environments.

I found this blog by dev.to very useful if you’d like to setup SOPS with Terraform/Terragrant.

This essentially involves the following process:

1. Store your secrets in a file like secrets.yaml

2. Encrypt the file using SOPS CLI (you can also use a Terraform null resource with local provisioner).

3. Use the SOPS Terraform provider’s sops_file data source to decrypt the file and use throughout your Terraform configuration.

# secrets.yaml
api_key: "supersecret"

data "sops_file" "secrets" {
  source_file = "${path.module}/secrets.enc.yaml"
}

locals {
  api_key = data.sops_file.secrets.data["api_key"]
}

Pros:

• Extremely flexible. Encryption methods can be mixed and matched to various providers.

• Supports both cloud-managed and local services.

• Sensitive values can be safely commited to VCS like Git, as it is ciphertext. This can make development more transparent, and provide better history for auditing.

Cons:

• Sensitive value plaintext can still be stored in Terraform state file if the decrypted value is stored in a resource’s unprotected attributes.

  • Note: if you are using Terraform v1.10 or later, use Ephemeral values to prevent it from being stored in state at all.

• Key rotation is supported but may be more difficult to implement when using local GPG or age keys.

Detect Leaked Secrets

Time for your homework assignment. That’s right, action!

You can take a proactive approach to keeping your code (infrastructure and otherwise) secure by running a free, open-source, tool like trufflehog!

I’m not sponsored by anyone, but seriously…with just a few minutes, you can install trufflehog and run it against your local Git repo.

Tools like these can even be executed as pre-commit and pre-receive hooks, as well as CI/CD, so you don’t accidentally leak secrets in the first place.

What we learned.

Managing secrets is an every-day task for DevOps and platform engineers. These secrets can be a major attack vector if we’re not careful.

We’ve seen the examples of what NOT to do. With multiple solutions for AWS and other environments, we know what secure Terraform secrets should look like.

I expect figures like those reported by GitGuardian at the beginning will continue to increase each year; however, hopefully these best practices will help you reading this to break that trend.

✨ Introduction

If you’re into space, objects in orbit, and data… I have a treat for you today! I have really enjoyed getting more familiar with Pyspark 4’s Custom Data Sources, and recently wanted to explore more datasets for real-time streaming use cases.

One day, I stumbled across this TLE API, a public API providing data from CelesTrak, a non-profit that hosts data for the space community.

Satellite data is super compelling for real-time use cases and they’re constantly moving. Tens of thousands of these objects (over 63k known to the public in fact) are in orbit for a variety of purposes: communication satellites like Starlink, weather satellites, navigation/GPS satellites, reconnaissance satellites, and even natural satellites like asteroids or the moon.

In this post, we’ll build a real-time streaming pipeline using Apache Spark to track satellites using Two-Line Element (TLE) data, predict their near-future positions, and visualize their motion in 3D.

🔌 Data Sources

First, we need to identify our data sources. How does one get this data, much less determine a satellite's position or velocity? If you work in space telemetry or are already a hobbyist, you probably already know the answer; for the rest of us, let’s take a moment to understand a piece of data known as TLE (Two-Line Element).

🏛️ A Brief History of TLE

TLEs (Two-Line Element Sets): The standard format for describing satellite orbits.

It’s called “two-line” because it literally consists of two lines of 69-character data that encode orbital elements necessary to calculate a satellite’s position and velocity at a given time.

Format example:

ISS (ZARYA)
1 25544U 98067A   24101.21315433  .00002243  00000+0  48609-4 0  9991
2 25544  51.6392  59.7121 0003000  73.4153 286.7121 15.50011348397460

The US Air Force developed this format, and it was adopted by NORAD (North American Aerospace Defense Command) during the Cold War to track objects in space. It was designed to be machine-readable, which at the time meant early mainframe systems and punch cards!

For details on each field encoded into this format, check out the Wikipedia for TLE.

Public Sources

TLE is publicly available from sources like:

• Celestrak (or APIs like TLE API built-on Celestrak data)

• Space-Track.org

TLEs are updated frequently and can be streamed periodically.

Simplified Perturbations Models (SGP/SDP)

Unfortunately, TLE cannot tell us a satellite's position or velocity by itself. For this, we need to use an established mathematical algorithm like SGP4 or SDP4, which are known as Simplified Perturbations Models.

Don’t worry—we don’t have to be a rocket scientist to understand this. Think of it as a function of two inputs—the TLE and a timestamp. In other words, we can say, “Based on the TLE, where will this object be at this point in time?”

🏗️ Architecture Overview

• Ingestion: TLE data is streamed in with a custom spark source we build that calls the HTTP TLE API, ingesting TLE for one or more satellites.

• Processing: Transform the satellites’ state into a usable format (coordinates we can plot relative to Earth).

• Prediction: Use SGP/SDP models combined with TLE to predict the satellites' position and velocity at various points in time.

• Visualization: 3D plots in matplotlib.

📡 Ingesting Satellite Data

We’ll cover a lot of code here, but I’ll break it down step by step. As a prerequisite, know that this will require Pyspark 4 (I used pyspark==4.0.0.dev1 at the time of this writing).

Project Setup

First, we will bootstrap the project with some dependencies. I am using uv to manage the Python project.

uv init --python python3.11 spark-ingest-tle
cd spark-ingest-tle
uv add \
  grpcio \
  grpcio-status \
  pyspark==4.0.0.dev1 \
  astropy \
  sgp4 \
  matplotlib \
  pandas

Now for the main code, let’s take care of our imports and setting up the Spark Session:

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from pyspark.sql.datasource import DataSource, DataSourceStreamReader, DataSourceReader, InputPartition
from pyspark.sql.types import StructType
from pyspark.sql import SparkSession, functions as F

import requests
from datetime import datetime, timedelta

spark = SparkSession.builder.appName("SatelliteStream").getOrCreate()

Implementing Streaming Reader

To support streaming with our custom source, we must implement the DataSourceStreamReader from PySpark. This provides the interface we must use to implement how partitions are generated, how data is read, and how offsets/checkpoints are defined.

def fetch_tle_data(norad_id: str, api_key: str) -> dict:
    headers = {"Accept": "*/*", "User-Agent": "curl"}
    url = f"https://tle.ivanstanojevic.me/api/tle/{norad_id}"
    response = requests.get(url, headers=headers, params={"api_key": api_key})
    response.raise_for_status()
    return response.json()

class SatellitePartition(InputPartition):
    def __init__(self, norad_id: str, name: str,tle_line1: str, tle_line2: str, tle_timestamp: datetime, size: int):
        self.norad_id = norad_id
        self.name = name
        self.tle_line1 = tle_line1
        self.tle_line2 = tle_line2
        self.tle_timestamp = tle_timestamp
        self.size = size

class SatelliteStreamReader(DataSourceStreamReader):
    def __init__(self, schema: StructType, options: dict):
        self.norad_id = options.get("norad_id").split(",")
        self.size = int(options.get("size", 5))
        self.timedelta = options.get("timedelta", "seconds=1").split("=")
        self.timedelta = {self.timedelta[0]: int(self.timedelta[1])}
        self.api_key = options.get("api_key", "DEMO_KEY")

    def initialOffset(self):
        """
        Return the initial offset of the streaming data source.
        A new streaming query starts reading data from the initial offset.
        If Spark is restarting an existing query, it will restart from the check-pointed offset
        rather than the initial one.

        Returns
        -------
        dict
            A dict or recursive dict whose key and value are primitive types, which includes
            Integer, String and Boolean.
        """
        return {}
    
    def latestOffset(self) -> dict:
        """
        Returns the most recent offset available.

        Returns
        -------
        dict
            A dict or recursive dict whose key and value are primitive types, which includes
            Integer, String and Boolean.
        """
        # Fetch the TLE data for each of the given satellites
        offsets = {}
        for norad_id in self.norad_id:
            tle = fetch_tle_data(norad_id, self.api_key)
            tle_line1 = tle["line1"]
            tle_line2 = tle["line2"]
            name = tle["name"]
            tle_timestamp = tle["date"]
            offsets[norad_id] = {"name": name, "tle_line1": tle_line1, "tle_line2": tle_line2, "tle_timestamp": tle_timestamp}
        return offsets

    def partitions(self, start: dict, end: dict):
        partitions = []
        for id, val in end.items():
            if id not in start:
                partitions.append(SatellitePartition(id, val["name"], val["tle_line1"], val["tle_line2"], val["tle_timestamp"], self.size))
            else:
                if start[id]["tle_line1"] != val["tle_line1"] or start[id]["tle_line2"] != val["tle_line2"] or start[id]["tle_timestamp"] != val["tle_timestamp"]:
                    partitions.append(SatellitePartition(id, val["name"], val["tle_line1"], val["tle_line2"], val["tle_timestamp"], self.size))
        return partitions

    def read(self, partition: SatellitePartition):
        from datetime import datetime, timedelta
        from sgp4.api import Satrec, jday, SGP4_ERRORS
        
        satellite = Satrec.twoline2rv(
            partition.tle_line1,
            partition.tle_line2
        )

        for i in range(partition.size):
            ts = datetime.now() + (timedelta(**self.timedelta) * i)
            jd, fr = jday(
                ts.year,
                ts.month,
                ts.day,
                ts.hour,
                ts.minute,
                ts.second,
            )

            e, r, v = satellite.sgp4(jd, fr)
            if e != 0:
                e = SGP4_ERRORS.get(e, 'Unknown error')
            r = {"x": r[0], "y": r[1], "z": r[2]}
            v = {"x": v[0], "y": v[1], "z": v[2]}
            yield (
                ts,
                r,
                v,
                e,
                partition.norad_id,
                partition.name,
                datetime.strptime(partition.tle_timestamp, "%Y-%m-%dT%H:%M:%S%z"),
                partition.tle_line1,
                partition.tle_line2,
                jd,
                fr
            )

Let’s break it down.

It starts with the constructor __init__ which is where we’ll grab any custom options we’d like to define.

    def __init__(self, schema: StructType, options: dict):
        self.norad_id = options.get("norad_id").split(",")
        self.size = int(options.get("size", 5))
        self.timedelta = options.get("timedelta", "seconds=1").split("=")
        self.timedelta = {self.timedelta[0]: int(self.timedelta[1])}
        self.api_key = options.get("api_key", "DEMO_KEY")

These are the options we can pass in like so: spark.readStream.option(“norad_id”, “33499”)…

Disregard the schema argument here for now. We won’t be using this since our data source will have a fixed schema based on the TLE API.

initialOffset and latestOffset

Offsets are how Spark Structured Streaming keeps track of which data has already been processed in a stream. Think of it like a bookmark, or a cursor, in database terms.

When developing your own custom pyspark source, you get to define what these offsets look like. How you choose to model this will heavily depend on the system(s) that your source is consuming; in a durable stream like Kafka or Kinesis, this would involve storing attributes like the sequenceNumber and shardId. In our case, the TLE API doesn’t offer a way to retrieve previous data, so we just won’t worry about replayability; however, the API does return a date field, and we can check if the date, or the contents of line1 or line2, have changed since the last time we fetched the TLE.

Our Offset Data Model

Since I want to compare these values each time I poll the API, I’ll store these things in my offsets. Therefore, the offsets will look like this:

{
  satellite_id: {
    "name": name,
    "tle_line1": tle_line1, 
    "tle_line2": tle_line2, 
    "tle_timestamp": tle_timestamp
  },
  ...
}

Using the satellite ID (NORAD ID) as a key allows us to track multiple satellites in the same stream, which will be useful and more scalable later on.

Initial Offset

Our class’s initialOffset function will initialize the offset data for new streams. Once a stream is started and successfully processed, the offsets will be written to Spark checkpoints and no longer use the initialOffset function. In our case, we haven't fetched the TLE data yet, so we’ll initialize the offset to an empty dict.

Latest Offset

    def latestOffset(self) -> dict:
        # Fetch the TLE data for each of the given satellites
        offsets = {}
        for norad_id in self.norad_id:
            tle = fetch_tle_data(norad_id, self.api_key)
            tle_line1 = tle["line1"]
            tle_line2 = tle["line2"]
            name = tle["name"]
            tle_timestamp = tle["date"]
            offsets[norad_id] = {"name": name, "tle_line1": tle_line1, "tle_line2": tle_line2, "tle_timestamp": tle_timestamp}
        return offsets

The latest offset should return the most recent offset available for our source. This is where we will actually make the HTTP request to fetch TLE data for each of the satellites being tracked.

As you recall from the class constructor, we allow the user to pass in multiple satellite IDs separated by commas.

Partitions

If offsets act like pointers for the source, then the class's partitions method involves taking Point A and Point B (start/end) and dividing them into one or more logical partitions.

The read() method will be invoked for each partition we return by our source.

In my case, I’d like to only emit new data when there is new/updated TLE information for the satellite, and I’d like to generate a partition for each satellite. Thus, I’ll model my partition like so:

class SatellitePartition(InputPartition):
    def __init__(self, norad_id: str, name: str,tle_line1: str, tle_line2: str, tle_timestamp: datetime, size: int):
        self.norad_id = norad_id
        self.name = name
        self.tle_line1 = tle_line1
        self.tle_line2 = tle_line2
        self.tle_timestamp = tle_timestamp
        self.size = size

Then, the partitions are generated by looping over the end offset (latest offset) and comparing it to the start offset (last offset that we processed). If there is new data, we’ll generate a partition.

    def partitions(self, start: dict, end: dict):
        partitions = []
        for id, val in end.items():
            if id not in start:
                partitions.append(SatellitePartition(id, val["name"], val["tle_line1"], val["tle_line2"], val["tle_timestamp"], self.size))
            else:
                if start[id]["tle_line1"] != val["tle_line1"] or start[id]["tle_line2"] != val["tle_line2"] or start[id]["tle_timestamp"] != val["tle_timestamp"]:
                    partitions.append(SatellitePartition(id, val["name"], val["tle_line1"], val["tle_line2"], val["tle_timestamp"], self.size))
        return partitions

Read and Predict

Next, we’d like to predict the position and velocity of the satellite using math. We’ll encapsulate the gory details in our custom source so users don’t have to look like this Charlie Day meme:

One of the most important notes about the read() method is that it should be stateless. You should not access class member variables or try to change the class’s state from within this function. This also means that any modules that need to be imported should be imported from within this function.

Our read function will take the TLE data and calculate the object's position (r) and velocity (v). To make things more interesting, we will calculate not only for the current time but also for N future intervals, therefore predicting where the satellite will be in the near future.

    def read(self, partition: SatellitePartition):
        from datetime import datetime, timedelta
        from sgp4.api import Satrec, jday, SGP4_ERRORS
        
        satellite = Satrec.twoline2rv(
            partition.tle_line1,
            partition.tle_line2
        )

        for i in range(partition.size):
            # SGP requires the timestamp to be in Julian date format
            ts = datetime.now() + (timedelta(**self.timedelta) * i)
            jd, fr = jday(
                ts.year,
                ts.month,
                ts.day,
                ts.hour,
                ts.minute,
                ts.second,
            )

            e, r, v = satellite.sgp4(jd, fr)
            if e != 0:
                e = SGP4_ERRORS.get(e, 'Unknown error')
            r = {"x": r[0], "y": r[1], "z": r[2]}
            v = {"x": v[0], "y": v[1], "z": v[2]}
            yield (
                ts,
                r,
                v,
                e,
                partition.norad_id,
                partition.name,
                datetime.strptime(partition.tle_timestamp, "%Y-%m-%dT%H:%M:%S%z"),
                partition.tle_line1,
                partition.tle_line2,
                jd,
                fr
            )

Finishing the Custom Data Source

To put it all together for our custom data source, we now need to implement the DataSource class, which is the top-level wrapper.

class SatelliteDataSource(DataSource):
    """
    A data source for satellite data.
    """

    @classmethod
    def name(cls):
        return "satellite"

    def schema(self):
        return """
            ts TIMESTAMP,
            pos STRUCT<
                x:DOUBLE,
                y:DOUBLE,
                z:DOUBLE
            >,
            velocity STRUCT<
                x:DOUBLE,
                y:DOUBLE,
                z:DOUBLE
            >,
            e STRING,
            norad_id STRING,
            name STRING,
            tle_timestamp TIMESTAMP,
            tle_line1 STRING,
            tle_line2 STRING,
            jd DOUBLE,
            fr DOUBLE
        """

    def streamReader(self, schema: StructType):
        return SatelliteStreamReader(schema, self.options)
   
    

Finally, we can register the source so Spark knows about it.

spark.dataSource.register(SatelliteDataSource)

🤩 Using the Custom Source

To use the source, we write some familiar, idiomatic, Pyspark code:

df = (
    spark
        .readStream
        .format("satellite")
        .option("norad_id", "25544,33499")
        .option("size", "5")
        .option("timedelta", "minutes=1")
        .option("api_key", "DEMO_KEY")
        .load()
)

# Processing the stream
q = (
    df
        .writeStream
        .format("console")
        .outputMode("append")
        .trigger(once=True)
        .start()
        .awaitTermination()
)

Processing

Now that we have a streaming source that can read TLE data for multiple satellites, we can build processing use cases around this.

There’s all sorts of processing we could do against this TLE data: anomaly/drift detection, collision prediction, data visualization, etc.

Let’s examine a simple processing use case of transforming the data into Earth-centered, Earth-fixed (ECEF) coordinates and then visualizing the data in a 3D plot.

Convert TEME to ECEF Coordinates

The raw position coordinates we have for now are in the True Equator, Mean Equinox (TEME) coordinate system. We need to convert these into ECEF coordinates to plot them in relation to the Earth.

This is a classic use case for data transformations. Let’s write a helper function, which we can later use as a User-Defined Function (UDF).

from astropy.time import Time
from astropy.coordinates import TEME, ITRS
from astropy import units

# Convert TEME to ECEF
def teme_to_ecef(rx, ry, rz, jd, fr):
    teme_position = TEME(
        rx * units.km, 
        ry * units.km, 
        rz * units.km, 
        obstime=Time(jd + fr, format="jd")
    )
    ecef_position = teme_position.transform_to(ITRS(obstime=teme_position.obstime))
    return ecef_position.x.value, ecef_position.y.value, ecef_position.z.value

Data Visualization

Static visualization can be done with matplotlib. We will use a 3D scatter plot to represent satellite positions and a wireframe sphere to represent the Earth.

Let’s put this in a helper function to be reused:

from pyspark.sql import DataFrame
import matplotlib.cm as cm

def visualize(df: DataFrame):
    pDF = df.toPandas()
    pDF["color_id"] = pd.factorize(pDF["norad_id"])[0]

    # Generate a color map for unique norad_ids
    unique_ids = pDF["norad_id"].unique()
    id_to_color = {
        norad_id: cm.viridis(i / len(unique_ids))
        for i, norad_id in enumerate(unique_ids)
    }

    # Convert TEME to ECEF
    pDF["x_ecef"], pDF["y_ecef"], pDF["z_ecef"] = zip(*pDF.apply(lambda row: teme_to_ecef(
        row["pos"]["x"], row["pos"]["y"], row["pos"]["z"], row["jd"], row["fr"]
    ), axis=1))

    # Plotting
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.set_xlim([-7000, 7000])
    ax.set_ylim([-7000, 7000])
    ax.set_zlim([-7000, 7000])
    ax.set_xlabel('X (km)')
    ax.set_ylabel('Y (km)')
    ax.set_zlabel('Z (km)')

    # Plot the Earth
    u, v = np.mgrid[0:2*np.pi:100j, 0:np.pi:50j]
    x = 6371 * np.cos(u) * np.sin(v)
    y = 6371 * np.sin(u) * np.sin(v)
    z = 6371 * np.cos(v)
    ax.plot_wireframe(x, y, z, color='blue', alpha=0.1)

    # Plot the satellites with different colors for each one
    for norad_id, g in pDF.groupby('norad_id'):
        ax.scatter(
            g["x_ecef"], 
            g["y_ecef"], 
            g["z_ecef"], 
            color=id_to_color[norad_id],
            s=10,
            label=f"Satellite {norad_id}"
        )

    ax.legend()
    plt.show()

🚀 Putting It All Together

This is where we put it all together! We’ll read-stream TLE data for three satellites simultaneously, compute their position and velocity, predict where they will be at each minute over the next 4 hours, convert those positions to an ECEF coordinate system, and finally plot these predictions on a 3D visualization.

df = (
    spark
        .readStream
        .format("satellite")
        .option("norad_id", "25544,33499,46362")
        .option("size", "240") # 240 predictions
        .option("timedelta", "minutes=1") # 1-minute predictions. 240 x 1 minute = 240 minutes = 4 hours
        .option("api_key", "DEMO_KEY")
        .load()
)

# Processing the stream
q = (
    df
        .writeStream
        .outputMode("append")
        .trigger(once=True)
        .foreachBatch(lambda df, batch_id: visualize(df))
        .start()
        .awaitTermination()
)

Result and Final Thoughts

Wow. Obviously, I am not the best at matplotlib, and this data visualization does leave some to be desired; however, it’s amazing how simple and idiomatic it is to build.

⚠️ Cautionary Comments

Although it’s quite easy to build your own custom sources now with Spark 4 and newer Databricks Runtimes. I would caution you to think very carefully about the following implementation choices:

• What to use for offsets

  • Offsets are a critical piece of spark streaming. If your upstream source has any knobs or attributes that make it durable or replayable, you should definitely consider leveraging these in your source.

• How partitions are created

  • The number of partitions your source generates from the partitions() method will directly impact the amount of parallel processing that Spark can do with its executors. Avoid having too few partitions if your dataset is very large, and avoid over-partitioning, which can lead to a Large Number of Small Files Problem.

  • NOTE: alternatively, if you don’t need partitioning, you can implement the SimpleDataSourceStreamReader instead of DataSourceStreamReader.

• Avoid accessing state or mutating class members from the read() method. As a general rule of thumb, I try to avoid using the self keyword in this method.

Source Code

All of the source code is available on GitHub. I have also implemented support for batch reading on this source code, so take a look if you are interested in more than just streaming.

I recently gave my home network a fresh makeover with UniFi hardware and came across this API. But I’m also a data & AI maker nerd.

Today, in just a few minutes, we’ll see how to build our own MCP server implementation that runs locally on your machine and uses the local UniFi API. This will let you talk to your network using modern AI Agents like Goose and Claude to unlock natural language access.

Setup / Prerequisites

1. Create an API key for accessing the UniFi Network API

   1. Go to your UniFi console at https://unifi.ui.com

   2. Go to Settings » Control Plane » Integrations and click Create API Key

2. Install uv which we’ll use for managing the Python project.

Create Project Structure

Next, we’ll initialize the project/codebase with a few simple commands:

uv init mcp-server-unifi
cd mcp-server-unifi
uv venv
uv add "mcp[cli]" httpx requests

uv will also create a `hello.py` file which we don’t need so you can delete that.

rm hello.py

Implementing the MCP Server

Let’s begin the implementation by importing some needed modules and defining environment variables for configuration.

# File: main.py

from typing import Any, List, Dict, Optional
import os
from mcp.server.fastmcp import FastMCP
import requests

# Configuration
UNIFI_API_KEY = os.getenv("UNIFI_API_KEY", "YOUR_API_KEY_HERE")
UNIFI_GATEWAY_HOST = os.getenv("UNIFI_GATEWAY_HOST", "192.168.1.1")
UNIFI_GATEWAY_PORT = os.getenv("UNIFI_GATEWAY_PORT", "443")
UNIFI_GATEWAY_BASE_URL = f"https://{UNIFI_GATEWAY_HOST}:{UNIFI_GATEWAY_PORT}/proxy/network/integration"

# Initialize FastMCP server
mcp = FastMCP("unifi")

When we run our server later, this will make it easy to configure if you have a gateway.

Helper Functions

This entire MCP server is simply encapsulating the REST API, which means lots of duplicate code. To keep things DRY let’s add a helper function for making the API calls.

def unifi_request(path: str, method: str, params: Optional[Dict[str, Any]] = None, data: Optional[Dict[str, Any]] = None):
    """
    Make a request to the Unifi API

    Args:
        path (str): The path to the API endpoint
        method (str): The HTTP method to use
        data (Optional[Dict[str, Any]], optional): The data to send to the API. Defaults to None.

    Returns:
        dict: The response JSON from the API
    """
    url = f"{UNIFI_GATEWAY_BASE_URL}/{path}"
    headers = {
        "Content-Type": "application/json",
        "X-API-Key": UNIFI_API_KEY,
    }
    response = requests.request(method, url, headers=headers, params=params, data=data, verify=False)
    return response.json()

Defining Resources

Now, the fun part! We are going to define some resources for our server. In MCP, resources allow the server to expose data and content that can be read by the clients (like Claude) and used as context for LLMs.

Each resource has a URI that follows this format:

[protocol]://[host]/[path]

Using the UniFi Network API, we can read data about the UniFi sites, which are at the top of our UniFi hierarchy. Within a site, we can also read data about adopted devices, connected clients, etc.

So, for the following endpoint, we could come up with a resource and URI like sites://{site_id}/devices

To implement resources in our Python code, we’ll make use of the server’s decorators like @mcp.tool() and @mcp.resource().

@mcp.resource("sites://")
async def list_sites() -> List[Dict[str, Any]]:
    """List all sites in the Unifi controller"""
    sites = []
    params = {"limit": 200, "offset": 0}

    while True:
        resp = unifi_request("/v1/sites", "GET", params=params)
        sites.extend(resp["data"])
        if resp["count"] != resp["limit"] or resp["totalCount"] <= len(sites):
            break
        params["offset"] += resp["limit"]

    return sites


@mcp.resource("sites://{site_id}/devices")
async def list_devices(site_id: str) -> List[Dict[str, Any]]:
    """
    List all devices in a specific Unifi site
    
    Args:
        site_id (str): The ID of the site to list devices for
        
    Returns:
        List[Dict[str, Any]]: List of devices in the site
    """
    devices = []
    params = {"limit": 200, "offset": 0, "site_id": site_id}
    
    while True:
        resp = unifi_request(f"/v1/sites/{site_id}/devices", "GET", params=params)
        devices.extend(resp["data"])
        if resp["count"] != resp["limit"] or resp["totalCount"] <= len(devices):
            break
        params["offset"] += resp["limit"]
    
    return devices

Running the MCP Dev Server

Finally, we’re ready to run our MCP server! We just need an entrypoint in the code, so don’t forget to add:

if __name__ == "__main__":
    # Initialize and run the server
    mcp.run(transport='stdio')

You can run your MCP server standalone just to test that it starts up successfully, with uv run mcp dev main.py. This will also start the MCP Inspector at http://localhost:5173 which is great for testing and debugging.

Configuring for Goose

If you’d like to use Block’s Goose AI Agent, open Goose and go to Settings » Extensions » Add custom extension.

In the form that prompts you, fill in the following details, and be sure to change “username” to your own username as used by your machine:

• ID: unifi

• Name: unifi

• Description: Get information about your UniFi network

• Command:

  • /Users/username/.local/bin/uv --directory /Users/username/path/to/mcp-server-unifi run main.py

• Environment Variables:

  • UNIFI_API_KEY: input your API key

Click Add to save the changes and you’ll have successfully added the MCP server to Goose for local use!

No need to restart Goose, go ahead and try it out:

Configuring for Claude Desktop

If you’d like to configure your MCP for use with Claude Desktop app, open Claude and go to Settings » Developer » Edit Config.

This will open a claude_desktop_config.json file in a text editor. If it opens your file explorer / Finder application, just open the file it has selected automatically.

Paste the following into this configuration file, and be sure to change “username” to your own username as used by your machine.

{
    "mcpServers": {
        "unifi": {
            "command": "/Users/username/.local/bin/uv",
            "args": [
                "--directory",
                "/Users/username/path/to/mcp-server-unifi",
                "run",
                "main.py"
            ]
        }
    }
}

Save your changes to the file, then restart the Claude Desktop application and give it a try!

Conclusion

Hopefully, you’ve enjoyed yet another MCP blog. Yes, MCP is very popular right now. While it may not be as “game-changing” as tech influencers will claim, it’s certainly very easy to use and build upon—and standards are almost always a good thing!

The source code for this is available on GitHub (although not accepting PRs as it is a demo only): https://github.com/zcking/mcp-server-unifi.

I want to hear from you though…what do you think about MCP? Can you think of anything you’d like an MCP server for but doesn’t exist yet? Tip: check out this growing awesome-list of MCP servers and subscribe to MakeWithData for more!

As the adoption of large language models (LLMs) like GPT-4, o1, Llama, and Gemini increases, developers are turning to frameworks and higher abstractions to simplify building Gen AI applications powered by these models.

Unless you were asleep the past 2 years, you’ve probably at least heard of LangChain, a modular framework for building applications with LLMs, including RAG, chatbots, agentic workflows, and more. LangChain has grown extremely popular, with nearly 100k stars on GitHub at the time of this writing 🤩

Today, though, I want to discuss another open-source framework, Haystack, developed by a Berlin company, deepset.

First Thoughts

My first opinion of Haystack was that it is extremely simple. Now remember that statement because, in a moment, we’ll see how that could be Haystack’s kryptonite, too.

For context, I have been writing Python code for around a decade, so I had no syntax struggles with either framework. However, the semantics in Haystack had a much smaller learning curve, as everything boils down to either a component or a pipeline. This is immediately pointed out in their overview.

After a couple of hours reading the “Getting Started” docs and following a simple RAG example, I successfully implemented my own custom component to retrieve data from a REST API and integrate it with a prompt builder and LLM. Creating custom components was very Pythonic and honestly made sense even if I weren’t messing with LLMs.

That said, as I began trying to make more dynamic patterns like an agentic workflow or multi-modal applications, it felt more limiting—the pipelines were easy to adapt to linear workflows, but more organic, complex flows didn’t feel as intuitive. This may be just my lack of experience.

Comparing Customization

Pictures speak 1,000 words; code speaks 10,000 😅 Let’s see how they compare by writing a simple custom retriever in LangChain and then in Haystack:

LangChain Custom Retreiver

This is pretty straightforward, although we’re clearly just hardcoding fake documents and doing one of the most naive and inefficient searches possible.

With LangChain, we just inherit the BaseRetreiver abstract class and implement the _get_relevant_documents() method. The class uses Pydantic to model the class’s required inputs, which we only define one: documents. A more realistic retriever might implement an embedding algorithm, query a vector database, call a REST API, or even use an LLM (e.g., generate relevant questions based on the user’s query).

Haystack Custom Retriever

Haystack looks pretty similar, but I took the liberty of using the InMemoryDocumentStore, which has BM25, a probabilistic information retrieval model that basically uses a bag-of-words and term frequency.

With Haystack, the component is a Python class marked by the @component decorator, and a run() method.

These retrievers are not usually invoked directly; they are often part of a chain or pipeline. Let’s see another example that uses a component in a pipeline: custom prompts.

Haystack Pipelines

Okay, let’s create a more…fun, custom component. This calls the Chuck Norris joke API and returns the joke that we will then use in a pipeline to generate images 🤣

Next, you create a Pipeline, add the components, and connect them.

As you can see, this is very straightforward with Haystack. You register each component with a name and then wire them together, essentially connecting each component’s inputs and outputs.

In this example, I am pipelining the Chuck Norris API → PromptBuilder → DALL-E image generator.

Notice that I don’t connect anything in the pipeline to the style template variable; instead, I leave it empty to pass that in via the command-line arguments and change the style as I like.

Running this pipeline gave me some great laughs, and even if you hate Haystack, I recommend giving it a shot for your own fun.

Joke: When Chuck Norris break the build, you can't fix it, because there is not a single line of code left.

LangChain Chains

If I were to write this in LangChain, the simplest way would be to write custom tools with the @tool decorator, or you could use classes and subclass them from the BaseTool class provided by LangChain. Afterward, we would construct a chain, such as the SequentialChain, and provide it with a list of our tools.

I found that the process of creating a PromptTemplate in LangChain and using it between the joke generator and the image generate was strangely verbose; however, this makes sense given how LangChain is designed to be more open and caters well if you’re building agents / agentic workflows.

Final Comparison

Both are very similar, but I found Haystack simpler and with an easier learning curve. Haystack would be great if you just want to get up and running fast, building applications such as RAG, Q&A, search, or general sequential workflows.

LangChain, on the other hand, has a steeper learning curve but is more robust. LangChain is better for agentic workflows and advanced concepts such as tool calling, ReAct (Reasoning and Acting), and chat memory. LangChain probably has more community tools available now, although Haystack also has tons.

At the end of the day, you shouldn’t listen to the hype about specific tools; try Haystack and LangChain out and see which one fits your needs and development style better!

🚨 Hey! I’m currently working on a personal health and fitness Haystack application that uses data from apps like MyFitnessPal, Hevy, and leading LLMs to give personal feedback and recommendations. If you’d like to see a project-based post for this in the future, be sure to subscribe!

MakeWithData is free today. But if you enjoyed this post, you can tell MakeWithData that their writing is valuable by pledging a future subscription. You won't be charged unless they enable payments.

That said, much of the trends and startups were pure hype, and you either drank the Kool-Aid or played it slow with the risk of “falling behind” (seemingly, at least). The difference in 2025 will be that generative AI tools are currently in the “trough of disillusionment.” This means we can expect to see these tools mature and some of the more gimmicky offerings that are just out for a quick buck to die down.

So, as data engineers and stewards of our organization’s data, we must prepare for the pragmatic integration of AI into our data products this upcoming year. I’ll share several use cases in categories such as:

• Developer Experience

• Data Processing

• Data Consumption

• New Products

Predictive Modeling and Forecasting

Data engineers can integrate AI models into data pipelines to generate predictions and forecasts. For example, we may predict customer churn, forecast sales or website trends, and identify anomalies.

Here are some tools you can use to quickly forecast on your data:

• Prophet (by Meta): Prophet is a time series forecasting library designed for handling data with seasonality. It is particularly effective for forecasting business metrics, such as sales, website traffic, and resource usage.

• Apache Spark MLlib (+ Prophet): Spark MLlib is a scalable machine learning library that is integrated with Apache Spark. It provides various classification, regression, clustering, and collaborative filtering algorithms, making it suitable for building predictive models on large datasets.

• Databricks ai_forecast() function: ai_forecast() is a table-valued function in Databricks designed to extrapolate time series data into the future. Literally just a SQL function—it can’t get any easier than this.

Retrieval Augment Generation (RAG) on your Data

Your data is your company’s most valuable asset—period, full stop. Many companies, even outside the tech sector, have realized this value in recent years. Gen AI has expanded this value further with the Retrieval Augmented Generation (RAG) concept, in which a large language model (LLM) dynamically retrieves data from a vector database to provide more contextual and correct answers.

As data engineers, we must start thinking about the most valuable data for this purpose. RAG isn’t limited to structured data—in fact, it is typically more valuable in unstructured data, which is traditionally more difficult to search or parse. For example, all of the following could be great candidates you could create “embeddings” for and store in a vector database:

• PDF Documents (tip: check out this PDF reader for Apache Spark!)

• Images/Pictures

• Audio Files

• Logs and other Text Files

• Excel/Powerpoint/Word Documents

Of course, AI can also access your structured and semi-structured data, but you may leverage concepts like AI Tools or Function Calling to achieve that.

For RAG, here are some tools that can be used to build RAG pipelines, create and manage vector data storage, and integrate RAG with your applications:

• Haystack: an open-source framework for building production-ready LLM applications, RAG pipelines, and state-of-the-art search systems that work intelligently over large document collections. Of course, there is LangChain, which you should also check out.

• Databricks: Databricks offers building blocks for RAG, such as its Vector Search, serverless model serving endpoints, Tools or function-calling, AI agent evaluation, and various LLMs.

• Azure AI Search: managed Azure cloud service that enables efficient searching of unstructured data, offering features like full-text search, autocomplete, and semantic ranking across various data formats and languages.

• Amazon Bedrock: managed AWS cloud service supporting a variety of Foundation Models (FM) and facilitating data vectorization and vector storage.

AI for Data Quality

Whether you’re already experienced in managing data quality or are just tackling issues as they come, AI offers clear opportunities for enhancement.

One of my favorite use cases here is using AI to automatically detect and obfuscate sensitive information automatically. Databricks has a great solution accelerator for this use case with the Protected Health Information (PHI): https://www.databricks.com/solutions/accelerators/automated-phi-removal

I’m excited about using similar features from Databricks, such as ai_classify() and ai_mask(). These simple SQL functions use generative AI to classify input text according to the labels you provide or mask things like PII, respectively.

Not to make it all about Databricks, but I have to give them kudos once more for Lakehouse Monitoring. Lakehouse Monitoring helps you monitor your data integrity, how your data changes over time, statistical distribution, drift detection, and more. More importantly, as we build AI and use AI more ourselves, we need to ensure that AI has appropriate data quality, and Lakehouse Monitoring covers this by tracking model performance/accuracy with metrics like F1 score.

AI for Business Intelligence - cut out the middle man!

BI is usually a back-and-forth process for data analysts and analytics engineers to maintain SQL queries, data visualizations, and business stakeholders' constant intake of different analytical questions. Wouldn’t it be nice if you could just ask the questions and create those pretty bar charts on the fly?

This is one of the most interesting AI use cases for data practitioners, and I expect we’ll see many more in 2025. Major players, such as Databricks with its AI/BI Genie and Microsoft’s Copilot for Power BI, have already begun successfully demonstrating it.

The key to this being successful will come down to the process in terms of “who” manages the tooling and “how” it is administered. Some tools, like Databricks Genie, require the setup of a “space” that is configured with data sources (tables), instructions, and example prompts/queries—these are crucial to get right, and you’ll get out of it what you put into it (effort).

Creating New Data Products with AI

While AI has many uses for increasing our productivity and internal workflows, I believe it also opens the door for us to generate new types of data products. The marketplace concept is already quite popular with cloud providers like AWS, Azure, GCP, and even leading data platforms like Databricks; these marketplaces have always been hosts of a myriad of datasets and applications, so if your organization has data that it would like to monetize it could do so easily.

AI has expanded this market with the opportunity to create AI applications or turn-key components for AI integration. Here are a few examples I can think of, some of which are already being seen:

• Pre-computed Vector databases of common datasets

• Fine-tuned small language models (SLMs) for targeted use cases

• Data integrations for proprietary SaaS vendors: data retrievers, automated vector synchronization, function calling libraries.

• Self-service for enterprise customers to access their own data via technology like Delta Sharing, as more customers seek to utilize their data from vendors in their own AI.

Coding Assistants

AI coding assistants were some of the very first to take off and show significant value to engineering teams. That also makes them some of the most mature already, so if you’re not using one already—stop waiting.

Mileage may vary, but the ones I recommend the most:

• Databricks Assistant: for all coding inside Databricks. Plus, it’s free!

• GitHub Copilot: Although I read many negative comments about Copilot compared to some more recent startups, it’s still one of the best all-purpose coding assistants I have used, regardless of language. It also supports settings like opting out of your completions being used for product improvements (training) and disabling matching public code.

• Amazon Q Developer: Perfect if your organization runs tight vendor security and only allows AWS-native services.

Fixing Errors

Interpreting error stacktraces is a vital skill for engineers when troubleshooting issues. Still, even experienced engineers may need several minutes to parse through the error, search solutions such as StackOverflow, and figure out how to apply the fix to their code.

AI can save you a ton of time by at least interpreting the error for you and often even fixing your code for you. For this reason, many coding assistants like the ones mentioned above have commands like /fix.

Pro-Tip: Databricks even has a Public Preview for its AI assistant diagnosing your failed jobs: https://docs.databricks.com/en/notebooks/use-databricks-assistant.html#diagnose-errors-in-jobs-public-preview

I have heard complaints that the error explanations are sometimes wrong or that the suggested fixes frequently have bugs. You can significantly improve the accuracy of these outputs by toggling whether the tool (e.g., Amazon Q or GitHub Copilot) should index your entire code workspace. This informs the assistant about your complete codebase and provides better context.

Conclusion

I hope you have enjoyed these use cases and have some new ideas to integrate AI into your data engineering workflows and data products in 2025.

Remember, these tools and services are on track for maturity and true enterprise readiness, so don’t settle for less by grabbing the first thing you read about! Try some of the tools I mentioned and decide for yourself. Let me know in the comments if you recommend any other AI tools!

Also, if you don’t mind supporting the MakeWithData blog (I promise it’s really just me, an individual guy who loves to nerd out), please like this post and consider pledging for future posts below!

MakeWithData is free today. But if you enjoyed this post, you can tell MakeWithData that their writing is valuable by pledging a future subscription. You won't be charged unless they enable payments.

Introduction

Event-driven architecture (EDA) is a software design paradigm that revolves around producing and consuming events to trigger processing. In this architecture, systems are built to respond to events in real-time or near real-time, allowing for more dynamic and responsive applications.

Data Engineers often find comfort in the simplicity of cron schedules and workflow orchestration tools like Airflow, Kestra, and Databricks Jobs. However, EDA too can be a powerful pattern in the data engineer toolkit.

Event-driven data engineer use cases

We’ll focus on design patterns in a sec, but first let’s recap a few use cases where EDA might come in handy:

1. File Processing: process or ingest files with an unpredictable file arrival schedule.

2. User Submissions: asynchronously process user submissions such as BI reports, security audits, or data deliveries.

3. Cyber Security: monitor and react to Endpoint Detection and Response (EDR) events to keep devices and environments secure.

4. IoT data processing: process and react to streams of data from IoT devices.

5. Healthcare Monitoring: can monitor patient data from various sources, providing alerts and insights to healthcare professionals for better patient care.

1. Change Data Capture (CDC)

Change data capture, or CDC, is a method of capturing and processing all the inserts/changes/deletes to a table. If you’re using a data table format such as Delta Lake or Iceberg, you can simply enable CDC on the table and query the CDC feed.

CDC can be useful for several of the aforementioned use cases, with the exception of file processing, since the table itself can store any structured or semi-structured data.

2. Trigger on File Arrival

I am a huge fan of Databricks and have to make a plug for its File Arrival Triggers, which I have an entire separate blog post on here:

However, if you aren’t on Databricks, have no fear. You can also build this pattern with common cloud services. For example, in AWS you could use S3 + S3 Event Notifications + Lambda. Additionally, some orchestration tools like Kestra support S3 triggers natively.

Obviously this can be used to process any arbitrary file data as it lands in storage. Another great use case for this pattern is to take user submissions and generate a manifest file in S3 that describes the request, then use this pattern to trigger a data process against the submission. Some ideas for user submissions could be an intensive data analytics report, kicking off ad-hoc security audits, or a data resynchronization process for runbooks.

Note: if you are receiving a large number of files and very frequently, a solution using Databricks Auto-Loader would scale better.

3. SNS + SQS + Lambda

Another popular AWS recipe is Simple Notification Service (SNS), Simple Queue Service (SQS), and Lambda Functions.

At this point nearly all AWS services have some integrations with SNS, as do many 3rd party applications.

By integrating your SNS topic with an SQS queue, you will make the processing more scalable, durable, and performant as SQS provides the retry (re-drive) ability, batching, and optionally you can also set up a Dead Letter Queue (DLQ) for stale messages.

This design pattern is useful for so many use cases because SNS is so easy to integrate. Many monitoring tools like CloudWatch, Datadog, New Relic, Grafana, and others support SNS topics as a notification destination so you could create some alert processors for real-time operations. You could also use this for user activity and IoT events, ingesting it to a lakehouse table or OLTP database.

4. Streaming Services

Moving into more continuous data feeds? High volume, low latency? This is when scalable streaming services such as Kafka, Kinesis, or Apache Pulsar are a better choice.

Streaming tools like these excel at high-volume, continuous data that needs immediate processing. They also typically store data in a persistent manner, allowing you to replay events if needed.

Many technologies make it easy to either read or write to/from streams like Kinesis and Kafka; you could:

• Trigger an AWS Lambda function directly

• Read/write with Apache Spark

• Read/write with Apache Flink

This would be appropriate for more demanding use cases such as high-volume streams of IoT data, auditing API usage or user activity, real-time anomaly detection, processing financial transactions, and more.

Cautions…

As with any design pattern, we should be mindful of its limitations. In EDA solutions, these limitations may not always exist but some things to be aware of are:

• Event Ordering: does sequence matter to your system? Does your EDA design enforce order? E.g. if using SQS, you may need a FIFO queue, and with Kinesis pay attention to your sequenceNumber

• Duplicates: can events be duplicated and is this ok? Tip: at times this can actually be a feature, akin to replaying data.

• Observability: EDA systems usually have higher complexity; metrics and monitoring are important for observability. E.g. are you able to monitor if events fail to deliver, or are delayed? If it does fall behind, can you monitor the backlog needing to be processed?

• Complexity: again, EDA can be more complex to monitor and troubleshoot issues with. Consider if you use case even demands EDA. E.g. if using for file arrival triggers, do your files arrive pretty consistently which could be aligned to a simple schedule? Remember, if all you have is a hammer… everything looks like a nail.

Conclusion

We have concluded this rundown of a few EDA design patterns for data engineering. These patterns make data ingestion/processing more real-time, potentially more cost-effective, and responsive.

I hope you have enjoyed this read. I would seriously love to hear feedback and other ideas for your design patterns!

Also, if you don’t mind supporting the MakeWithData blog (I promise it’s really just me, an individual guy that just loves to nerd out) please like this post and even consider pledging for future posts below!

MakeWithData is free today. But if you enjoyed this post, you can tell MakeWithData that their writing is valuable by pledging a future subscription. You won't be charged unless they enable payments.

Pledge your support

Apache Spark v4.0 is coming soon and is jam-packed with awesome features, among which includes the much-awaited custom data sources API for PySpark (also available in Databricks Runtime 15.3+). Previously, this feature was only available to Scala programmers.

This enables data engineers to create integrations for reading and writing data, in batch or with streaming, with idiomatic Spark code. Let’s take a tour of this new feature by building our own custom data source for AWS Simple Queue Service (SQS).

Implementing the DataSource class

The first thing we need to create is the DataSource class which is the top-level class that represents our custom data source.

from pyspark.sql.datasource import (
    DataSource,
    DataSourceReader,
    InputPartition,
)
from pyspark.sql.types import StructType

from datetime import datetime
from typing import Iterator, Tuple
from dataclasses import dataclass


class SQSDataSource(DataSource):
    """
    PySpark data source for batch querying data from a
    SQS queue.
    """

    @classmethod
    def name(cls):
        return "spark_sqs"

    def schema(self):
        return """
            message_id STRING, 
            receipt_handle STRING, 
            md5_of_body STRING, 
            body STRING, 
            sent_timestamp TIMESTAMP
        """

    def reader(self, schema: StructType):
        return SQSDataSourceReader(schema, self.options)

The name method tells Spark the name of our custom source (spark_sqs) which is what we’ll later use in spark.read.format(“spark_sqs”). Then we have a method to return the data source schema, as well as a simple data source reader.

Note: the DataSource API supports streaming readers, as well as writing your own sinks to write data. In this part we will be looking at the simple reader only.

Implementing the DataSourceReader

Next, we must implement the simple data source reader.

class SQSDataSourceReader(DataSourceReader):
    def __init__(self, schema: StructType, options: dict):
        print('Initializing SQS Data Source Reader')
        self.schema: StructType = schema
        self.options: dict = options
        if not options.get('queue_url'):
            raise ValueError('queue_url is required')
        self.queue_url = options.get('queue_url')
        self.region_name = options.get('region_name', 'us-east-1')
        self.max_messages = int(options.get('max_messages', '10'))
        self.visibility_timeout = int(options.get('visibility_timeout', '20'))
        self.wait_time_seconds = int(options.get('wait_time_seconds', '20'))
        self.delete_message = options.get('delete_message', 'false').lower() == 'true'


    def read(self, partition: InputPartition) -> Iterator[Tuple]:
        import boto3 # Must import here for serialization
        sqs = boto3.client('sqs', region_name=self.region_name)
        response = sqs.receive_message(
            QueueUrl=self.queue_url,
            MaxNumberOfMessages=self.max_messages,
            WaitTimeSeconds=self.wait_time_seconds,
            VisibilityTimeout=self.visibility_timeout,
            MessageSystemAttributeNames=['SentTimestamp']
        )
        for message in response.get('Messages', []):
            yield (
                message['MessageId'],
                message['ReceiptHandle'],
                message['MD5OfBody'],
                message['Body'],
                datetime.fromtimestamp(int(message['Attributes']['SentTimestamp']) / 1000)
            )
            if self.delete_message:
                sqs.delete_message(
                    QueueUrl=self.queue_url,
                    ReceiptHandle=message['ReceiptHandle']
                )

This is mostly handled by the read method, which we will use to receive messages from the queue and yield them; the rest of this code is just boilerplate to offer flexibility so we may configure SQS poll settings.

Using the custom data source

To use our custom data source, we must first register it to make Spark aware of the implementation.

spark.dataSource.register(SQSDataSource)

Now we can read from a queue with idiomatic Spark code:

spark.read.format("spark_sqs").options(
        queue_url='https://sqs.us-east-1.amazonaws.com/1234567890/makewithdata-events',
        region_name='us-east-1',
    ).load().show()

+--------------------+--------------------+--------------------+--------------------+--------------------+
|          message_id|      receipt_handle|         md5_of_body|                body|      sent_timestamp|
+--------------------+--------------------+--------------------+--------------------+--------------------+
|7322a8a2-180f-4a3...|AQEBb9rBz2DJ8RAWA...|c89b42169df3bbb3e...|{\n    "_id": "67...|2024-12-16 20:13:...|
|d6a1d9b8-c15f-44c...|AQEBVnrFbfP/qrBpg...|daa063a083235cf91...|{\n    "_id": "67...|2024-12-16 20:13:...|
+--------------------+--------------------+--------------------+--------------------+--------------------+

Next Steps

This is merely a quick way to play around with custom sources, but our SQS source could be a lot more useful if we added more features:

1. Implement Streaming Reader that deletes messages as the reader writes checkpoints.

2. Poll until queue is empty.

3. Implement a Sink so we can write data to SQS as well.

4. Extract all available Attribute data from the messages.

If you’d like to see these next parts implemented, leave me a comment below and subscribe to the MakeWithData blog!

Today, we will look at many of the most common pitfalls I've seen (and learned the hard myself a time or two) and how to fix them using best practices and rich features from the famous Data Intelligence platform!

1. Relying on Schedules to "orchestrate" Tasks

If you're like me and started using Databricks several years ago, you know that we could not always run multiple tasks in a job, much less orchestrate them. Often, data engineers need to break down a job into smaller steps or tasks, such as separating runtime dependencies, creating sequential and parallelized flow, or modularizing the code. A common mistake here is to build separate jobs and do a bit of cron schedule surgery to align these jobs so they run in a well-timed chain.

For example, we may build a data pipeline for a typical medallion architecture like so:

Meanwhile, other users created their own Apache Airflow servers to orchestrate the task. Fortunately, Databricks saw the need for robust orchestration features and now has several features built-in like multiple tasks in one job, task dependency (can run some tasks in parallel, or others in sequence), as well as conditional task runs (e.g., "run this task if XYZ happened"). The above example is wrong because it is too brittle--if any of the jobs takes too long, it "misses the train" for the next step. Likewise, if any of the jobs fails, there is no way for the other jobs to know to skip, so instead, they run unnecessarily, adding more cloud costs and potential alerts.

Using these features, we should only have one job for this use case. Each step is a Task configured to depend on the previous task completed. You can even refresh an SQL dashboard as one of your job tasks. However, at the time of this writing, Databricks only supports legacy SQL dashboards in this feature. Hopefully, it will add Lakeview Dashboards soon!

2. No Retries

Another common mistake is forgetting to configure max_retries on your Databricks job or its tasks. This is one of the easiest ways to add fault tolerance to your jobs and prevent transient issues from derailing critical workflows.

3. Let it Fail! Don't Swallow Exceptions

Retries are tremendous but only valid when the task actually "fails." This can be an oversight if you write code that catches exceptions and then sends them to a monitoring tool like Prometheus, CloudWatch, DataDog, etc.

cw = boto3.client('cloudwatch', region_name=region_name)

try:
	df = spark.read.table("...")
	# application code...
except Exception as e:
	# Log error to CloudWatch
	cw.put_metric(...)

	# BAD!! Exception gets swallowed, and the job run is considered "successful."
Copy

If you want to instrument some error metrics this way, remember to re-throw the exception to allow Databricks to flag the task as "failed." This will trigger the retries, send notifications, and enable other features like conditional runs.

except Exception as e:
	# Log error to CloudWatch
	cw.put_metric(...)

	# Re-throw to trigger job failure
	raise e
Copy

4. Not Using Notifications

One more easy victory in Databricks Workflows is the notifications. As of this writing, Databricks supports notifications via email, Slack, Microsoft Teams, Pagerduty, and generic Webhooks.

This often replaces convoluted monitoring solutions that require maintaining custom integrations or instrumentation in your code.

5. Git Folder vs. Git Integration for Notebooks

This may be a personal preference, but I strongly support it. If you are using "Git Folders" (formerly known as "Git Repos") in the Databricks workspace as the source for your notebook jobs, you should strongly consider switching to Git Integration.

What's the difference? Git folders are typically created by users and kept in their user directory on the workspace. More importantly, Git folders require explicitly pulling to receive changes; if you go this route, you should use the Databricks SDK from your CI/CD pipeline (e.g., GitHub Actions) to automate checking out new tags/branches or pulling changes from a main branch.

Jobs have built-in Git integration that relieves much of the stress. As shown in the picture above, this feature allows your job to effectively checkout the specified repo and Git ref when a run starts. If you point it at a main branch like 'main' or 'master,' you can achieve a minimal "CD" pipeline with zero CI/CD tooling!

6. Someone Leaves the Company, Jobs Fail?

If this one has burned you, I apologize, but we'll quickly learn how to avoid it in the future. Databricks makes it very easy to convert a prototype notebook into an automated/scheduled job—almost too easy. When users create jobs via the UI, they often leave the default "Run As" set to themselves and think nothing of it. The job runs fine for months or years; then, one day, that user leaves the company. The next day, your team is greeted by failure alerts and emails about missing data!

This is very common, as organizations take measures to close accounts and access as a regular part of employee exit policy. Our Databricks jobs may fail when they run as a user and rely on that user's access to Unity Catalog or notebooks in their workspace directory (although default settings in Databricks do not delete workspace files after the user is deactivated).

The solution concept is not unique to Databricks as this is a common problem in any platform as a service: Service Principals.

Honestly, this is one of the more tedious resources to manage in Databricks. As a Databricks Account Administrator, we can create new service principals easily, assign them access to workspaces, and create OAuth credentials for Machine-to-Machine (M2M) authentication in our applications. However, to combine this with the lesson learned in #5, we must use the OAuth credentials for the Service Principal to impersonate it and call the Databricks REST API POST /api/2.0/git-credentials. This effectively impersonates the Service Principal and creates Git Credentials for it.

Note: Databricks, if you are reading this, please consider adding the ability to manage Git credentials for Service Principals. You can upvote my idea on the Databricks Ideas Portal here!

Once you get it set up (hopefully not often), Service Principals make your jobs much more reliable to changes in org structure or employees leaving the company, and it enhances the overall security within your Unity Catalog by not over-exposing write access to individual users--your coworker, Bob, is not the one writing to that gold table, it's the daily-customer-report Job that writes to it 😉.

Conclusion

Let's wrap up. Databricks Workflows are awesome--one of the platform's most mature offerings in my opinion--and with these tips we can maximize their resiliency and usability.

I implore you to consider these lessons the next time you create a new Databricks Workflow. If you have tips and tricks for Databricks yourself, please let me know in the comments!

Follow for more content like this!

Databricks released the File Arrival Trigger feature for its Jobs/Workflows this year. Essentially, you point the job at a cloud storage location, such as S3, ADLS, or GCS, which triggers whenever new files arrive.

This is very useful for creating event-driven jobs. Previously, accomplishing this would have required gluing together a lot of cloud-specific tooling, such as AWS S3 combined with S3 Events combined with a Lambda function that finally invokes the Databricks SDK / API / CLI to run the job. So, I see this new feature as a great simplification and a very practical one.

Let's tour a few use cases where you may want to use file arrival triggers.

1. Irregular Schedule

The first use case is even called out in the aforementioned Databricks docs:

You can use this feature when a scheduled job might be inefficient because new data arrives on an irregular schedule.

Say you receive files on S3 from a data vendor that collects and provides cross-reference data for your industry. The vendor does not have a regular cron schedule for delivering new files. There is usually one file per day, anywhere from 1 pm to 6 pm UTC, but because they push the files manually, there sometimes isn't a file received on holidays or weekends if they forget to push it or have backup staff.

• Naive Solution: Schedule a job to run every day from 1 p.m. to 6 p.m. (six runs per day). Write code in the job to look for new files and exit if none are found.

  • We have guaranteed at least five wasted runs every day. More on holidays and weekends when there are no files.

• File Arrival Solution: use file arrival triggers. There are no wasted runs, and the code focuses on the business logic.

2. Batch Submissions

We now work for a FinTech company, and the data team works closely with the front-end team. The front-end team wants users to be able to request an export of all their transactions for a given date range, then send an email to the user with file attachments.

We can build a solution using file arrival triggers and a clever manifest file. We created the following design.

When a user fills out the submission form on the web app, the front end generates a JSON manifest file that encapsulates their input:

{
	"userId": 123,
	"emailRecipients": [
		"john.doe@example.com"
	],
	"requestType": "EMAIL_REPORT",
	"parameters": {
		"dateRange": {
			"start": "2024-01-01",
			"end": "2024-09-01"
		}
	}
}

This file is dropped into an S3 bucket, which triggers a Databricks Job. Our code processes the request and emails the user. This solution can easily extend with additional parameters and request types as the application evolves.

3. User Content Moderation

Similar to #1, we may receive irregular data when end users generate it. Working at a media hosting company, we have images and videos uploaded by users sporadically. We are required to moderate and tag the content for anything out of compliance.

File arrival triggers will allow you to create a job focusing on content moderation (e.g., illicit image detection, age restrictions, copyright violation, etc.) as new images/videos are uploaded.

For an end-to-end solution for content moderation, the job may ingest the unstructured data to a Delta Lake table and store additional columns, such as the auto-moderation status and tags.

4. CCPA Requests

The California Consumer Privacy Act (CCPA) provides consumers with rights, such as requesting that their data be removed from companies' products or deleted altogether.

Similar to #2, a good solution is to use file arrival triggers to process manifest files encapsulating the CCPA requests. We can create a Databricks Job to process the file/request, updating tables and/or deleting from them where rows match the requested email address or PII.

Bonus Topic: We can facilitate this even better by using table and column tags in Unity Catalog to tag data containing PII. The information can then be retrieved programmatically by querying the information_schema.

Conclusion

We barely scratched the surface of what's possible with File Arrival Triggers. I encourage you to try them out and hope Databricks continues these strategic technical features in Workflows.

I love seeing solutions across industries--let me know in the comments if you have another use case for file arrival triggers!

This is Part 2 of the set of practical Terraform tips. If you missed Part 1, check it out here:

MakeWithData

Practical Terraform: You're Doing it Wrong

We've all written Terraform IaC that we're not proud of before--it happens. I'm here today to talk about the Terraform you write that you think you're proud of...until it outgrows your team, becomes hard to manage, and terrifies you anytime you terraform apply…

Read more

2 years ago · Zach King

We'll explore several more Terraform pitfalls and how to avoid them so your infra teams can succeed long-term!

1. Too many conditional resources

Have you ever seen a module that creates too many conditional resources using syntax like for_each and count and ternary operators? These are less readable and generate a lot of complexity for maintainers.

For example:

# file: modules/storage/main.tf

variable "create_s3_bucket" {
  type        = boolean
  description = "Whether to create an S3 bucket as well"
  default     = false
}

resource "aw_s3_bucket" "this" {
  count = var.create_s3_bucket ? 1 : 0
  ...
}

# Other storage resources... EFS volumes, Backup configuration, etc.
...

Used in moderation, this can be a fine way to add flexibility to your module. However, take it too far, and you'll have a Frankenstein of faux-array references like aws_s3_bucket.this[0] or worse: var.create_s3_bucket ? aws_s3_bucket.this[0] : "". It also makes your configuration more difficult to read and reason with.

When it becomes too much, decoupling these conditional components into a separate module is better. Instead of using ternary operators, you can instantiate a module or don't.

2. Resources that don't belong in a Module

We will only travel a short distance from the previous example. Let's say we have the above module with the conditionally-created S3 bucket. We may also have a few folders for deploying to Dev, Test, and Prod environments:

# file: deployments/dev/main.tf

module "storage" {
  source           = "../../modules/storage"
  create_s3_bucket = true
  env              = "dev"
}

# file: deployments/test/main.tf

module "storage" {
  source           = "../../modules/storage"
  create_s3_bucket = false
}

# file: deployments/prod/main.tf

module "storage" {
  source           = "../../modules/storage"
  create_s3_bucket = false
}

In this scenario, we create the S3 bucket only in one environment. This is typical of infrastructure resources unique to one environment's purposes, such as Developer sandboxes, QA tooling, and ad-hoc troubleshooting devices.

Using conditional resources with variables is the wrong way to solve this use case; if a resource is specific to one environment, you should only create the resource in that environment's deployment. Either of the following refactors would work in this example:

1. Create a new module named s3_storage and move the S3 bucket and related resources inside. Remove the variable and count syntax, then instantiate the module only during the dev deployment.

2. Don't use a module for the S3 bucket and related resources. Simply put them in the deployments/dev/ folder directly.

3. Sandbox Environments

This next tip is a godsend for parallelization and will also increase your confidence in your ability to spin up and tear down the infrastructure.

We often write Terraform with our environments in mind, like Dev, Test, Stage, Prod, etc. We create resources with the environment keyword in the resource name, such as naming a Lambda function sqs-ingestor-${var.env}-${var.region}, which creates sqs-ingestor-dev-us-east-1. This may be fine for your team, especially on a smaller scale and when you work by yourselves; however, what do you do when your colleague needs to test their version of the Lambda function while you're testing your feature branch?

This is where Terraform Workspaces come in very handy. Say the two developers are myself and Sara Hollis, working on different new features simultaneously. If we plan the Terraform with parallelized work-streams in mind, we can name the Lambda function including a Terraform built-in variable ${terraform.workspace}, which contains the name of the current workspace.

resource "aws_lambda_function" "sqs_ingestor" {
  name = "sqs-ingestor-${terraform.workspace}"

  tags = {
    Name = "sqs-ingestor-${terraform.workspace}"
  }
}

Then I proceed with my work:

zcking> git checkout feature/new-redundancy-options
zcking> terraform workspace new dev-zcking
zcking> terraform workspace select dev-zcking
zcking> terraform init && terraform apply

While Sara does the same for her work:

shollis> git checkout feature/json-schema-evolution
shollis> terraform workspace new dev-shollis
shollis> terraform workspace select dev-shollis
shollis> terraform init && terraform apply

Both will apply the resources successfully and in parallel--two Lambda functions will exist afterward. Terraform calls these workspaces, but I also like to refer to them as sandboxes because we each have our own isolated environment to play in, like a sandbox.

Note: This is usually feasible, but every use case is different. Sometimes, you may prefer to keep certain resources global or shared, such as ECS/EKS clusters, databases, and others—usually for cost and data reasons.

4. Remote States

My final tip for creating a more practical Terraform configuration is a data query to remote state files. This is where you programmatically query resource information stored in another remote Terraform state file, such as from another environment or team. By querying a remote state file, you can reuse outputs and modularize your project further without requiring the resources to be managed in the same deployable scope of Terraform code.

For example, we may deploy core networking infrastructure like our VPC with one state file:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "vpc/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_vpc" "main" { ... }

output "vpc_id" { 
  value = aws_vpc.main.id
}

Now in a separate Terraform project, we can query the remote state and access the output to deploy an EC2 instance into the VPC:

data "terraform_remote_state" "vpc" {
  backend = "s3"
  config = {
    bucket = "my-terraform-state"
    key    = "vpc/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t2.micro"
  vpc_security_group_ids = [aws_security_group.web_sg.id]
  subnet_id     = data.terraform_remote_state.vpc.outputs.vpc_id
}

You may wonder, "Couldn't I just use a data query to the aws_vpc resource rather than interrogate the Terraform state?" Technically, yes, and in your case, you may prefer that simplicity.

The tradeoff is that when using a traditional data query, you will need to include filters based on resource ID, tags, or other attributes; furthermore, some resources do not offer a data query to look up the infrastructure. Querying a remote state defers to the source of truth—the Terraform that deployed the dependency resources, like the VPC in this case.

Conclusion

I hope you enjoyed this expansion of my tips for practical Terraform-ing!

Follow for more content like this!

We've all written Terraform IaC that we're not proud of before--it happens. I'm here today to talk about the Terraform you write that you think you're proud of...until it outgrows your team, becomes hard to manage, and terrifies you anytime you terraform apply.

1. Monolithic Modules

Terraform modules are a set of reusable code, similar to a class in object-oriented programming. I like to say that the module is the "cookie cutter"; anytime you use it, you are "instantiating" it.

When you create a module, try to keep it short and sweet. Users of your module don't want to worry about a heap of baggage, like 50+ variables, 4+ providers, and 100+ direct resources, spat out in the first plan.

Take this module for a web application, for example:

webapp_bad
├── api_gateway.tf
├── cloudfront.tf
├── cloudwatch.tf
├── cognito.tf
├── iam.tf
├── lambda.tf
├── load_balancer.tf
├── outputs.tf
├── postgres.tf
├── providers.tf
├── variables.tf
└── vpc.tf

This is a typical stack for a web application on AWS, but I'll let your imagination fill in the myriad of resources. It's a lot. The mistake here is that we've created a monolithic module that only serves the purpose of 1 specific application stack.

A better approach would be to decouple these into smaller, more reusable units:

webapp_good/
├── backend/...
├── frontend/...
└── network/...

2. Useless Modules

Next on our pitfall hit list are the modules that simply aren't needed. If you find yourself writing a module with variables for almost every argument defined on the resources, ask yourself: Am I accomplishing something novel or simply repeating what's already easily handled by the direct resource(s) I'm creating? Likewise, we don't want a module containing only 1-2 resources, as that could be achieved easily on our own.

Remember, modules should accomplish one or more of the following tasks:

1. Automate standard configuration or resource best practices (e.g., tagging, security policies, naming conventions, etc.).

2. Organize configuration into reusable blocks of code, with some flexibility but not so much that we re-implement the individual resources.

3. Break down a complex infrastructure solution into smaller units that are easier to maintain.

For example, let's try not to write useless modules like this s3 bucket module:

# file: useless_s3/main.tf

variable "bucket" {
  type = string
}

variable "acl" {
  type = string
}

resource "aws_s3_bucket" "this" {
  bucket = var.bucket
  acl    = var.acl
}

# file: main.tf

module "useless_s3_bucket" {
  source = "./modules/useless-s3"
  
  bucket = "my-useless-bucket"
  acl    = "private"
}

This can be achieved without a module. The module here only adds another layer of nesting, and it will surely frustrate your team when they have to traverse the module tree just to add a simple argument to the underlying aws_s3_bucket resource.

3. Folder Layout & Blast Radius

Next, let's talk about file structure when writing Terraform. This is crucial when scaffolding your Infrastructure-as-Code configuration for a brand-new project. You can be a whiz at the HCL syntax and functions but still create inefficiencies and undue complexity if not laid out carefully.

Try to think about folders in terms of two types: modules and deployments. Modules are where you define a set of re-usable Terraform and should represent components of your infrastructure, like ingredients in a recipe. I typically refer to the folder where you terraform apply from as the deployment scope.

A common mistake is to create a monolithic module that defines the entire infrastructure solution but with variables, then create multiple deployments of this mono-module, such as a dev, test, and prod deployment:

├── deployments
│   ├── dev/
│   ├── test/
│   └── prod/
├── modules
│   ├── backend_app/
│   ├── ecs/
│   └── frontend_app/

This example could be better because it creates a huge blast radius for each of the deployments/environments. Any change within the frontend_app will still require terraform apply to re-scan configuration for resources in instance of the ecs/ module too.

Consider breaking your deployments into more than one deployable scope. Ask yourself the following questions:

1. How often will these resources need to be changed/redeployed?

2. How critical are these resources? Are they more or less critical than other groups of resources?

Critical resources such as the VPC and ECS/EKS clusters are great to segregate into their deployable scope/folder; those do not change very often and are vital to the operations of application services. This ensures a safer, leaner, more agile Terraform project structure like so:

├── deployments
│   ├── dev
│   │   ├── app //         <-- Instantiates backend_app/ and frontend_app/
│   │   └── ecs_cluster//  <-- Instantiates ecs/ and VPC resources
│   ├── prod
│   │   ├── app
│   │   └── ecs_cluster
│   └── test
│       ├── app
│       └── ecs_cluster
└── modules
    ├── backend_app
    ├── ecs
    └── frontend_app

4. Version Constraints

Terraform supports version constraints for the required_providers {} as well as the version of Terraform itself. We'll focus on providers for now. Let's examine how you might use version constraints with increasingly better examples.

Worst

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

The worst way to use version constraints is not to use them at all.

Bad

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.71.0"
    }
  }
}

I might spark some controversy by saying this, but yes, I think this is bad. Terraform providers like AWS, Azure, GCP, and others constantly release new versions, constantly adding new features and fixing bugs.

By pinning your provider to a specific version, you will likely find yourself combing through dozens of Terraform modules every now and then to update every single version constraint. There's no forgiveness here--all your Terraform must agree on the exact same version of the provider--making it a colossal pain to perform upgrades or receive new features/fixes.

Better

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.71.0"
    }
  }
}

This is better because it uses the pessimistic constraint operator ~>, which in this example will allow the last version number (the patch version) to change, but not the major or minor version numbers.

It allows updates only to patch versions of 5.71.x, meaning only versions that are >= 5.71.0 and < 5.72.0. Again, we will have to painstakingly update tons of these lines in our modules when we inevitably need to upgrade versions.

Best

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.71.0, < 6.0.0"
    }
  }
}

Actually, the best can be written with ~> as well:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.71"
    }
  }
}

This is great because it allows the minor and patch version numbers to slide forward but not the major version. Most reputable providers, such as AWS, follow semantic version (SemVer) standards very well, and they typically do not introduce breaking changes until a new major version is released.

It's your choice to use ~> or the more verbose >= x.y.z, < a.b.c but I've had people ask about the mysterious ~> syntax enough times that I favor simplicity--everyone looks at >= 5.71.0, < 6.0.0 and understands it quite easily.

By using this constraint we ensure that the configuration can gracefully receive new features/fixes (minor and patch version upgrades) without breaking changes. To upgrade the provider version within this constraint, simply run terraform init -upgrade.

Conclusion

Let's wrap up. Terraform is powerful, but as Uncle Ben famously said, it comes with great responsibility.

We can make our lives, and our coworkers' lives, a lot easier by applying the principles from this guide to the Terraform we write. Consider those pain points the next time you find yourself modifying old Terraform code or doing a deployment and dreading the large, scary Terraform plan. I hope you will think back to these tips and find them useful!

10/14/2024: Part 2 is out now! Get even more, better, practical Terraform tips with Part 2 here!

Follow for more content like this!

As a southerner, I love BBQ and smoking meats, and the Traeger is no muss no fuss. However, I'm also a nerd that loves doing things in my homelab. It started out early one morning after putting a beautiful pork butt on the smoker for some pulled pork. I went back in the house and was trying to think of something to do while it got started... and thus was born the idea to see what I could make with data from my Traeger Ironwood XL.

Today I'll show you how I was able to hack my Traeger grill to capture data such as the grill temperature, pellet level, probe goal, and more, as well as feed that data to a database and visualize with a Grafana dashboard all within my own homelab.

1. Information Gathering / Network Scanning

The first step in any hacking exercise is to gather as much information as possible to understand the target and what makes it tick. I'll break this down into 2 simple goals initially:

1. Identify the grill's IP address

2. Scan the grill for open ports and software versions

I was able to find the IP address of my grill easily by visiting my router, or gateway, web UI (commonly located at http://192.168.1.1) and viewing the attached devices, which listed the Traeger grill quite obviously: 192.168.1.19. Nice, moving on.

Next, I used nmap to scan for open ports and version info:

nmap -sV -p 1-9999 192.168.1.19

I didn't find anything open :(

Doing some searching about Traeger's data protocols online I found a customer success story on AWS about how Traeger migrated to use AWS IoT, which uses MQTT (Message Queuing Telemetry Transport).

2. Intercepting Data from the Grill

Knowing the grill likely uses AWS, it would explain why I couldn't find any open ports on it. I often use the Traeger mobile app to monitor and control my smoker while cooking, so the mobile app is most likely talking to a server in AWS, and so is the grill.

I wanted to test this theory using a Man-in-the-Middle (MITM) proxy. Essentially we just run a proxy to capture all HTTP(S) traffic and then configure our device (mobile phone or PC) to use the proxy. You could do this using mitmproxy on one machine and then configure the mobile phone to use the proxy.

I noticed there was also a desktop Traeger app for Mac OS so I opted to try that just so I could stay on one device for my testing. So I installed the Traeger app for Mac OS, fired up Wireshark and began capturing packets on my Mac's wireless network interface en0 . While that's running, I signed in on the Traeger app and clicked around a few times to give it enough changes to fire off requests, then stopped the packet capture in Wireshark. With some luck you can find DNS requests like these:

The app resolves the DNS record mobile-api.iot.traegergrills.io which is a CNAME record resolving to 1ywgyc65d1.execute-api.us-west-2.amazonaws.com. This appears to be an API gateway based on the format of the inner domain name.

With some more sniffing, I discovered the following pieces of information:

1. The app makes API calls to an API Gateway mobile-api.iot.traegergrills.io.

2. The API calls have a Bearer authentication token in the Authorization header.

3. Upon decoding that auth token with https://jwt.io it appeared to be generated by AWS Cognito because the token payload has "iss": "https://cognito-idp.us-west-2.amazonaws.com/us-west-2_..." in it.

4. There are API requests to endpoints such as /prod/mqtt-connections, confirming the app does use MQTT.

This all makes sense if the folks over at Traeger are using AWS and the AWS IoT service.

3. Calling the APIs

Now that we know how the app works, we should be able to write a script that emulates the app's behavior, and therefore access data from our grill programmatically.

Fortunately I found someone else has done this as well and written a Python script to integrate Traeger with HomeAssistant: https://github.com/sebirdman/traeger/blob/master/traeger.py

With some minor tweaks I was able to reuse this code and see data pretty quickly:

❯ python -i traeger.py
>>> t = traeger('MY_EMAIL@EXAMPLE.COM', 'CHANGEME_PASSWORD', requests)
>>> t.get_user_data()
{'userId': 'REDACTED', 'givenName': 'Zach', 'familyName': 'King', 'fullName': 'Zach King', 'email': 'REDACTED', 'username': 'REDACTED', 'cognito': 'REDACTED', 'urbanAirshipId': 'REDACTED', 'teams': [{'teamId': 'REDACTED', 'teamName': 'REDACTED', 'thingName': 'REDACTED', 'userId': '3d8477eb-0605-4f47-a8fc-44580b9de96d'}], 'things': [{'thingName': 'REDACTED', 'friendlyName': 'Sir-Ribs-A-Lot', 'deviceTypeId': '2205', 'userId': '3d8477eb-0605-4f47-a8fc-44580b9de96d', 'status': 'CONFIRMED', 'productId': '53059272-02e0-4c00-914e-83f0e65edb16'}]}
>>>

Then subscribe to updates from the MQTT broker:

>>> t.refresh_mqtt_url()
>>> t.subscribe_to_grill_status()
>>> t.grill_status
{'REDACTED': {'thingName': 'REDACTED', 'jobs': [{'jobId': 'Yos_Combined_IronwoodXL_REDACTED', 'thingName': 'REDACTED', 'currentStatus': {'status': 'REJECTED', 'timestamp': 1710372539}, 'isFinished': True, 'isInProgress': False}], 'status': {'acc': [], 'ambient': 99, 'connected': False, 'cook_id': '', 'cook_timer_complete': 0, 'cook_timer_end': 0, 'cook_timer_start': 0, 'current_cycle': 0, 'current_step': 0, 'errors': 0, 'grease_level': 0, 'grease_temperature': 0, 'grill': 147, 'grill_mode': 0, 'in_custom': 0, 'keepwarm': 0, 'pellet_level': 25, 'real_time': 0, 'seasoned': 1, 'server_status': 1, 'set': 325, 'smoke': 0, 'sys_timer_complete': 1, 'sys_timer_end': 0, 'sys_timer_start': 0, 'system_status': 99, 'time': 1728263173, 'ui': {'ambient_light': 0, 'screen_brightness': 0}, 'units': 1, 'uuid': 'FC0FE708F7DF', 'probe_con': 0, 'probe': 0, 'probe_set': 0, 'probe_alarm_fired': 0}, 'features': {'cold_smoke_enabled': 0, 'flame_sensor_enabled': 1, 'grease_sensor_enabled': 0, 'grill_light_enabled': 1, 'grill_mode_enabled': 0, 'lid_sensor_enabled': 1, 'limits': {'max_grill_temp': 500}, 'open_loop_mode_enabled': 0, 'pellet_sensor_connected': 1, 'pellet_sensor_enabled': 1, 'pizza_mode_enabled': 0, 'super_smoke_enabled': 1, 'ui': {'ui_type': 0}}, 'limits': {'max_grill_temp': 0}, 'settings': {'config_version': '2205.001', 'device_type_id': 2205, 'feature': 0, 'fw_build_num': 'c0321c9d-20230221_113102', 'fw_version': '01.03.21', 'language': 0, 'networking_fw_version': '1.4.2', 'rssi': -68, 'speaker': 1, 'ssid': 'REDACTED', 'ui_fw_build_num': 'e4ea0a2-20220715_205356', 'ui_fw_version': '01.03.12', 'units': 1}, 'usage': {'ac_ignitor': 10740, 'auger': 231583, 'cook_cycles': 18, 'dc_ignitor': 0, 'error_stats': {'auger_disco': 0, 'auger_ovrcur': 0, 'bad_thermocouple': 0, 'fan_disco': 0, 'ign_ac_disco': 0, 'ign_dc_disco': 0, 'ignite_fail': 0, 'low_ambient': 0, 'lowtemp': 0, 'overheat': 0}, 'fan': 329530, 'grease_trap_clean_countdown': 0, 'grill_clean_countdown': 0, 'hotrod': 0, 'light': 8494, 'runtime': 329711, 'time': 0, 'ui': {'screen_on': 0}}, 'custom_cook': {'cook_cycles': [{'slot_num': 4, 'populated': 0}]}, 'details': {'thingName': 'REDACTED', 'userId': '3d8477eb-0605-4f47-a8fc-44580b9de96d', 'lastConnectedOn': 1728240177, 'thingNameLower': 'fc0fe708f7df', 'friendlyName': 'Sir-Ribs-A-Lot', 'deviceType': '2205'}, 'stateIndex': 33112, 'schemaVersion': '2.0'}}
>>>

4. Sending the Data to Elasticsearch

Now we need to persist the data so we can later do some visualizations in Grafana. I chose Elasticsearch so let's modify the def __init__() constructor to initialize a connection to the database:

def __init__(self, username, password, request_library, es_host): 
	# Initialize Elasticsearch connection 
	self.es_client = Elasticsearch([es_host])
	...

Then once the MQTT subscription is made, any time events come in from the grill, the grill_message(…) method will be invoked. So that's our hook we need to write to the database from:

def grill_message(self, client, userdata, message): 
	if message.topic.startswith("prod/thing/update/"): 
		grill_id = message.topic[len("prod/thing/update/"):] 
		
		# Parse the grill status from the message 
		grill_data = json.loads(message.payload) 
		self.grill_status[grill_id] = grill_data 
		
		# Add a timestamp for Elasticsearch (Grafana uses this for time-series queries) 
		grill_data['timestamp'] = datetime.utcnow().isoformat() 
		
		# Write the parsed data to Elasticsearch 
		try: 
			self.es_client.index(index='grill_status', id=grill_id, body=grill_data) 
			logging.info(f"Grill data for {grill_id} written to Elasticsearch successfully.") 
		except Exception as e: 
			logging.error(f"Failed to write grill data to Elasticsearch: {e}") 
		
		# Trigger any callbacks registered for this grill ID 
		if grill_id in self.grill_callbacks: 
			for callback in self.grill_callbacks[grill_id]: 
				callback()

5. Visualizing the Data

Finally, we can use our data from Elasticsearch as a data source in Grafana and create a dashboard to put it all together.

I haven't gotten to this quite yet, it's on my TODO list while migrating part of my homelab to a Kubernetes cluster running on mini-PCs (follow my newsletter and let me know if you're interested in a series on building that homelab!). However, the dashboard should contain some graphs such as grill temperature, probe 1 temperature, and probe 2 temperature. It should also contain a gauge indicating the remaining pellet level from 0-100%, and a few single stat indicators for Super Smoke mode, probe goals, and more depending on the specific grill features.

The following is a basic example to illustrate the idea (credit: user/natty_patty from the r/homelab subreddit):

Conclusion

This was a fun dive into IoT apps and learning how yet another device around the house works. You may be surprised what you find simply by scanning your network and investigating API calls.

I hope you found this useful or at least entertaining. Please consider sharing and following my newsletter if you like this content as it motivates me to write more! Let me know in the comments if you have an interesting IoT data story, what challenges did you face?

This guide covers the essential skills, tools, and technologies for becoming an effective data engineer. Whether you're just starting or looking to deepen your expertise, this is your roadmap to modern data engineering.

Programming Languages

SQL

Python and SQL are must-haves for both aspiring and experienced data engineers. SQL, especially in its ANSI-compliant forms, is widely used due to its simplicity and expressiveness, making it a fundamental skill.

Python

Python remains the top choice in the data and AI world. Its simple syntax and powerful built-in features make it ideal for scripting and data manipulation. With a vast array of libraries (e.g. pandas, spark, scikit-learn, pytorch) for data analysis, machine learning, and big data, Python continues to lead the pack in data engineering and AI.

Scala (Bonus)

Scala is an elegant cross between functional and object-oriented programming languages, and runs on the Java Virtual Machine (JVM). Great for robust and large enterprise projects, and still a major footprint in big data frameworks like Apache Spark, don’t sleep on this programming language.

Key Frameworks and Tools

Apache Spark

First and most importantly as far as frameworks go there is Apache Spark. Spark is an open-source analytics and big data processing engine that leverages a distributed computing architecture.

Spark supports Python, Scala, SQL, and R programming languages. You can deploy Spark applications on a variety of cloud computing services such as Amazon EMR, or deploy it yourself with its native Kubernetes support. Spark clusters can be anywhere from 1 to thousands of nodes, and can process data up to petabytes in scale.

>>> from pyspark.sql.functions import col, when

>>> df1 = df.withColumn(
    "life_stage",
    when(col("age") < 13, "child")
    .when(col("age").between(13, 19), "teenager")
    .otherwise("adult"),
)

>>> df1.groupBy("life_stage").avg().show()
+----------+--------+
|life_stage|avg(age)|
+----------+--------+
|     adult|    53.5|
|     child|     3.0|
|  teenager|    13.0|
+----------+--------+

Note: PySpark also has strong interoperability with pandas, arrow, and koalas dataframes, making it a powerhouse for data science and machine learning with big data.

PyTorch

PyTorch is a leading deep learning framework, ideal for both research and production environments. PyTorch is extremely relevant for ML and data engineers working on AI-driven data pipelines, model training, and model inference.

Its support for dynamic computation graphs allows for efficient processing of complex data models, while libraries like TorchServe simplify deploying ML models. Don’t think ML frameworks like PyTorch are only for data scientists and ML engineers; data engineers will stand apart from the crowd when equipped with even a basic understanding of ML concepts and popular frameworks like PyTorch.

Start learning PyTorch today with "Learn PyTorch for Deep Learning: Zero to Mastery" free online book

Streams: Kafka, Kinesis

Many business use cases demand data and updates real-time. Streaming tools like Apache Kafka and Amazon Kinesis are essential tools for modern data engineering. Kafka is widely used for its ability to handle high-throughput, real-time data streams in distributed systems—especially with Kafka being open-source and cloud agnostic.

Amazon Kinesis is a fully managed streaming service, with similar capabilities plus integrations into the AWS ecosystem. Both platforms are invaluable for applications requiring low-latency data processing and event-driven applications, such as IoT, financial systems, and fraud detection, positioning them as critical tools for data engineers navigating real-time data challenges.

Note: while streaming engines like Kafka aren’t going away anytime soon, they are often selected too hastily as the required tool for the job. With modern data formats (e.g. Delta Lake) and query engines (e.g. Spark), I would encourage you not to underestimate the capability of simpler micro-batch processing too.

Delta Lake

Delta Lake is an open-source storage layer that brings reliability and performance to data lakes. It allows data engineers to build scalable and robust pipelines by adding ACID transactions and schema enforcement to the flexibility of data lakes.

Delta Lake enables incremental data processing with time-travel capabilities, making it easy to track changes and roll back to previous versions of data. It also optimizes data lakes with features like compaction, z-order indexing, and liquid clustering, ensuring high performance for queries.

As data lakes grow in size and complexity, Delta Lake is essential for ensuring consistency, reliability, and efficiency in managing large-scale datasets.

Through the Delta Kernel project you can now use Delta Lake through a wide variety of programming languages and tools, such as Rust, Go, Python, Flink, Power BI, and many more, making it an even more versatile engine even in smaller datasets.

Databricks

Remember all the previous tools, frameworks, and languages we just covered? If you could take all of those and so much more, bundle it up into a single unified platform, that platform would be Databricks.

Databricks truly does unify the experience and solutions for data engineering, data analysis, data science, ML engineer / MLOps, and BI and reporting. Founded by original creators of Apache Spark, Databricks earned its respect initially as a highly efficient proprietary runtime for Spark, and made notebooks and jobs simpler and more collaborative than ever. Since then Databricks has continued to add new products to the platform including several favorites such as:

• Notebooks with support for Python, Scala, SQL, and R.

• Jobs / Workflows with advanced orchestration features.

• SQL Dashboards.

• Machine Learning model training, evaluation, logging, and model serving.

• Gen AI development, evaluation, serving, and guardrails.

• Data Governance through Unity Catalog with access control, row-level and column-level security, data lineage, and attribute-based access control (ABAC).

• Serverless jobs, SQL, and model serving

Databricks is multi-cloud, supporting AWS, GCP, and partnered with Microsoft Azure. Note: some features may be more available on AWS and Azure than GCP.

Conclusion

There are many more tools I couldn’t list for risk of making this too long; the list above are some of my top picks, especially in 2024/2025.

Several other worthy mentions:

• Apache Airflow

• Iceberg (similar to Delta Lake)

• DLT

• Apache Airflow

• DuckDB

• MLFlow

• Terraform

• Kubernetes

Thank you for reading and if you liked this content please consider subscribing and sharing the post with friends and colleagues!

In case you haven’t been paying attention the last few years, there is a war being waged… a war of which open table data format will come out on top: Delta Lake, Iceberg, and Hudi. These data formats each offer similar and unique features to manage and scale big data lakes, and so the choice of which format to use has been a hot topic among the data engineering community for quite some time.

Organizations looking to maximize the value and scalability of their data lakes are really the ones the draw the short straw. No one wants to be torn between multiple tools, but companies also want to ensure the choice they make doesn’t become obsolete or turn out to be wrong, only to have to perform a complex migration. Fortunately, the table format wars can finally resolve in peace thanks to UniForm by Databricks.

Background: Iceberg vs. Delta Lake

Delta Lake and Iceberg have been the dominate formats in this field for a while, so we’ll focus on these:

Delta Lake

• Developed by Databricks, open-sourced in April, 2019.

• Offers ACID transactions for data integrity and consistency guarantees.

• Maintains metadata to support scalable operations and versioning, distinct from data files which use the Parquet format.

• Versions data and enables time travel queries to view historical data easily.

• Downloaded >20M times per month as of the time of this writing.

• Delta Kernel is a set of libraries for the core Delta Lake logic, allowing users to operate on Delta Lake tables from almost any language or engine (Java, Python, C++, Rust, Spark, Trino, Pandas, Polars, DuckDB, etc.) without re-writing the core behaviors.

• Project home page: https://delta.io/

Iceberg

• Developed by Netflix, donated to Apache Software Foundation in 2018.

• Offers ACID transactions for data integrity and consistency guarantees.

• Also maintains metadata and data files separately, enabling scalable operations on large tables.

• Versions data and enables time travel to view historical data. Also borrows popular concepts from version control (e.g. Git) such as cherry-picking, branches/tags, and merges.

• Supports a wide range of engines (Spark, Trino, Flink, Amazon Athena, etc.) and maintains these implementations within the main repository of Iceberg itself except for a few exceptions such as Trino and Presto. See https://iceberg.apache.org/multi-engine-support/

• Project home page: https://iceberg.apache.org/

For years these formats have competed for dominance in the data lake ecosystem. Businesses often had to choose between them, hedging their bets against a particular format and set of languages and engines deemed better than the rest (or at least lower effort depending on the organization’s existing tech stack). This division created challenges in terms of interoperability, data governance, subject matter expertise, and overall data lake management.

UniForm, our savior 🙌

Introduce UniForm (Universal Format), a feature offered by Databricks to bring peace. UniForm aims to unify the interoperability of the Delta Lake, Apache Iceberg, and Apache Hudi data formats by capitalizing on the fact that all 3 have the same fundamental trait: parquet data files combined with a format-specific metadata. Because the metadata is the only real difference, UniForm compromises the format wars by allowing users to automatically and asynchronously generate the metadata for all of the data formats (Delta, Iceberg, Hudi), or only the ones you wish to enable in the event that you only care about 2 out of the 3.

It’s simple, when creating a table in Databricks you just set a property to enable the formats you want metadata for:

CREATE TABLE main.sales.skus (sku_name STRING, inserted_at TIMESTAMP)
TBLPROPERTIES('delta.universalFormat.enabledFormats' = 'iceberg,hudi');

Now, for engines that may only support Iceberg and not Delta Lake for example, you can access this table as if it were just an Iceberg table (and vice versa). You can also begin using UniForm on existing tables, the property does not have to be set at the time of creation:

ALTER TABLE table_name SET TBLPROPERTIES ('delta.universalFormat.enabledFormats' = 'iceberg');

Databricks acquires Tabular

As a matter of fact, this is old news–Databricks began offering UniForm in 2023? So what’s all the buzz, aren’t we at peace already? Well no, not exactly. These changes take time, both from the technology and community perspective. From a community perspective, this new feature takes time to recognize and adopt, even though it is incredibly easy to get started with. From a technology perspective, it is very challenging and time-consuming for a single project (Delta Lake) to now try to reconcile all the ongoing changes and features from the other two formats, which in niche cases may create a parity gap where in fact it feels like your faux Iceberg table does not integrate successfully.

On June 4, 2024, Databricks announced it had agreed to acquire Tabular. Tabular is a data management company founded by Ryan Blue, Daniel Weeks, and Jason Reid, among who were also the original creators of Apache Iceberg at Netflix. According to Databricks in the press release:

Databricks intends to work closely with the Delta Lake and Iceberg communities to bring format compatibility to the lakehouse; in the short term, inside Delta Lake UniForm and in the long term, by evolving toward a single, open, and common standard of interoperability. Databricks and Tabular will work together towards a joint vision of the open lakehouse.

Credit: Databricks Press Release

Furthermore, Databricks CEO, Ali Ghodsi, along with Ryan Blue, shares at the 2024 Data + AI Summit keynote that this strategic decision is intended specifically to combine their technical subject matter expertise between the two leading formats, Delta Lake and Iceberg, such that we can expect the unilateral development within UniForm to become more and more ubiquitous.

For this, these data technology giants have my respect as I believe the long-term value is openness and maximum flexibility for all. What a breath of fresh air after countless hours of the normal one-sided marketing pitches that lead to vendor lock-in.

What is FineWeb?

FineWeb is a large-scale web corpus dataset created by Hugging Face to train state-of-the-art Large Language Models (LLMs). Of course FineWeb is massive: it contains over 15 trillion tokens of cleaned and deduplicated English web data from the CommonCrawl project.

According to Hugging Face, their dataset “was originally meant to be a fully open replication of  RefinedWeb, with a release of the full dataset under the ODC-By 1.0 license. However, by carefully adding additional filtering steps,[they] managed to push the performance of FineWeb well above that of the original RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama) on our aggregate group of benchmark tasks.”[1]

FineWeb vs. RefinedWeb vs. The Pile

• Size and Scale: FineWeb contains over 15 trillion tokens and has a downloadable size of 45 TB, whereas RefinedWeb only contains about 5 trillion tokens / 1.68 TB, and The Pile with about 1.35 trillion tokens / 886 GB.

• Data Cleaning: the team at Hugging Face focused on high quality web pages by using advanced filtering algorithms to remove spam, duplicate data, and low-quality pages.

  • PII Formatting is also used to anonymize email addresses and public IP addresses scraped from web pages.

Data Sources

So it comes from the Internet but how did HuggingFace obtain the data exactly?

• Dynamic Web Scraping: FineWeb uses more sophisticated web crawling techniques that prioritize higher quality content such as authoritative sources, recent content, and filtering out harmful content.

• Expert Vetting: the dataset was ran on the datatrove library and went through a process of expert vetting and analysis to determine the optimal deduplication approach, rather than just applying a single deduplication method across the entire dataset. 

• Community Feedback Loop: The creators at Hugging Face have indicated they intend to iteratively improve the dataset over time, through surveys, polls, open-ended discussions, and direct interactions

What does FineWeb contain?

Let’s take a look at the general makeup of FineWeb’s dataset.

• The dataset is based on internet crawls between the Summer of 2013 and the Winter of 2024.[2]

• It was created by processing and distilling 38,000 TB of CommonCrawl dumps into a 45 TB dataset ready for language model training. [2]

• The dataset contains multiple configurations allowing you to load the default or a more specific dump/crawl such as CC-MAIN-2023-50. See here for the table of dumps/crawls you can selectively load.

• Contains a variety of text from academic papers, books, news articles, blogs, forums, and more, aiming for a holistic representation of language use.

  • Also includes structured data types like tables and lists to train models on data extraction and interpretation tasks.

Use Cases & Applications

One of the things I love about FineWeb is the considerations taken to try to anonymize Personally Identifiable Information (PII) and avoid toxicity or biased content. We can certainly debate whether this is the dataset’s responsibility to solve. I like having the flexibility to know this data has received that treatment, and ultimately it gives the community another fantastic corpus to train with in the NLP and LLM applications.

FineWeb covers a very wide range of topics and styles, with high quality. One use case we may begin to see is FineWeb used to improve benchmarking and evaluation of language models, adding more comprehensive representation to existing benchmarks.

If you are considering FineWeb for coding tasks, please note the known limitation shared on the README:

As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use FineWeb with a code dataset, such as The Stack v2.

[3] Source

I’m excited to see what new use cases and advancements this will bring to the GenAI community; in fact dozens of open source models have already been registered having been trained on FineWeb: https://huggingface.co/models?dataset=dataset:HuggingFaceFW/fineweb

The post 🍷FineWeb: the new Pile 🤔 appeared first on MakeWithData.

Whether you’re a cloud veteran on Azure, AWS, GCP, or just getting started on your cloud computing and solution architecture journey, official certifications are still a great way to learn and improve your skill set.

I actually often hear questions like “are certifications even still useful or required to get a job?” Honestly, your mileage will vary just like it does in today’s debate about college degrees being a prerequisite to technical jobs — it all depends on the employer and the individual. If you find yourself turned down purely based on the lack of one of these artifacts alone, that’s probably a red flag that you’re in the wrong place to begin with (portfolios and prior experience are often just as important, if not more).

However, I’m a firm believer that certifications are an excellent form of curated learning and development no matter what your opinion of standardized testing is.

I recently acquired my first Azure certification: Azure Administrator Associate (AZ-104), after less than 2 weeks of preparation. I have 7 years of experience with AWS and a focus on data engineering, DevOps, and solution architecture; I wanted to broaden my overall cloud expertise by learning Azure, the 2nd-most leading cloud provider close behind AWS. My goal: to learn what a Solution Architect on Azure needs to know, or acquire the AZ-305 Azure Solutions Architect Expert credential. Before reaching the AZ-305 though, there’s a prerequisite exam required by Microsoft, the AZ-104: Azure Administrator Associate, so that’s where my story begins.

Study Materials and Resources

I didn’t exactly plan to take the exam with only two weeks of preparation, but I’m comfortable taking tests, have a decent memory, and let’s face it — pretty impatient. So what did I study in those 14 days?

Mostly two things:

1. John Savill’s study cram video for AZ-104

2. Microsoft Learn website’s learning paths listed on the exam page

I also took the practice assessment on the AZ-104 exam page linked above; I did this once before I did ANY studying, to get a baseline measurement of my knowledge based on context clues and similar knowledge from AWS. I took the practice exam a few more times after all my studying, until I scored a 98% on it and felt confident.

Note: the practice assessment on Microsoft Learn does repeat the same questions a lot after 2 or 3 attempts so keep this in mind.

Study Cram Video

John Savill is a very well-known instructor for Azure and you’ll quickly see his name pop up in search results and Reddit threads when looking for study material. What I love is his study cram, a VERY long 4-hour video covering a high-level of every Azure service and topic in the exam. Unfortunately for me, he released a V2 of the AZ-104 video just a few days before my exam was scheduled, but is a great testament to his commitment to stay up-to-date.

Note: if you’re looking for a similar instructor for AWS, check out Stephane Maarek on Udemy. He is excellent!

Remember when I said I was impatient? Well, when watching this 4-hour long video, I actually set the speed to 2x and enabled closed captions. I often do this on learning videos simply because reading is faster than listening, and I find that instructional videos are generally more slowly spoken anyway. You may prefer a slower speed like 1.5x or even a faster speed, but I highly recommend this if you’re able to digest and retain the information.

Microsoft Learn

John’s videos are great, but there’s no way to cover hands-on experience in a format like that (it’d be longer than even 4 hours…), so I also tried to complete all the Microsoft Learn paths listed right there on the AZ-104 exam page. Plus c’mon, it’d be a discourtesy to ignore all that free material from the very platform that you’re learning about.

As of this writing, there are 5 learning modules listed with this exam, covering topics like compute resources, virtual networks, storage, and identities and governance. Each learning path has several Modules, and each Module contains several Units. I’m not sure how Microsoft estimates this, but when viewing a Learning Path you’ll see an estimated time — usually several hours for a single Learning Path. Personally I was able to complete these much quicker, usually about a couple hours per Learning Path. Most of the content is shear reading, then some hands-on labs, and finally a very short knowledge check (multiple choice questions).

Some contain simulated Labs, which are just a video-like experience that you can click through without spinning up resources in your subscription; these are great because it lets you see the Azure portal UI as you walkthrough a scenario but you can still click through at your own pace. Real hands-on labs require a Sandbox subscription, which you can get up to 10 per day, and are the real deal as you literally spin up resources in Azure (don’t worry, the Sandbox costs you nothing, just stick to the script as far as what you’re supposed to be deploying), but these take the longest amount of time so be practical and complete these if it’s a topic you really feel you need the extra experience on.

This may sound silly but my greatest challenge was honestly just staying awake while doing the Microsoft Learn content. Reading instantly makes me sleepy, and I need to re-read paragraphs a few times when my attention span gets cut short. I found myself usually working through about 1 Learning Path per day, but sometimes less and it’d take multiple days, and usually right before going to bed so as not to disrupt my sleep schedule too much when I inevitably got drowsy from reading.

I want to emphasize that my rapid pace method is what works for me, and you should NOT try to “game the system”, taking these exams simply as early as you can. The point is not just to tack on another badge to your resume or profile, and nothing can replace real hands-on experience. So in addition to the above study methods, I also spent a little time doing hands-on things in my own Azure subscription like creating storage accounts, trying out Azure Files, creating an Azure Functions App that automatically cleans up Resource Groups based on a tag, using RBAC to allow that very Function App permission to do its behaviors, etc.

Exam Experience

I scheduled my exam for a Friday very early in the morning at 7am (in hindsight, this was a terrible idea). Honestly I wanted to get it over with before the weekend, but early enough so it didn’t interfere with me getting to work that day. I took the exam at home/online, proctored by Pearson Vue (Microsoft’s choice of testing provider at the time of this writing).

The night before I made sure to prepare my room where I’d be taking the exam — clearing off my desk, covering up any bookshelves, removing any pictures with words on them, etc. This may sound overkill, but in my testing experience it really depends on the proctor. Make sure to read the exam guidelines and room requirements in advance before taking your tests.

Pro-tip: you can begin your check-in to Pearson 30 minutes before your appointed exam time, and you should always take advantage of that. It almost always comes in handy for dealing with technical difficulties or minor requests during the check-in process such as forgetting to unplug your monitors, etc. Also, make sure to close out of every other application beforehand so it doesn’t interfere with the testing software or get flagged.

For obvious reasons I will not be sharing details about any questions from the exam. However, once I got into my exam, it let me know how many questions I would have (e.g. multiple choice, case study, etc.) just as Microsoft says on the Learn website. I went through each question carefully. On questions I was unsure of, I clicked the checkbox to mark it for review so I could come back to it at the end if there was time.

When I got to the end, that’s where it gives you the chance to go back and review applicable questions, so I used my extra time to double back on some. Note: Microsoft now allows you to access the Microsoft Learn website during the exam so you can lookup API references and documentation, which I found really useful for questions about specific things you’d have trouble memorizing.

My only piece of feedback to Microsoft for the exam, was the false sense of “completion” I got when I answered all the questions from the first “section.” Your exam might have different sections, like a multiple choice section followed by a case study, for example. In my case, I was faced with the screen showing me the questions I could go back and review, so I thought I was at the end… it was only when I was about to submit everything at the end that I clicked “Next” and realized there was a whole other section waiting for me 🤦‍♂️ I got lucky and scored high enough on the rest of the exam that this didn’t matter, but please learn from my mistake! And hopefully Microsoft, or Pearson, makes the delineation between section ends vs the exam end better in the future.

Conclusion

Everyone learns in their own way and at their own pace. I hope my story is helpful to you reading this now and inspires you to take that next step in your learning and development journey.

Have you received an Azure or AWS certification recently? Comment below what your experience was like, or if you have questions about what to learn next.

It’s 2024 and every tech company is wanting to do more, with less, and that means reducing AWS costs. Review this simple checklist to find quick wins that save AWS costs on your next bill.

Also, with industries everywhere taking up GenAI initiatives to find the next way to add value and distinguish themselves in their market, the total cost of ownership for cloud infrastructure is becoming even more important.

1. Use gp3 EBS Volumes vs gp2

If you use EC2 instances at all, you’re definitely using EBS volumes as well, even if it’s just for the required root ( / ) volume. An often-overlooked saving is upgrading from gp2 to gp3 generation EBS volumes, mainly due to their pricing structure and performance flexibility.

Here’s why you should switch to gp3 if you haven’t already:

• Lower costs per GB. Up to 20% lower, compared to gp2.

• Generally more performant all around — offering the base performance of 3,000 IOPS and 125 MiB/s.

• Flexibility to scale IOPS (Input/Output Operations Per Second) and Throughput independent of the storage capacity. This means scaling up to meet disk performance demands is more cost-effective than gp2, which requires provisioning larger volumes.

Don’t take my word for it, check out Amazon’s blog on gp2 vs. gp3 volumes here: https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-gp2-to-gp3-and-save-up-to-20-on-costs/

2. Use VPC Endpoint for AWS S3 Costs

Do you know how your requests are reaching Amazon’s APIs when using S3? Ever wonder why your NAT costs are so high each month?

By default, network traffic to access Amazon S3 from a VPC is routed over public Internet. This typically uses a NAT (Network Address Translation) gateway, which is charged per GB processed. Your VPC is in AWS, and so is S3, so wouldn’t it nice if we could keep that traffic on AWS’s backbone… yes!

You can create a VPC Endpoint for S3, which requires zero additional configuration to your applications or workloads, and ensures S3 requests are routed directly from your VPC to S3. This direct connection can often reduce your NAT costs by 50–70%, depending on data volume and which region you use, but also improves network performance.

Again, don’t take my word for it, read Amazon’s blog and instructions here: https://aws.amazon.com/blogs/architecture/reduce-cost-and-increase-security-with-amazon-vpc-endpoints/

Note: you can create these endpoints for certain other AWS services, such as DynamoDB, Kinesis, Lambda and more, but S3 is the most beneficial for data-intensive organizations. Please also note the difference between “Gateway” and “Interface” type endpoints.

3. Watch out for S3 object versioning

Accidents happen — we’ve all been there when something gets accidentally deleted and panic ensues. That’s why we create backups and build BCDR (Business Continuity / Disaster Recovery) plans.

If you use S3, you may have given yourself added protection to accidental deletion by enabling S3 Versioning which keeps versions, or copies, of every object in a bucket and allows recovery to a previous version.

However, the risk you run with this feature is letting additional versions accumulate out of control, especially if applications overwrite the same objects frequently such as in data lakes and Lakehouse architectures. The extra versions of your objects are referred to as “noncurrent” objects because they’re not the active copy of the object. Add a lifecycle rule to your buckets to ensure “noncurrent” objects are expired after X days; you can make more complex lifecycle policies as well if you need to retain at least 1 noncurrent copy.

See the user guide for more information about deleting object versions: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html

4. S3 Lifecycle, Storage Classes, and Analytics

Amazon provides a few great ways to analyze the volume and trends for your S3 buckets: Amazon S3 Storage Analytics Insights which includes S3 Storage Lens and S3 Inventory features.

S3 Storage Lens is useful for identifying your largest buckets by object count, physical size, noncurrent versions, and how these metrics trend over time. For example, when you implement #3 from this checklist, use S3 Storage Lens to view the percentage of your S3 storage occupied by noncurrent versions. When you implement lifecycle rules, Lens is helpful to validate those changes (in addition to seeing 🤑 in Cost Explorer).

Another useful way to look at your data in Lens is by storage class. There are other lifecycle rules you may be able to define on S3 buckets to get the best price-to-performance, in addition to the default S3 Standard. For infrequently accessed data, S3-Infrequent Access (S3-IA) and S3 One Zone-IA are suitable classes, with One Zone being ideal when high availability isn’t necessary, leading to a cost reduction of up to 40%. For cold storage access very rarely, like backups and archives, use one of the S3 Glacier storage classes providing up to 95% in storage cost saving depending on data volume and specific Glacier tier.

If your S3 access patterns are predictable and designed with enough separation of buckets and/or path prefixes, you can save tons of money by defining lifecycle rules on buckets to transition objects from hotter storage classes (e.g. S3 Standard) to cooler storage classes when appropriate.

Alternatively, if your access patterns are less predictable or changing, you may benefit from S3 Intelligent-Tiering. With this feature, you pay a small monthly object monitoring and automation charge, in exchange for Amazon automatically monitoring access patterns and moving objects to a lower-cost access tier when they haven’t been accessed enough.

5. AWS Costs: Everything else …

There are honestly several ways to keep costs down, and I’d rather you walk away from this equipped with as many tactics as possible. So if you read this far, please accept this greater list of cost saving optimizations:

1. Use Spot Instances for low-impact workloads that don’t require high availability, and save up to 90% compared to On-Demand prices.

2. Use Savings Plans to save up to 72% on compute by committing to one- or three-year terms of usage.

3. Routinely check AWS Compute Optimizer for recommendations and right-size your instances to fit the workload.

4. Avoid using CloudWatch for custom metrics… you’re better off with open stacks like Prometheus + Grafana, or even a managed SaaS for observability.

5. Use Auto-Scaling Groups (ASGs) to reduce overpaying for idle resources.

6. Prefer Container based services like ECS or EKS over individual host/VM-based deployments. Containers make it easier to ensure full utilization of resources so you get what you’re paying for.

7. Add retention policies on your CloudWatch Log Groups. Don’t accidentally leave these with the default retention, Never Expiring!

8. Search for unused EBS volumes, or volumes in the “Available” state.

9. Consider putting a AWS CloudFront distribution in front of your S3 bucket to provide caching and reduce data transfer costs or cross-region movement.

10. Remember that DynamoDB has an Infrequent-Access table class, similar to S3 Infrequent-Access (S3-IA).

Before you go…

I hope you find these tips useful for keeping your AWS costs lean and it enables you or your organization to do even more!

• Have a cost saving tip of your own that I missed? Leave a comment and I’ll try it out!

• Follow this newsletter and my LinkedIn for more tech content, or on YouTube @MakeWithData!