Spinifex 1.19.0: Nineteen Releases Later

After five months and nineteen releases, we have shipped Spinifex 1.19.0. In this time we have taken the platform from supporting three AWS services into eleven, and turned a claim about production readiness into real production-tested code. When we released 1.0.0 at the end of March, we only supported EC2, EBS and S3. You could launch an instance, attach a volume, and put an object. There were no load balancers, no Kubernetes, no containers and no databases. Since then we have expanded significantly, going from 129 supported actions to 371. Eleven services What arrived since 1.0.0, on top of the EC2, EBS and S3 we started with, plus VPC and IAM: - ALB / NLB (1.1.0) - L7 on HAProxy and L4 on nginx, HTTPS listeners, listener rules, health checks - EKS (1.7.0) - K3s control plane, managed node groups, IAM-authenticated kubectl , IRSA, add-ons - ACM (1.7.0) - Certificate import, then issuance and renewal from a tenant Private CA - ECR (1.9.0) - OCI registry for docker ,crane andskopeo , lifecycle policies, immutable tags - ECS (1.10.0) - Task definitions, services, awsvpc ENI-per-task, capacity providers, task IAM roles - RDS (1.15.0) - Managed PostgreSQL 18.1 and MariaDB, snapshots, parameter groups, backups - Ochre (Bedrock) (work in progress) - Converse and InvokeModel, knowledge bases, guardrails, self-hosted weights on GPU Load balancers were the first thing we added. ALB runs on HAProxy, NLB on nginx stream , and both handle health checks and HTTPS listeners with ACM certificates. Listener rules route on host, path, header, method or source IP, so you can put one load balancer in front of several services and split traffic the way you would on AWS. EKS was the hardest of these to build, and the first major milestone. Aim eksctl or stock terraform-aws-eks at a Spinifex endpoint and you get a cluster, with a K3s control plane doing the work underneath. aws eks get-token authenticates your kubectl against IAM access entries, so there is no aws-auth ConfigMap to keep in sync. Pods assume IAM roles through IRSA, backed by a per-cluster OIDC provider. It started out single-node. In 1.8.0 the control plane ran three etcd servers on separate hosts, and by 1.15.0 those spread across availability zones, so losing an AZ no longer takes the cluster with it. Persistent volumes work through the aws-ebs-csi-driver add-on backed by Viperblock: a gp3 PVC binds, mounts, and survives a pod reschedule. The AWS load balancer controller gives you real L7 ALB ingress. If you need GPUs, 1.12.0 exposed them to pods through the device plugin, and threw in etcd snapshot and restore, so you can rebuild a whole cluster from a snapshot. ECS and ECR came in together, and we built them for sites with no internet. Workers pull images from the internal registry without needing external DNS, which is the difference between working and not working at an air-gapped site. Under awsvpc each task gets its own ENI and VPC IP, assumes IAM roles through the task credential endpoint, and registers with ELBv2 target groups so your containers sit behind a load balancer. Reboot a host and the ECS agent picks its containers back up. Delete a cluster and it tears itself down instead of stranding task definitions and container instances. RDS landed with PostgreSQL in 1.15.0 and MariaDB in 1.17.0. Each database sits on a private endpoint inside your VPC, scoped by security group, with no public address anywhere. Subnet groups decide where that endpoint lands and parameter groups tune the engine. Backups run in your window and get swept once they pass retention, and you can take a snapshot by hand and restore it into a new instance. Core EC2 functionality kept growing amongst all the new features: capacity reservations in 1.9.0, spot instances and AMIs from snapshots in 1.11.0 and launch templates in 1.12.0. GPU passthrough covers NVIDIA and AMD, with MIG slicing if you want to carve a card into pieces. Tagging now works the way you expect across instances, volumes, snapshots, AMIs, key pairs, VPCs and gateways, at creation or after the fact, with tag: filters on describes. Identity, and why a stock cloud image boots In 1.0.0, IAM had users, policies and SigV4; the absolute minimum. Today it implements 75 of the 159 operations in the IAM API: the core of users, groups, roles, policies, instance profiles and OIDC providers. Most of what is missing is stuff that doesn't fit as part of a self hosted deployment, so MFA devices, login profiles, password policies, SAML federation and credential reports. The real gap you are most likely to hit from Terraform is policy versioning, and it is on the TODO list. Roles and instance profiles came first, in 1.6.0, with STS on top of them: AssumeRole , GetCallerIdentity , GetSessionToken , and the AssumeRoleWithWebIdentity that IRSA on EKS depends on. Groups followed in 1.11.0, members inheriting their permissions, and you can attach managed policies or write inline ones on users, roles and groups alike. Tagging arrived across users, roles, policies, instance profiles and OIDC providers in the same release, and 1.12.0 put inline policy editors in the console. From 1.8.0 the console itself dropped static long-lived keys and instead signs in with short-lived STS session credentials. Registering an operation is the easy part. The hard part is enforcement: getting every service to reach the same verdict on the same policy, whether the caller is a user, a role session, an instance holding IMDS credentials, or a pod holding an IRSA token. At 1.8.0 a session from AssumeRole was blanket-denied on the control plane, so instance-role credentials coming out of IMDS could authenticate and then do nothing. That release started authorizing those sessions against the policies attached to the role, which is what made IMDS credentials useful. Inline role policies followed in 1.10.0, enforced by S3 rather than stored and ignored, and 1.11.0 extended that to inline policies on users and to permissions a user inherits from a group. ECR began enforcing IAM on registry operations in 1.13.0. 1.15.0 enforced iam:PassRole on ECS task and service role ARNs, and scoped rds:* by principal class and resource. 1.18.0 and 1.19.0 brought policy evaluation up to the AWS model. Policies evaluate against the resource ARNs a request names, across every service, so a policy scoped to one bucket or one instance authorises that one. Condition blocks are enforced, and S3 supplies condition context keys on the data path so bucket and object policies can use them the way they do on AWS. Policy variables like ${aws:username} and ${aws:userid} resolve before matching, and one that cannot resolve fails closed. sts:AssumeRole is gated on the caller's own identity policy, PassRole authorizes against the stored role ARN, and a role or policy ARN handed in by a caller is checked against the stored canonical form and re-anchored onto that caller's account, so nobody reaches into another tenant's namespace by writing a convincing string. IMDS is the piece that makes an unmodified cloud image work. Each instance gets its own metadata endpoint on the link-local address, served on a per-tap datapath, and cloud-init's stock Ec2 datasource reads it the way it does on AWS. Boot a stock Ubuntu image and it collects its own hostname, network configuration, SSH keys and user-data at runtime. One AMI serves every instance and you customise nothing. The rest of it matters once you are moving real workloads across. A client pinning any dated IMDS version resolves the same metadata tree, so SDKs and cloud-init work against it unchanged. You can drop an instance back to IMDSv1 with HttpTokens=optional when something old needs it. And since 1.17.0 IMDS serves the deployment CA at /spinifex/ca.pem , so rotating certificates reaches your guests without anyone rebuilding an image. Pointing Terraform at it The whole promise falls over if your existing tooling can tell the difference, so a large share of these nineteen releases went into behaviour you notice by nothing going wrong. Idempotency came first. A retried create carrying a ClientToken stops duplicating resources across EC2, EKS and ELBv2, and 1.19.0 extended that to seven more EC2 creates plus RunTask and CreateLaunchTemplateVersion . Deletes are idempotent on already-deleted resources, so a tofu destroy that half-failed converges when you run it again rather than erroring on the resources it already removed. Error codes carry more weight than you would think, since the SDK's retry logic reads them. InsufficientInstanceCapacity used to return a 4xx, which meant SDK retries never fired, so it moved to 503 alongside the other capacity errors. AWS error codes propagate through nested wraps rather than collapsing into something generic. DeleteSecurityGroup names the resource blocking it in the DependencyViolation . Underneath all of that sits a conformance harness. Every successful modelled response the integration suite produces gets validated against the pinned aws-sdk-go models: shapes, required members, types. Error envelopes are checked against the service models where those declare errors. EC2's model declares none, so EC2 is checked against a catalog curated from AWS's published error reference, including the documented split between 4xx client and 5xx server codes. Performance The load balancer microVM was the first big win. In 1.4.0 we moved it to QEMU direct boot and dropped its start time from 3.2 seconds to 264 milliseconds, which also removed the system AMI import from the install path. Block storage came next and took longer. 1.12.0 bounded parallel chunk uploads with coalesced checkpoints and removed write amplification from the live checkpoint path. 1.15.0 replaced the linear extent scan with an ordered index, batched read resolution, repaired the persisted-data cache, and narrowed the write-path lock so concurrent volume throughput stopped serialising. 1.16.0 made multipart uploads stream, so a large object no longer co