Migrating Side Projects to a New EC2 Instance
I migrated two side projects, which share a login service and a Postgres instance, off an aging t2.micro onto a t4g.micro, which is Amazon’s Graviton ARM instance. It’s cheaper and faster for the same spec, but it’s also a full architecture change, so there’s no shortcut like copying an AMI over. Everything on the box, the OS packages, the database, had to move by hand. Getting there was its own small project before the app itself even came up. Once it did come up, behind CloudFront, I hit six distinct bugs between the database restore and a fully working state. None of them were hard once I had the actual error message in front of me. All six cost time because the first symptom pointed somewhere other than the real cause. This is a record of the move itself, then each bug, in the order I hit them.
Amazon Linux doesn’t have apt
The first command I ran on the new box was sudo apt update, out of habit. It came back with sudo: apt: command not found. Amazon Linux 2023 uses dnf, not apt. Every install for the rest of the setup, Postgres, the .NET runtime, nginx, went through dnf install -y <package> instead. Small thing, but it meant I couldn’t just paste commands from old notes without checking which package manager they assumed.
scp between two EC2 instances needs a key both sides trust
Once I had a Postgres dump to move, I tried to scp it directly from the old instance to the new one. That failed with Permission denied (publickey,gssapi-keyex,gssapi-with-mic). The old instance was trying to authenticate to the new one, but it had no private key the new instance would accept, since each instance only trusts the .pem key pair it was launched with. Neither box holds the other’s key by default.
Rather than sort out agent forwarding or drop a key onto either box, I just routed everything through my local machine, since I already had the .pem key for both instances there: scp down from the old instance, then scp up to the new one. Two hops instead of one, but no key management on the servers themselves. I used the same round trip for the systemd service files and nginx.conf, not just the database dump, since none of those needed to exist on both boxes at once anyway.
Every Postgres fix in the next three sections is something I’d already made once, on the old t2.micro, at some point I no longer remember. None of it was written down anywhere. A fresh Postgres install on the new box came back with all the same defaults I’d changed years earlier, so I ended up relearning each fix from the error message instead of from a note.
Postgres defaults to ident auth for TCP connections
The API and worker services came up and immediately started failing to connect to Postgres. The error was Npgsql.PostgresException: 28000: Ident authentication failed for user "app_admin". Ident auth checks the OS username of the connecting process against the Postgres role it’s trying to authenticate as. My app connects with a username and password, not an OS identity, so it always fails.
The cause was the default pg_hba.conf that ships with the PGDG postgresql-server RPM on Amazon Linux. It sets peer for local Unix socket connections and ident for TCP connections to 127.0.0.1 and ::1. Both are OS-identity auth methods, and neither does password authentication. This is the stock file, unmodified, so every fresh install on Amazon Linux has this problem until someone changes it.
The fix is to change the two host lines from ident to scram-sha-256, then reload:
host all all 127.0.0.1/32 scram-sha-256
host all all ::1/128 scram-sha-256sudo systemctl reload postgresqlReloading pg_hba.conf doesn’t require a restart or drop connections, so this is safe to do live.
A crash loop that turned out to be the same bug
Separately, top showed systemd-coredump pinned at 100% CPU whenever the login service ran. coredumpctl info on the crashing dotnet process showed signal 6, ABRT, with a stack that went through IL_Throw, DispatchManagedException, and PROCAbort. That sequence is CoreCLR’s standard path for an unhandled managed exception: something threw, nothing caught it, and the runtime aborted the process. The native stack trace itself is useless for finding the actual exception, since the managed frames show up as n/a.
.NET prints the unhandled exception to stderr right before it aborts, and journald captures that. journalctl -u app_login -n 100 showed the real exception: the same Ident authentication failed error from above, thrown inside ClientSeeder.StartAsync, which is an IHostedService that seeds OpenIddict scopes on startup. A hosted service throwing during Host.StartAsync isn’t caught anywhere, so .NET treats it as unhandled and the process aborts. This wasn’t a second bug. It was the same pg_hba.conf issue, just manifesting as a crash loop in one service and a connection-pool warning in the others depending on how each service structured its startup.
Postgres 15 locks the public schema by default
With auth fixed, restoring the database with pg_restore threw a wall of errors, starting with relation "public.work_items_id_seq" does not exist and cascading through every table. The first real error, not the downstream noise, was permission denied for schema public.
Postgres 15 changed the default privileges on the public schema. In earlier versions, every role got CREATE on public automatically. Since 15, only the schema owner does, and the owner is a special role called pg_database_owner. I had created the database as the postgres superuser and then tried to restore as a separate app_admin role, which meant that role had no CREATE privilege on public at all. Every CREATE TABLE in the restore failed for the same reason, and everything that depended on those tables failed after it.
The fix is to make the app’s role the actual owner of the database at creation time, since the database owner in Postgres 15 is automatically a member of pg_database_owner:
CREATE DATABASE app_db OWNER app_admin;Granting privileges after the fact with GRANT ALL PRIVILEGES ON DATABASE does not cover this. It grants privileges on the database object, not on the public schema inside it, so the schema-level lockout stays in place unless you also fix ownership or explicitly grant on the schema.
Peer auth doesn’t work for a role that isn’t the OS user
Once ownership was fixed, running pg_restore against the dump I’d scp’d over as sudo -u postgres failed again, this time with Peer authentication failed for user "app_admin". sudo -u postgres connects over the Unix socket by default, which hits the peer line in pg_hba.conf, and peer auth requires the OS user to literally match the Postgres role name. The OS user here was postgres, not app_admin, so it failed regardless of what role I passed with -U.
The fix was to skip the local socket path entirely and connect over TCP as the actual app role, which hits the scram-sha-256 line instead and does real password auth:
PGPASSWORD='...' pg_restore -h 127.0.0.1 -U app_admin -d app_db --no-owner -v dump.fileThis also sidesteps a second, unrelated permission problem. Running the restore as sudo -u postgres against a dump file sitting in /home/ec2-user failed to even open the file, because that home directory is 700 and the postgres OS user can’t traverse into it. Connecting as the app user directly means the restore runs as ec2-user, which owns its own home directory, so that problem disappears along with the peer-auth one.
nginx routes by server_name, not by port
With the database restored and the login service starting cleanly, I tried to hit it directly at http://<ip>:5020/ and got a connection timeout. ss -tlnp | grep 5020 on the box came back empty. The service wasn’t listening on 5020 at all. journalctl showed the actual bound address: Now listening on: http://127.0.0.1:5120. Wrong port, and bound to loopback only.
Even after using the right port, hitting <ip>:5120 from outside would never have worked, because nginx does virtual hosting by the Host header, not by which port you hit. Two different server blocks in the config had server_name login.appone.dev proxying to 5020 and server_name login.apptwo.dev proxying to 5120. Neither is reachable by IP and port alone. The right way to test through nginx without DNS being set up yet is to force the Host header manually:
curl -v -H "Host: login.apptwo.dev" http://<ip>/That reliably tells you whether nginx and the app are correctly wired together, independent of DNS or which port you assumed the app was on.
CloudFront doesn’t forward X-Forwarded-Proto
After cutting DNS over to CloudFront, the login flow started failing with an OpenIddict error: error:invalid_request, error_description: This server only accepts HTTPS requests, error_uri: ...ID2083. I’d skipped setting up Let’s Encrypt on the new box on the assumption that CloudFront terminating TLS for the browser was enough. It isn’t, because CloudFront to origin is a separate hop, and nginx on this box only had listen 80 configured.
nginx was sending proxy_set_header X-Forwarded-Proto $scheme;, and $scheme reflects nginx’s own connection, which was http since nothing was listening on 443. That header is what ASP.NET Core’s forwarded-headers middleware uses to decide whether the original request was HTTPS, and OpenIddict enforces HTTPS strictly. I assumed CloudFront would set X-Forwarded-Proto itself, the way an ALB does, and that nginx passing through $http_x_forwarded_proto instead of its own $scheme would fix it. It didn’t. I confirmed why by adding a temporary debug location that echoed the header back:
location /debug-proto {
add_header Content-Type text/plain;
return 200 "X-Forwarded-Proto received: [$http_x_forwarded_proto]\n";
}Hitting that through the real CloudFront domain, not a direct-IP curl, showed the header arriving completely empty. Unlike an ALB, CloudFront does not automatically inject X-Forwarded-Proto on requests to custom origins. Since CloudFront was the only way into this box and always terminates TLS at the edge, the actual fix was to stop trying to forward a header that never arrives and just hardcode it:
proxy_set_header X-Forwarded-Proto https;The proper long-term fix is still to get a cert on the origin and set CloudFront’s origin protocol policy to HTTPS, so the whole path is encrypted end to end. The hardcoded header is what got login working immediately without waiting on that.
A missing IAM role breaks S3 presigned URLs, and the client hides it
Once login worked, I logged in and my account showed zero workspaces, on an account I knew had two. Querying the database directly showed both tenant membership rows present and correct, status = 'active', tied to the right user id. The data had migrated fine.
The actual failure showed up in the browser network tab: GET /tenants/list was returning a 500, Unable to get IAM security credentials from EC2 Instance Metadata Service. The endpoint’s query joins tenant_memberships to tenants and, for any tenant with a logo_url, calls out to S3 to turn the stored key into a presigned download URL. Generating a presigned URL still requires valid AWS credentials to compute the signature, even though the URL itself doesn’t touch S3 at request time. The AmazonS3Client in this app is constructed with no explicit access key, so it relies on the SDK’s default credential chain, which falls back to asking the EC2 instance metadata service for an IAM instance role. The new instance never had that role attached, so credential resolution failed and the whole request threw.
The Flutter client made this much harder to see than it should have been. The code that fetches the tenant list checks if (result is List) and silently treats anything else, including a 500 with an error body, as an empty list. A backend failure and an account with genuinely zero workspaces looked identical in the UI. The fix on the AWS side is attaching the correct IAM instance profile to the new instance so the credential chain resolves the way it did on the old one. The fix on the client side, which I still need to do, is to stop swallowing non-200 responses as empty state and surface the actual error instead.