And yes, that's a succinct rephrasing.
When you first use iac it maybe seems logical to put your db and app server in the same "thing" (stack or state file) but now that thing is "pet like" and you have to take care of it forever. You can't safely have a "destroy" action in your pipeline as a last resort.
If you put the stateful stuff in a separate stack you can freely modify the things in the stateless one with much less worry.
If you run a terraform apply and only update microservices but you also have your dbs/stateful things in the same stack/app, you're only updating the microservices so how would this affect the db/stateful at all?
On the opposite end - I feel like there would be scenarios where I needed to update the stateful AND stateless services with the same terraform apply. Maybe I'm adding a new cluster and adding a db region/replica/securitygroup and that new cluster needs to point at the new db region.
In your scenario I would have updated microservices trying to reach out to a db in a region that doesn't exist yet because I have to terraform apply two different stacks. How would you deal with a depends_on?
Maybe I'm misunderstanding this.
Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually.
It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.
I work on the systems test automation side of things. I use TF to deploy EC2 clusters, including the creation of VPC, subnet, SG, LBs, etc. Once done, I tear down the whole thing.
But from what I'm hearing from yourself and others in this thread, it sounds like I could (/should) be breaking those out separately, even though my use case isn't dev/test/prod(/prod2,/prod3) or even multi-regional.
To rephrase, it sounds like it might be useful for me to be able to create some separation, eg, tear down the EC2 instances in a given VPC/subnet while I ponder what to do next and still leaving the other AWS resources intact, for example. Maybe even deploy another subnet to the same VPC, but a different test intention. I know I can simply specify a different AMI and run tf apply and get one kind of desired deployment change.
Bigger picture: when I need to run one test, I'll copy a dir from our GH repo, edit the tfvars, and kick things off. Another test (in parallel, even), I'll do the same but to a fresh dir. (Wash, rinse, repeat.) And I suspect you're already cringing :) but I get why. It makes me cringe. Ideally I'd be working with a single source of truth (local GH fork).
There's also the consideration of possibly making edits to the stack while I deploy/destroy, while at the same time wanting stability with another deployment for a day or two. I suppose that would require having 2 copies of the remote GH repo. Which is several copies fewer than I'm working with these days.
Fwiw, I've already got "rewrite the stack" on my todo list, but can't get to it for probably another 2 months. So I'm eagerly collecting any "strive to do X, avoid Y like the plague" tips and recommendations prior to diving into that effort.
I think you're talking about putting stateful and stateless objects inside of a single module. So you've got /terraform/modules/mybigapp/main.tf that has your microservice + database inside of it.
If I'm right and that's what you mean that's really interesting I don't think I've ever seen or done that but now I'm curious. I'm pretty sure I've never created an "app1" module with all of its resources.
Am I totally off here?
It’s common for an entire environment for a whole biz unit or even company to be a single “stack”.
This pretty much only works if the “terraform apply” is centrally orchestrated I.e gitops is the only way to trigger the terraform run.
A team that e.g owns its own microservice IaC can absolutely maintain a setup long term with app & db in the same state, it just requires care and love (it's easier in some ways, it just doesn't scale along certain dimensions).
Maybe you have other controls / factors that can make it work for you.
But my experience is that as you split things across more teams or have more complicated IaC "supply chains" (eg teams supplying modules to each other or have lots of ppl working on different bits of the same thing) you need to look at ways to make things more foolproof, easier to support and give yourself as much "wiggle room" as possible for upgrades. At this point having state split out is very helpful (almost essential).
Because the terminology seems to be tripping up the conversation I'd be inclined to phrase it like this: a single "terraform apply" should be touching a precious, stateful stack or a disposable stateless stack and these should be clearly delineated. Ideally the stateful stacks should be as small as possible, as much stuff as possible should be in the stateless stacks.
