Nobody cares what your code looks like. Nobody cares what your architecture looks like. As engineers we should worry about creating forgettable experiences (at least in enterprise). They care that it does the job well, with low load times and an easy to use UX. That's it. If you did your job right they'll forget that software helped them do their job or complete their task faster / better because the experience was so seamless they spent the entire time focused on their own goal, and not how to navigate your software to get there. How you get there is irrelevant. Now it's your job as an experienced and intelligent engineer to make quick, thoughtful decisions on how to get your product to where it needs to be so that your _customers_ can create more impact on their own business / lives faster.
Nobody cares what your code looks like. Nobody cares what your architecture
looks like.
But if your warehouse is a mess, if the aisles of your warehouse are blocked by crates strewn everywhere (that you've been meaning to put away for ages), then it does matter if your warehouse is neat and clean because it takes forever to maneuver your forklift in to retrieve products to ship them out. If your warehouse is disorganized, and you can't find anything when you need it, then it matters.
The real trouble, I think, is when people have reasons to choose not to keep the warehouse organized and functioning smoothly (like workers who see it as drudgery or managers who want workers working on something else) and they try to excuse it by saying, "Our customers never visit."
(I don't think the person above is doing that, incidentally. I think they're just agreeing with the keep your eyes on the prize message.)
I work with a product that sees a massive difference in use between the low end customers and the very highest end of customers. For the low end customers it doesn't matter what your architecture is at all. Could be a single small machine in a closet and it would suffice.
For our large customers it matters 100%. They are doing throughput 4 to 5 orders of magnitude higher than our most basic customers per day. They will push NAS and database performance to the limit. Every time we think "eh, no one is going to do that much" our customers come back and prove us wrong. Databases expand to billions and billions of rows. Query performance falls apart. Design decisions lead to lack of ability of horizontal scaling.
And when the systems get to this size the costs of operating the system add up quickly, and if your competitors choices lead to much lower operational costs, then expect to lose customers to them, especially as your product equalizes in capability, or at least meets their set of needs.
Part of being a professional engineer is recognizing that your knowledge is always limited. No matter how much you think about these things, they will always break. Being too careful is as bad as being reckless.
Instead of trying to prevent failure, accept that it's inevitable and make sure you are fast at detecting and fixing.
And note that this is more than just scale of use. It is also scale of development. Keep adding changes to something, and it will break. Stability of application requiring stability of development is just not something we care to admit, that often.
That is, don't hide when/why something will break. But also don't get blinded looking for how to avoid all breakage. If you can, find ways to isolate failures and block off entire sections.
My boss once told me, "[toast0], you can't have that kind of attitude if your shit doesn't work"
Rather than change my attitude, I made sure my shit worked.
Here is how I personally define good software engineering:
- Does it do the job? - Is there low cognitive load in regards to following the code? (Can you jump back in the code in 6 months and get your feet quickly... AKA: Don't over-abstract). - Is it performant? - Can you easily make changes?
If those 4 things are true, the rest doesn't matter from my experience.
we can spend ages solving every theoretical.
and to be fair NOTAMs were rock solid until that happened, so if anything this failure was a good thing long-term
Isn't this sneakily hiding the assumption that what your code and architecture looks like has no effect on how good/fast your product is? That's a pretty common implicit sentiment these days, especially as part of the (valid) sentiment that chasing fad languages/frameworks/tools is not a good idea. But it's pretty clearly not true in general.
That being said, caring too much about customer success also has its pitfall, businesses can be driven by the present more than by the future, and by low-impact guaranteed results more than high-impact hypothetical risks.
Do I not count as somebody? ;)
> Now it's your job as an experienced and intelligent engineer to make quick, thoughtful decisions on how to get your product to where it needs to be so that your _customers_ can create more impact on their own business / lives faster.
Quick and thoughtful are often in tension. Sometimes you really do need to take a step back and take the 10000 foot view and think about your systemic challenges. It may be worth investing in a new architecture that will allow you to iterate faster and make quicker, better decisions in the future. The first instinct should of course always be to try do the smallest thing within the limitations of the current system to deliver customer value, but there are times when you are spending so much time fighting the system, that it is a wise strategic investment to rearchitect. It takes significant maturity as a developer to recognize the difference between this type of investment and hype-driven churn.
There are always going to be the kind of developers (often quite smart and idealistic but relatively inexperienced) who are easily susceptible to hype, discover a shiny new toy and want to rewrite everything in its image. This type of developer can waste a huge amount of time, both their own and of their colleagues, engaging in fruitless technical debate.
Also, it is not just the customer who matters. You as the developer also matter. If you hate the system you are working with so much that you desperately want to rearchitect it, you either should find a way to do that or leave the project/company regardless of whether you are able to provide value for customer.
But here is the employee paradox: He may sooner or later shift his main focus from customers to his own career. So he reinvents the wheel building the next level hot js framework, start measuring his goals in Github stars and conference bookings...
In short, his tech stack IS the product, again.
It feels more like we engineers use such logic of 'better code/application' to justify ourselves doing what we would like to do and build with the code than anything else that is related to the user.
This isn't an absolute. Yes, people using your thing don't care about the architecture. But developers sure do. If your system is built in COBOL, you will have a smaller pool of developers who will have an interest in working on it.
Pretty much every other incentive in an organization is for product. Its really easy to build things as fast and cheap as possible until you literally can’t anymore because no one understands the code.
It's a thing of beauty.
And absolutely no one but me, cares.
That's exactly what I want.
I'm a little bit butthurt that no one wants to gaze in adoration at my cool toy, but it's not open-source (as if that would even matter, as I've discovered that no one actually ever seems to look at open-source code). The app will work much better, as a result of the work that I've been doing for the last couple of months.
there are 3 circumstances in which i regularly look at open-source code:
a. i want to fix a bug or improve some rough UX.
b. i want to add new functionality.
c. i want to borrow something from it for use elsewhere.
you’re right that the better your code, the less likely i am to read it for reason (a). and the more sophisticated your code, the less likely i am to read it for (c), as i’ll be hunting for reusable libraries rather than cribbing directly from applications.
but consider that for any commercial product, i’m also less likely to read the code for reason (b) because of largely unwritten cultural norms (if i contribute code to a project that has paid devs and don’t get paid for that, it feels like i’ve been suckered).
this being an iOS app exacerbates all this because even if i find a bug and suspect a one-line fix that would be worth my time, i won’t pursue it because i have absolutely no idea how to deploy my patched version (i have to register with app store, maybe pay some licenses/fees, does Apple even allow me to distribute patched versions of someone else’s software?)
so, yeah, you’re not wrong, but you’re also working in a context that really amplifies what you witness around open source (and if you specifically want your beautiful code to be read by others maybe this shows which approaches are more/less likely to get that).
I _really_ wish our company would open-source the code just as a way to make the team feel better and hold ourselves to a higher standard. I know no one will actually read it unless they're trying to find a vulnerability - and that would only improve the product! But even with a restrictive license the company is concerned about "trade secret"/"intellectual property" stuff.
Personally I feel like if we figured out some tricky problem other people have then we have an ethical duty to share it with the community
The app has had a fairly long and winding trajectory. Most of the reason that it’s taken this long, is because the other stakeholders really didn’t know what they wanted, when we started, so the project kind of “accreted.” It’s really been a long prototype phase. Much better than an MVP, because we were free to make massive changes, whenever we wanted. It has a couple of backends, but they have been stable for years. Most of the thrashing has been on the UI.
The last few months, the functionality has stabilized to the point that I could factor out this SDK, and things will move very quickly, from here.
We’ll see what I can write, once I can come up for air.
People sometimes complain that software engineering isn't real engineering then push back on components like these, preferring hairball duct tape for short-term business reasons, which isn't exactly the quality that German and Japanese engineering have.
It's a closed-source app, and probably no one will ever see the source. I don't expect them to. I have plenty of other code that folks can look at, if they want. A lot of that code is actually integrated into the app (I use lots of dependencies; just mostly ones that I've written).
I was really mentioning that the stack doesn't matter to most of the stakeholders on the project, and I'm fine with that. It's the way that it's supposed to be.
Oh, and it's an SDK, not a framework. It happens to have most of the app operation embedded in it, but the framework is UIKit. Any Apple dev of reasonable skill would be able to take it and run. They should have some skill and experience, though, or they are likely to make a hash of things; with or without the SDK.
That's life. Did you call up the bus company and thank them for engineering a bus to get you to work safely?
In retrospect, it seems to have fallen flat.
This would depend on your company culture of course.
It's kinda a shame they reduced it to this. A single machine in a colo center is going to be far more reliable than single availability zone in AWS, which is all many people resort to. And maybe we've just gotten lucky, but in general our very simple 20 machine colo center setup has been more reliable than our single region AWS setup.
But yeah, if you literally put Joe's computer in a closet it isn't gonna be too reliable for all sorts of reasons completely unrelated to the reliability of modern computer hardware.
Colo is fine if you can set it up extremely quickly and it takes minimal ongoing support time. Depending on what you mean by Colo, that may or may not be possible.
I think that depends on the colo honestly. What is so unreliable about a single EC2 instance in a zone?
This isn't bias speaking, I work on Fly.io, our VMs are less reliable than EC2 VMs. AWS's pitch is that all the extra complexity in their infrastructure benefits you. So is ours! But it is, in fact, extra complexity that will bite you in the ass if you don't build your apps the right way.
What happens when there's an outage at the rack level where the 20 machines are?
Or when the whole colo is on fire? [0]
[0] https://www.datacenterdynamics.com/en/news/fire-destroys-ovh...
If you need to do something that's not just generic CRUD app/site... picking an exotic technology that high-powered programmers love, might make it easier to hire.
Compare:
"We're building a gig economy app to clean gas station bathrooms, and we eat our own dogfood!"
to:
"We're USING RUST to build something (not a crypto scam, honest), and WE WILL PAY YOU MONEY TO HACK RUST, and did we mention RUST!!!"
That aside, finding warm bodies to fill a position is usually not the problem, they need to actually help your product grow.
Focusing on the technology prominently is like hiring security guards by touting the big guns they’ll get. You’ll be rolling the dice on the people that come to you for the position.
Yes, but they do give a shit if:
- your product is slow, so they assume it's crap and leave your site
- your product breaks, no one notices or fixes it
- your product is bad, because your wrote it in a safe boring technology like cobol and therefore couldn't access the best developers. (exaggeration, but variations on this are true)
My point isn't that you must use the newest shiniest tech and infra to do everything, my point is that when people write blog posts like this, they want to make it seem like there's a clear line between what's "cool shiny tech" and what's "hard nosed business focused shipping".
In reality, it's a lot fuzzier and these things feed back into each other. For example, you shipped bad code early on to get velocity, but it turned out to be foundational and hard to replace, so your best developers leave as your codebase becomes a ball of mud that it's become clear will never be fixed. So now your worst developers are working on a ball of mud and making it worse... etc etc.
As always, the answer is that this stuff is hard, and you can't make a technology decision by just considering things a user wants to buy and saying everything else is navel gazing and messing around.
Yes, those things are always important (if not critical); it's not meant as a defense of bad architecture, merely to try to make those decisions through a product/customers-first lens.
(I did try to connect quality/reliablity, perhaps unsuccessfully, later in the post, e.g. in goals: "If the system is unreliable, we won’t ship as much product ... If we don’t replace this system, more and more customers will experience a broken product.")
I don't think there any tech stack that will have any significant (or even just measurable) impact the speed of the product, for 99% of the startups.
> - your product breaks, no one notices or fixes it
This has nothing to do with the tech stack. You can setup monitoring, alarms and metrics for any tech stack.
> - your product is bad, because your wrote it in a boring technology like cobol and therefore couldn't access the best developers.
99% of startups won't be able to afford the best developers anyway. 99% of startups don't need the best developers. Average developers can make great products.
---
You are taking "your tech stack does not matter" and twist it into "You will have problems if you have the worst tech stack and the worst developers!". Cool, but that's not what the article is about.
> You will have problems if you have the worst tech stack
Cool, so we've established that I'm saying the tech stack matters. All of the problems above have to do with the tech stack. Yes performance is part of your tech stack. Yes, how easily you can monitor it and is part of your tech stack.
> 99% of startups won't be able to afford the best developers anyway. 99% of startups don't need the best developers. Average developers can make great products.
Here again, the tech stack matters. If you have known mediocre developers, you should pick a language like Java or Go because otherwise it's going to be a mess. If you have good developers, you should trust them to pick the technologies.
if the product is really good compared to everyone else, people won't care (as much) that it's broken as long as it comes back up quickly
That said, I’d argue that if as a startup you believe you’re probably going to quickly and easily achieve product market fit, it is perfectly rational to focus on laying the right technical foundations so you don’t end up saddled with tech debt down the road. Once the product has become complex, rewrites are incredibly hard. And there are multibillion dollar companies out there dealing with the consequences of a tech decision a founder made in the first month of development while being focused on finding product market fit.
Laying the right technical foundations shouldn't be difficult -- it should be generally be picking the most boring[1], dependable choices that you've been able to rely upon in the past.
Facebook was PHP, Instagram was Django etc. They became what they became while using incredibly boring "old-school" tech.
The product is paramount, not the tech.
> Let’s riff. I’ve had my “macho engineer” days, I’ve built that stuff. Take me deep, I’m ready.
or
> No; customers are not paying for, nor give a shit about, these things. Sorry.
Just feels to me a bit condescending for no reason
Fair feedback, thanks!
I wrote that with myself as the listener in mind -- I often need the reminder that is the post's title -- but I can see how the delivery would sound abrasive to another reader. Gave it a light edit.
Really? That's pretty funny. How could anyone miss this lesson who has ever used a computer to do things.
However, it is natural for a myopic technical person to want to leverage their existing skillset, as this knowledge represents to them the key value that they bring to the organization. Unfortunately, blind loyalty to technical stacks is not likely to be an advantage, long-term. The ability to balance the need to leverage existing skills and taking advantage of new skills is critical.
I 100% believe that the tech lead gave this presentation, because part of my onboarding, such as it was, went over pretty much all the technical details of how the system was built, but never did I really get a sense of what the goals were, what functionality was in place, or any overall understanding of what all the pieces were supposed to be doing. Yeah, ok, it's great that you created a bunch of code using gRPC to communicate internally and connected up to bunch of REST backend services sending and receiving JSON, but so what? How did any of that address the functional needs of the stakeholders?
I say usually because some startups are building new computers, libraries, and software infrastructure and care a lot about things that SV-style startups don’t.
Which means: this isn’t universally true advice. If I’m buying a new computing system from you I’m certainly interested in your tech stack and it will be a key feature.
But if your business is getting people to buy more things they don’t need then yeah, use a batch script and some spreadsheets if that gets you to market faster.
But over time, for some...
I strongly want to believe that malleable software will find a niche and grow it. Expert users are a sight to behold, even with the limited offerings we have about, be it IFTTT, Excel, apple/windows automation tools, or what-have you. If a system can actively work to onboard, to layer it's complexity, to make it's flows observable & hackable, there's a good chance we could start emerging new kinds of power users, could start onboarding more people into software & systems literacy, in a non-overwhelming & fun way.
The web has a hackability like this, that is joyous: writing a small userscript to tweak or rework a site to your own pleasure can be phenomally easy & fun, is a great way to get folls started with & enjoying computing.
Again, the advice here is good. Most people are not trying to strike at the roots of human-computing interaction.
But some folks should be, knowingly, with intent to make open, transparent, hackable software. Soft software. And in these cases, the stack matters enormously, is the truthful reality behind the obfuscating veil of interface. Efforts like Naked Objects are about more honest systems, and when we invite the user in, invite them to our level, bestow real power upon them, these invisible opinions implicit in our stack become much more the shape of the thing.
or craigslist
i like new tech quite a lot but so much of this stuff are solutions looking for problems
BUT this is a system I know, and inspecting and changing data is so easy on Airtable. Managing content is ultra fast. And no, the Airtable API is slow as mud and which is causing my project to be slow as mud. HOWEVER, I launched this version very quickly. Much quicker than doing it on Supabase or SQLite or whatever... because I'm a designer, and it's hard for me to get started on those platforms.
If this takes off, somehow, then I'll worry about scaling later. Right now I've got a product to launch (and sell)!
While it's true that the stack just enables you build something for people, if you just chase market you have a good chance of piling up technical debt and slowing down before you get there.
I have seen far too many projects take twice or five times as much budget as needed because people didn't know what they were doing from the get go.
Greenfield applications should start with a rough idea of what the technical requirements will be (what ordr of magnitude of users / transactions per second) and start by cloning a reference implementation that does what's needed. Otherwise you will never get ahead of product.
Some amazing satire on this exact topic.
They sort of are if your bad architecture means features are slower to roll out than your competitors and each release introduces bugs and regressions.
I'm wary when I hear this sentiment. At the heart of very versatile companies is a definition of what makes them who they are, both in a product sense and a technical sense.
I've worked at very successful companies that wrote their own networking stacks, load balancers, cryptography, service catalogs, paging systems, etc... The stark views of fully embracing a Not Built Here mentality and the "build product and product only" are broken for me. I think executives make bets that authorizing a certain project will add to their direct and marginal gains. Direct gains being cost, performance, ease of use (UX/DX), etc... Marginal gains being the experiences that add up over time building custom things or being able to leverage the fully custom parts of your stack.
If all you care to have expertise in is the product people see you'll miss the products that make that product faster, more secure, and easier to maintain long term.