Instead of learning about those achievements and aiming to program for the same reliability, clarity, and sophistication, we see an abundance of software that cannot clearly describe their own behavior nor misbehavior.
Instead of incorporating the full functionality of XML/HTML/CSS/SVG/JS/WebGL into the development experience and providing ways to control them at the fundamental level, we reinvent crude approximations like the various web frameworks.
YAML and JSON often trumps XML/XSD until things get out of control, and even then, people still don't learn the lesson. Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.
Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right. Many people riding the microservice bandwagon have never even heard of Paxos, not to mention TLA+.
Many programmers keep learning new shining frameworks but are reluctant to learn about the crucial fundamentals, e.g., Introduction to Parallel Algorithms and Architectures, nor how to think clearly and unambiguously in the spirit of Coq/Agda/Lean.
No wonder ChatGPT exposes how shallow most of programming is and how lacking most programmers are in actual understanding. Linear logic and dependent types are there to help us design and think with clarity at a high level, but people would rather fumble around with OOP class hierarchies (participate in the pointless is-a/has-a arguments) and "architecture" design that only complicate things.
What is this madness? This doesn't sound like engineering.
https://twitter.com/maxkreminski/status/887815522061926400
"a reminder: if inexperienced creators are using your tool to churn out loads of half-baked garbage, your tool is a phenomenal success"
Software is such a powerful concept - it basically imbues physical objects with magic - that even bad software is hugely empowering to its users and takes off very quickly. The demand is staggering.
I appreciate that when you see yourself as the most intelligent person in the world it becomes intolerable to be surrounded by unthinking muggles scratching in the dirt, but after a while you realise that life is more complicated than that, people usually have good reasons for doing the things they do, and that perfection is neither attainable nor necessary for most of that.
Most tech choices are made on the basis of incomplete information and incomplete planning or even no planning ahead. Many are merely following hype, instead of truly looking at the options and making a wise choice. Hype creates awareness of products or software, which ultimately reaches the uninformed masses. People not in the business of making software themselves have usually vastly less information to base their choice on and often make questionable choices. This in turn generates more incentive to continue making software like the one they chose. This is what amounts to the quote you posted.
Though if you're talking about the lower bound for calling one's work "software engineering" you have a point, I think it's reasonable to expect some best practices from experienced developers. One problem seems to be how easy it is to underestimate the cost of bad software and what it takes to produce good quality. This reminds me a bit of the wildly inconsistent quality of CGI effects in today's movies, which is often the result of late changes and in turn insane deadlines, to some people VFX is magic anyway so why can't it also magically be done faster?
And a more subtle question: what happens below the acceptability theshold? It doesn't seem that acceptability can be driven up costlessly at zero time investment, so what this means is either "less software is produced" or "software is more expensive".
(We see a version of this with the app stores, which enforce a really crude "quality filter" by banning apps)
This also affects our people (usually our top engineers) - which is why I want to start developing our own products this year.
Cover your ass, ticking off items as launched are other areas that lead to speed over quality every time.
The OP post reads as very naive, without experience in the real world corporate politics. No one really gives a shit about most things on that list (unfortunately).
- please put this broken crap of 15 services written in 6 different programming languages on a 50 node cluster so we can read and write data with 300 kbit/s.
2 years later:
- could you do something about this cluster, it costs us too much.
> This doesn't sound like engineering.
It is precisely engineering. As opposed to pure science and art. (I consider mathematics to be, above all, art.)
Based on the post itself, you come with a theoretical mindset. You may consider purity to be more important than practical applications. Yet, if people write software to be used, they focus on the latter. Sometimes it results in hacky code even within an already hacky language. There are no extra points for purity.
Purity itself is a double-edged sword. Sometimes it makes the code more reliable. Other times - it generates a lot of abstract nonsense, which makes it harder to reason about the piece of software or change it.
On the positive side, look at the Rust language (and community!). While it has lovely abstractions and safety guarantees, it is a practical language - performant for writing and execution.
FWIW, I happen to have the opposite opinion. I find Rust easier to write than, say, Ruby or JS, because it is statically typed, has enums and great pattern matching, and traits are great.
Compare and contrast with pure languages that are less versatile. Sure, Haskell has its practical uses. From a popular example - Pandoc. Yet, I know many programmers who have been preaching it is the best language but could not write a performant program that solves a given task that does not focus on algorithmics.
By contrast, PHP or early JavaScript were not pure by any sane standard. Yet, they conquered the web. (Fortunately, ES6 and TypeScript made the JavaScript environment much saner.)
Is XML better than JSON? For longterm stability sure. A quick config file? Nope. You see, to understand XML you kind of have to know how to work with trees. Bread-and-butter for compsci grads, a nuisance for all others.
Recently I needed to query a SOAP-based API. It took me 3 days because I had no idea how a namespace in XML worked, because I was not sure which lib in Python to use (lxml was the solution), and because their API had some quirks. I read forum posts from the late 00's, the documentation is really scarce.
I would love to learn the fundamentals, I don't like half-assing things. But a) I'm not going back to uni for it, b) I'm not bright enough to be excellent in two fields. Learning the fundamentals takes a very, very long time. Where to start even? A complete math program? Programming fundamentals as such? Theoretical compsci?
At the end of the day, things need to work. If they fall apart next year I can tell my boss "ooohhh big problem give money". Using XML instead of JSON has no payoff today.
I have an opposite opinion. JSON is great solution for a protocol or serialization format, better than XML (JSON is faster, smaller, safer, easier to understand) But it is one of the worst possible choices for config file format. Even XML is better in that case. Lack of comments is complete showstopper in any source code file syntax.
Well, that would be deeply inadequate and people would have to invent more of them?
When you're shooting a multi-hundred million satellite into orbit it's worth the extra few million expense of formal verification because otherwise you might lose a gigantic investment and even kill people in the process.
When you're working on something without such extreme constraints, with vague SLAs, and limited to no business risk, then regular unit/integration/etc tests are good enough and exceedingly cheaper.
And even then the cost effects of that are so prohibitive that SpaceX transformed the industry with their "we cannot guarantee the hoverslam landing will work first time" approach.
Producing something imperfect quickly and then iterating beats upfront planning by such a large margin so often that it's not funny.
People can give themselves all the fancy titles they want; I don't think 99% of software development is anything approaching Engineering.
(This isn't so much an attack on my fellow programmers as a recognition that this field is very young and still very immature.)
> engineering; the discipline dealing with the art or science of applying scientific knowledge to practical problems
We do apply scientific knowledge to practical problems, all compsci material of data structures and algorithms is science being applied to practical problems.
Seems like there's not that much demand for what you would like to see.
Crappy, barely working, software now is better then 1 year delayed perfect one.
However it probably can hurt a brand reputation in the long run if you have a quality level below customer expectation.
So the quality we see would seem on average in line with expectations. Especially if things work most of the time and just fail to live up to expectations some of the time.
It seems to me that OP places a high level of usefulness or value of software at lower levels of abstraction, and I don’t disagree—it’s akin to how any manufacturing business is tremendously more valuable than the variety of consumer-facing products that can be made from it—but the cost of entry into such a business domain, and actually succeeding to make a profit, is often high.
From that I conclude that higher abstraction and making consumer-facing products through vertical integration (Apple) is more valuable than straightforward manufacturing. As is the old scheme of owning land with a valuable resource under it. And starting your company's name with the letter A.
That was the seed of the madness.
Instead of supplying dialog boxes(open, save, etc) for applications to use and then open files directly, the OS could have supplied handles (capabilities) from those dialog boxes for the applications to use. This would have allowed the user interface to be almost identical, and only require a few lines of code in the applications to change, in exchange for an environment which was almost immune to confused/rogue programs.
Because expectations were so corrupted in the desktop days, the situation now is effectively hopeless.
Applications should NEVER be trusted, especially not by the operating system.
> Applications should NEVER be trusted, especially not by the operating system.
This was basically impossible to do in the early personal computer era, before memory protection was a thing. So MacOS, Windows pre-NT, AmigaOS etc were all built around the assumption of applications reading each other's memory.
Very true. The user had a different option for that, since DOS was basically a program loader, and only one thing ran at a time, they could boot into program A, and use it with disks Q,R,S and know that no other disks could be corrupted. Then later, boot into program B, for disks X,Y,Z. It was capability based security, where each diskette was a capability. (Before hard drives ruined that trust model)
Not hopeless. MacOS, for example, added entitlements that get you there. Applications cannot list random directories or open files at will.
If an application opens a file browser, the operating system shows you your file system. When you pick a file, the operating system gives the application the right to read that file.
See https://developer.apple.com/documentation/bundleresources/en...
The limited availability of Formal Methods and Formal Verification training.
The growing demand for safety critical software for things that are heavy and that move fast and that fly over our heads and our homes.
In order to train, you need to present URLs: CreativeWork(s) in an outline (a tree graph), identify competencies (typically according to existing curricula) and test for comprehension, and what else is necessary to boost retention for and of identified competencies?
There are many online courses, but so few on Computer Science Education. You can teach with autograded Jupyter notebooks and your own edX instance all hosted in OCI containers in your own Kubernetes cloud, for example. Containers and Ansible Playbooks help minimize cost and variance in that part of the learning stack.
We should all learn Coq and TLA+ and Lean, especially. What resources and traversal do you recommend for these possibly indeed core competencies? For which domains are no-code tools safe?
If we were to instead have our LLM (,ChatGPT,Codex,) filter expression trees in order to autocomplete from only Formally Verified code with associated Automated Tests, and e.g. Lean Mathlib, how would our output differ from that of an LLM training on code that may or may not have any tests?
Could that also implement POSIX and which other interfaces, please?
>
The limited availability of Formal Methods and Formal Verification training.*From "Are software engineering “best practices” just developer preferences?" https://news.ycombinator.com/item?id=28709239 :
>>> From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :
>>>> Which universities teach formal methods?
>>>> - q=formal+verification https://www.class-central.com/search?q=formal+verification
>>>> - q=formal+methods https://www.class-central.com/search?q=formal+methods
>>>> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs? https://news.ycombinator.com/item?id=28513922
Formal methods should be required course or curriculum competency for various Computer Science and Software Engineering credentials.
> The growing demand for safety critical software for things that are heavy and that move fast and that fly over our heads and our homes.
"The Case of the Killer Robot"
> In order to train, you need to present URLs: CreativeWork(s) in an outline (a tree graph), identify competencies (typically according to existing curricula) and test for comprehension, and what else is necessary to boost retention for and of identified competencies?
A Multi-track "ThingSequence" which covers; And there are URIs for the competencies and exercises that cover.
> There are many online courses, but so few on Computer Science Education. You can teach with autograded Jupyter notebooks and your own edX instance all hosted in OCI containers in your own Kubernetes cloud, for example. Containers and Ansible Playbooks help minimize cost and variance in that part of the learning stack.
Automation with tooling is necessary to efficiently compete. In order to support credentialing workflows that qualify point-in-time performance, it is advisable to adopt tools that support automation of grading, for example.
> We should all learn Coq and TLA+ and Lean, especially. What resources and traversal do you recommend for these possibly indeed core competencies?
Links above. When should Formal Methods be introduced in a post-secondary or undergraduate program? Is there any reason that a curriculum can't go from math proofs, to logical proofs, to logical argumentation?
> For which domains are no-code tools safe?
Unfortunately, no-code tools in even low-risk applications can be critical vulnerabilities if persons do not understand their limitations. For example, "we used to have already-printed [offline] forms on hand [before the new computerized workflows [enabled by no-code, no review development workflows]]".
Should there be 2 or 3 redundant processes with an additional component that discards low-level outputs if there is no consensus? Is that plus No-code tools safe?
> If we were to instead have our LLM (,ChatGPT,Codex,) filter expression trees in order to autocomplete from only Formally Verified code with associated Automated Tests, and e.g. Lean Mathlib, how would our output differ from that of an LLM training on code that may or may not have any tests?
You should not be autocompleting from a training corpus of code without tests.
> Could that also implement POSIX and which other interfaces, please?
Eventually, AI will produce an OS kernel that is interface compatible with what we need to run existing code.
How can experts assess whether there has been sufficient review of an [AI-generated] alternative interface implementation?
It is also "good enough" for most use cases. So it is reasonable to start out with JSON and only upgrade to XML for complex documents/strutures.
( Melvin E. Conway, https://en.wikipedia.org/wiki/Conway%27s_law )
Thus, we may conclude the world has some people that choose to be "useless and unreliable". The software part is just an artifact of this ideology. =)
This is the original Amazon memo and the rationale for microservices.
The bigger picture is that if you look at the totality of the software and hardware system, from app to browser to OS to hardware, you see that it replicates the company and market structure that produces it. Which is partly why the web is so chaotic. The web is a standards battleground over which companies fight.
Ah, but the customer interface and product layer continues to degenerate regardless of the mission statement. The public always gets what they asked for... especially when it is cheap, awful, and disposable. =)
The only way to avoid this complexity is by keeping programs simple or by writing everything in house in a language like Ada/Spark very carefully, including formal verification and extensive testing. This is way too expensive for non-critical software. Btw, I don't think Erlang's concepts are general solutions, fault-tolerance by restarting services only eliminates certain kinds of problems and isn't suitable for all high-integrity demands.
You see the same thing in a lot of places. When people talk about shopping at Amazon, Wal-Mart, etc., they don't talk nearly as much about quality as about getting the lowest price. Well, the lowest price means cheap junk.
I've read some comments here that I agree with; in a lot of cases the goals in dev are set to implement a solution to a problem either as fast as possible, or without a limited scope.
"We need a software that can fly rockets" - Yeah, but where to?
I studied computer science twice. Both times I have been taught to implement corner cases instead of working on clearly defined projects with scope and specification. "Write a small program that does X" yeah, but the solution space is unlimited, then, maybe? Can you elaborate? Put a scope on it? I mean, can do, may I order compute time on the campus super-computer?
Crap software was there ever since, and it got worse ever since.
Yes, it's much easier once you get to that point. Getting there is the real work. Wrangling clarity into requirements from messy humans isn't amenable to all the nice formal methods, but it's a very critical part of the work.
Formal methods can help with exploration and cooperation.
Did you hear that everyone can build a bridge that stands but it takes an engineer to build a bridge that barely stands?
I'd say software development has a lot to do with engineering then.
Ivory tower academics will keep insisting that there are ways to make foolproof software.
If it were so, they could do that and make millions and trillions.
Software is not a science, its a craft, it depends on the outside world, not some cute lab demos
Not disagreeing with your fundamental point, but it's also true that most people don't value making huge piles of money as much as other goals.
In fact most commercial apps - banks, flight bookers, and so on - are usable. Not superb, not outstanding, but certainly good enough for the job. But the user friction comes from dark patterns and weird decisions.
Example: we booked a hotel on booking.com recently and there's an option to pick a double bed or two singles. Turns out that whatever option you choose the hotel gets a message that says "This booking needs a double or a twin room" and doesn't specify which.
There's probably a good reason for this bizarre and unreliable behaviours, but I have no idea what it might be.
The unreliability happens in the giant apps. Chrome had a problem with Streetview recently, and then it had a problem opening PDFs. OS updates are notorious across all platforms.
But these are huge, frankenstein projects with significant hardware dependencies. Even if you modularise them, formalise them, and test them, how are you going to do that for every possible feature on every possible hardware combination?
Civilization is a goddamn miracle.
1. Affine types in Rust.
2. Refinement types in Ada.
3. Dependent types in Coq/Agda/Lean/Idris/ATS-lang.
4. Separation logic in Facebook Infer.
These methods systematically/automatically eradicate errors that incomprehensive/human-intensive alternatives (unit-testing, code review) can't do at scale. If our industry could adopt and improve on this, we could create truly ambitious projects with confidence, at a lower cost, and at a faster pace because 1) correctness is scaled and 2) mistakes are contained.
Most likely, you also chose cheap over quality at some point in your life. I have furniture that would make a woodworker cry, sportswear that raise eyebrows, unreliable WiFi and bad cables. How can I judge people that don't even know what quality software is?
I have an education in computer science, I know how to write a Paxos setup, I am fluent in Erlang. I am still very much not interested in formal systems, proofs and verified computing, because there I will spend 10x much time for the same set of features. Bugs are normal. I will live with bugs. I won't live with 10x development time lost.
In the world of software - especially for startups - it is completely unacceptable to spend 5 years on building a MVP. There are some protected sectors where things move slower (defense, aerospace, for example), but if you're in the consumer field, you just can't spend too much time. You want to push out a MVP as soon as possible, and build on that.
And most software companies do not get penalized on shipping buggy software. Big game studios ship broken games, and spend a couple of years patching them up to their final form. People bitch and moan, but still throw money at 'em.
If you don't want bloated, broken, slow, and unreliable software - use your pocketbook.
- no engineering standards in computer science, including especially in education of computer science
- some corporate finance views which see technology as a cost center rather than a business enabler
- some corporate strategies where marketing decides what is possible and when it must be ready (tomorrow. or yesterday.)
- computing and software development as a "fun", creative endeavor - as opposed to a rigorous, formal process
I'm sure there are dozens more reasons.
As long as people are willing to pay for dross, people will continue to produce it. Just look at Hollywood, for a classic example.
For myself, I found a company that was interested in producing Quality, and spent most of my career, there. It had its trade-offs.
It certainly didn’t get me much respect from today’s software development community. If anything, it gets me scorn and outright disrespect.
I had to retire and work on my own (for free), in order to finally get to write software the way that I want. I doubt it would be considered commercially viable.
In any case, “doing it right” is in the eye of the beholder. The fact that I'm not writing all my native Apple software in Haskell, is "doing it wrong," to some folks. Don't even get me started on OOP or using UIKit/AppKit.
Cant' please everyone.
I’m best off, putting my values into practice, in my own context, and not being an old man, yelling at the sky.
But I really want to. It just won’t do any good.
If we could abstract away the storage so that it is user owned, or at least a "utility" like electrical power lines where everyone is playing by the same rules, we would be able to trust these platforms significantly more, and rest assured our data is safe even if the software isn't.
Transferring ideas and software discovery are difficult problems. Expecting everyone to know that tech x is superior to tech y and singularly pursue development of tech x is unreasonable.
Most people prefer work with a solution that you are familiar with or caters to a particular technology stack. JSON is a first-class citizen of Javascript, but can be secondary in other programming languages. Anything that provides the least cost or effort to resolving or minimizing the extent of the product at hand is all that is needed. There is almost never a case where point B must follow or supersede point A. Point A continues to solve the problems it was designed for or turned out to be useful for by chance: the technology can continue to be used where it is known and regarded.
I find that the learning curve to get on top of most abstractions is greater than what is required to understand the foundations. Knowing and applying foundational components tends to yield more performant and less bloated solutions. Debugging is easier because you don't need to unravel layers of abstractions which don't quite align with the domain you are working with.
I’m hoping such pressure arrives in the form of legal or regulatory stuff, even if it chills the industry and slows/shrinks it. Until then we’re still in the computing equivalent of the auto industry when people died in mild fender benders that today are just mild annoyances.
Also, I assume most people fall in the middle. Wants to make money but also wants to do a good job. It just turns out those with the purse strings have the incorrect assumptions about tech, and if you know anything about business people you know that a laundry list of facts and reasoning is not always a good way to convince some of them of what should be done. More than likely a majority would see an article about a company going big with technology X. Now they have it in their head they need that too.
Internally at my small business we have stopped using the word bug, and started calling it defect to change the thinking about this. Bugs sound innocent, like they just 'creep in' and there is almost nothing you can do about it as a software engineer. Defects can be prevented, mitigated, or fixed. It places the responsibility in our hands again to deal with it and prevent it. Zero defects is probably not a realistic target (same with any production line for hardware), but it does help to frame it in the same way and take proper responsibility.
I question what's your age and background, this view of the world is something I held when I was pretty young and thought everyone else were just stupid muggles. Until reality hits you in the face and you start to comprehend the constraints of the real world, software isn't some pure intellectual pursuit; it's a tool, a very very powerful tool that's used to enhance the real world somehow.
You can play around with pureness of theory if you want in academia. In the real world there are customers, they have needs and they want their needs fulfilled by a business, the business that achieves that earns profits, that's all there is to it.
Vs. with software, ~any clueless wanna-be kid can make some in an afternoon, at trivial cost. A PHB who could never manage to get a toaster designed & manufactured (let alone sold) can "lead" a team of crappy developers in creating some {cough} great {cough} new pieces of software.
- quality doesn't bring enough to the business value;
- high demand requires hiring people without enough knowledge/skill;
- other people have just different quality metrics than you;
- "know-how" transfer between generations is difficult...
IMHO the similar question may be asked about any domain of life: why not all the buildings are well-built? Why the political/social systems are not just? Why good art is so hard to spot nowadays? They have different answers, but there is one common theme: building "well" is difficult; creating a "just" political system is "difficult", creating a good art is difficult; also writing "useful and reliable" software is difficult.
It's only natural that not everybody is a perfect software engineer, therefore is not a surprise that not every software will be perfect. What's more, there are a lot more mediocre software engineers than the perfect ones, therefore the majority of software will be... mediocre. The same about art nowadays — the sheer amount of music produced (even the word "produce" suggests it) every day makes sure that, in overall, there is a lot more terrible music compared to good compositions. I believe that software engineering has the same issue — the rising number of software projects is an enough explanation for low quality of the software we are able to perceive.
In fact, I don't believe the quality of the software engineering degraded over the years, but it's just more difficult to spot the great projects in the avalanche of "normal" life.
To me this sounds exactly like engineering. You craft the tools / apps you want, you build frameworks around complicated concepts that simplify their understanding and usage, at the expense of losing some fundamentalities.
You don't need to know the history and all the evolution of technology to apply it. It is part of software science, for sure though. You can be good at software science and fundamental concepts, but this also does not imply you are a good software engineer.
Would software in general be better if mentality proposed to you would be standard? I doubt, because the learning curve to enter the industry and even begin to start doing something would be immense, you would need to study as much as doctors now do, and only after 10 years you would be able to be "trusted" with your work.
I do agree though that there is a bare minimum that one who calls themselves "Software Engineer" should know and understand, like OS fundamentals, basics of compiler theory, etc., etc., but it is not so restrictive as you suggest.
Because the only thing in common between the development of intangible products (software) and the engineering of tangible products is the cost of implementation.
Personally I would tend to agree, that using a sharper more capable tool is moving in the right direction. Where I disagree is that we can't all reach for the sharpest, most capable tools, as that would render many/most of us ineffective.
The next question is if only the people capable of using the sharpest, most capable tools can satisfy all production needs. That might be possible for core things such as energy production, but likely not possible for all things down to the mundane. Or to remove judgement/controversy of 'mundane', all things people are willing to pay for.
Customers will not pay for quality. Not in toasters, not in cars, and not in software. Cheap and fast wins the day.
You want your "fundamentals" to be learned? Make knowing them useful to achieve something and document them in nice skimmable, searchable manner.
This is a strange argument in the context of your other complaints (which I largely agree with fwiw). All of them are much safer to parse than ASN.1, which has been a decades long security nightmare, and add additional security concepts on top.
Never mind that they're also networked IPC protocols on top of binary serialisation formats, solving a host of security issues we've had with exchanging serialized Java/Python/PHP objects over the wire, DCOM, etc. pp.
Damned if you do, damned if you don't?
Most people are not prepared to pay for that level of excellence. They want the cheapest thing as quickly as possible.
The only way that this will improve is if the "baseline" quality of software improves so the average coder automatically produces better stuff because the popular tool of the day is providing it for "free" without them really thinking about it or having to opt-in to it.
Can someone elaborate or give hints more about this?
The only reason that other engineering professions does things properly is that bridge building is not free marked capitalism, but government mandated rules. Just look at how much GDPR rules in EU does for actually making companies do the bare minimum to respect privacy.
People don't pay for incorporating full functionality of "XML...Whatever else" they pay for ability to see cute kittens NOW not when dev will be done with reading and understanding ASN.1 well enough to use important parts.
2. Business dont care much about quality, as long as the product is good enough to sell - real improvement and development stops there.
Because it isn't.
