But does it also strike back with these downsides: 1. limits your ability to create "novel" or technically non-trivial systems; 2. moves you closer to the business and thus forces you to deal with many levels of management that influence not only "what" you program, but sometimes also "how" you do it;
Having enough experience with high-level stuff I have this stupid idea that going closer to the OS/metal and wisely choosing tools can somehow protect an engineer from politics and provide an environment with more grunted, real, technical constraints.
Anyone did such a switch? Please, let me know how this corresponds with your experience.
https://github.com/angular/angular/issues/37293
I really starting hating Angular due to performance issues.
Surely full-time kernel developers must have tons of shortcuts that can speed this up compared to the regular build process, no? There just was an article about booting a VM in hundreds of milliseconds on the frontpage for example.
But I was working for a hardware manufacturer too (One Laptop Per Child), making its own laptop designs and doing board bringups on them, and when the point is to test real hardware you don't get shortcuts like using a VM. :)
The shortcut we had for trying to maintain productivity while testing real hardware was actually to stop using Linux: we had a port of Open Firmware that was the first thing run on new hardware, and then running tests (e.g. bisecting how large the timeout on a device request needed to be) could be done in a live OFW Forth REPL, and then ported to Linux from there.
Low level programming is way harder with the C++ stack. I blame mostly the language rather then the level. C++ is just a pain in the ass. If you get a low level programming job that uses rust things might be better.
Basically at the higher level, when you hit an error it is (generally) much more trivial to solve... not so with C++. With C++ you can hit crashes and not know where it even came from, this is much more rare in higher level languages.
I know several people who vehemently hate C++ with a passion and if you're going into low level stuff, very likely you'll have to deal with C++.
Yep, it's definitely the language. You can do low-level programming with Rust and pretty much never have this problem (never say never, but it's certainly not a common issue).
As somebody who writes more C++ than Python, I have pretty much the opposite perspective:
If I hit a reproducible crash in C++, I can drop into a debugger, and work out where it came from. If necessary, I can look at the disassembled code, because it's reasonably feasible to map assembly language to C++ constructs.
Python code generally doesn't crash that way, it's true (unless it interfaces with C modules). But when errors occur, I have to debug them on a Python level. Dropping down the stack is nearly hopeless because the Python interpreter is just too thick an abstraction layer between assembly language and the Python code.
And I find myself having similar problems with Swift, for all its elegance. The mapping into assembly language is just considerably more complex than in C++, so you lose a low level debugging tool.
I currently write more C++ then python. I'm primarily a C++ programmer. This was not the case for most of my career it is now, so I have enough experience from both perspectives.
The python debugger can display the stack and you don't have to compile it in debug mode. The pycharm debugger is excellent, but even if you don't use that pdb libraries can drop the app itself straight into a debug console with one line of code...
pdb.set_trace()
But .. embedded vendors can suck. A lot. Terrible docs, epic levels of errata (e.g., a PowerPC vendor who, at one time, shipped a bigger errata document than the entire manual set for an x86 of the same era), and just.. marginal everything. In some cases, they are really interesting - the Broadcom Fastpath SDK was actually a good mix of excellent and terrible.
But the SDKs can be absolute trash even from name-brand vendors. Without naming names, I'll give an example: SDK contains a linux distribution from an CPU ASIC company shipped as a tar file. You have no indication whatsoever what the reference point for the linux kernel is, no idea what patches it has or does not have, and so on. When you ask them, they don't even understand why you care. Get ready for the fun of massive diffing. The network stacks on the really low-end can be shockingly bad.
From a career standpoint, embedded is a bit of a niche. You will probably always have a job - especially in the IoT era, but even without that. It is typically a somewhat solitary role - you do the code for the XXX block (if drivers) or for some set if functions (IoT-style embedded), but you don't end up doing "broaden your scope" things as much - no system-level nor distributed systems work, which can be somewhat limiting.
For what it's worth, as someone who's done both (Linux kernel maintainer and web developer), this sentiment reads as a kind of chauvinism to me -- the implication that higher-level work isn't real coding.
If I had to choose which was more difficult, I'll probably go with web development: the kernel simply had much less that you had to know to be effective, compared to the massive stack of protocols and frameworks that even a simple web app runs on top of and demands competency with.
Comparing work I do, the higher up I go the less it can feel like coding. Javascript, Python, and Ruby frameworks at times can feel like a paint by numbers puzzle - and I don't mean that to be complimentary under any interpretation.
That doesn't make good web development "easy", nor does it diminish the value of good ideas and fast product development.
So I see a real argument for the code that builds an application in the simplest way possible being the “realest” form of coding.
I just want to code algorithms and data structures and solve hard domain problems. Not solve difficult programming and operations problems in order to deliver on what is often (at least superficially!) simple domain problems. I think that sweet spot is in games and desktop development.
I don't want to do anything where drivers or hardware keeps sucking the life out of me. I also don't want to do anything where I need to read a dozen spells into some cloud shell in order to get a hello world from an app running on sixteen different machines.
This sang to me, friend. Someone suggested consulting once, to me, and I see it quite a bit in forums where the topic of engineering malaise is brought up.
Problem for me is that puts me a few layers closer to the business types and represents the exact opposite direction I want to go in terms of day to day interactions. Not that I think they're bad folks, I understand they have a part to play in the business same as I do; it's just there's a massive misalignment between "the business" and myself as an engineer in terms of...I guess comparative levels of tolerance for bullshit?
Something, ain't it? Been doing this for 22 years, management for the last 8 and I still can't find a better way to put that.
I had a job like this my whole career. I’m describing the area I want to stay in :)
It's a shame the industry itself sucks. Take a look at EA-spouse and the latest crunch for Cyberpunk - it sucks all enthusiasm out of you.
It is more satisfying to work at machine's native speed. Computers are _fast_, especially when you use all CPU cores. ES6 is not bad, but Rust is joy to use. Tooling is solid, and doesn't need periodical `rm -rf node_modules`. It's easier to peel off and debug compile-time abstractions than runtime/VM abstractions.
However, you mention "politics" and freedom how you program. I don't think that has anything to do with whether you write high-level or low-level code. If you work for an organization, you'll have to deal with management, whether you write JS widgets or firmware for a microwave. Programmers bikeshed at all levels, if it's not whether to use CSS-in-JS, then whether to use spinlocks.
The real, technical constraints of bare metal have sparked decades-long microkernel flamewars.
And then you're at the mercy of your customers!
Not necessarily. With consumer-facing products, especially if they're mass market and the price of a single subscription is low, the customers can't make any big demands (in fact, they can, but you can safely ignore them).
I'm part of a very small team that has a reasonably successful SaaS product aimed directly at consumers. We intentionally priced it very low so we can stay in control of the product, and not be at the mercy of our customers.
in general, everything's exactly the same imo
so if you're interested in the field, go for it
Usually you're working on different types of problems. The technical dangers are different, the type of business logic you're implementing is different, and the types of new things you'd create are different. There's also a huge ecosystem of tools and techniques that you probably haven't even heard about.
If you're just going to make a CRUD app in C or something, odds are you're going to experience the drawbacks you mentioned without much extra benefit. But if you work on actual low-level problems, it can be a completely different world where programming isn't even half the story.
(There was no attempt to compare different languages or their "abstractness" levels and to apply them to the same class of problems. Of course, that almost never the dilemma people are facing.)
It is frustrating to solve problems with the wrong tool. It is a joy when the tool fits the job. Need to write a web server? Not super fun in C. Need to access raw memory and machine intrinsics directly? Not fun in python.
Software is, by and large, software. The things that can make or break satisfaction are to me:
- coworkers, clients, and the political environment. This is completely independent of the technology and project requirements. As a consultant I did very different projects with the same co-workers.
- iteration and debuggability. The best for this, for me, is C# in Visual Studio. GDB is an acceptable substitute for the C family. Web applications .. the browser is OK, but now your app is always and forever in two pieces, and you can't cross-stacktrace. Many embedded environments are bad at this; you may find breakpoint debugging and printf debugging are both impossible. Or you have to do an Elon Musk and recreate the crash cause from the smoking debris field.
- fundamentally interesting problem and ownership of work. This can be tricky, but when you get to do things your way and produce great results that can be very satisfying. Again, completely independent of language; depends what impresses you and your circle.
Do you want something you can talk about at parties? I met the last surviving maintainer of the Aston Martin Lagonda's bonkers 1980s CRT-based 6V digital logic dashboard.
I have only a very few times got to choose the language for a new project, or indeed start a new greenfield project at all.
I enjoy the low-level work more, but that's more of a personal preference.
You can for sure solve high-level problems easier, being below it on the abstraction hierarchy, but your work just comes with new problems. I have never beat my head harder against the wall the trying to figure out something that was a chip bug that eventually becomes new errata (if you're lucky enough for the chip vendor to answer your emails in the first placeti get the errata reported).
It doesn't really protect you from politics unless you're the only low-level person in your shop and no one understands what you do, in which case you just stagnate instead. If you're not careful you tend to get paid less too since high-level folk tend to be closer to the flow of money.
I like the work, and wouldn't trade it, but it's not a panacea to engineering org issues in your career.
FPGAs are fun, because you can design logic circuits and hardware, but with some exception you will have to deal with huge proprietary toolchains and closed source 'compilers'. It offers a pretty unique way to solve problems, which is fun to do. Like playing with a tiny but powerful electrical circuit toolbox.
With micro controllers you might have to deal with proprietary compilers, libraries and IDEs but to a lesser amount than with FPGAs. Still fun to get something to blink, move or a display to work.
If you work on micro processors, then you could work on bootloaders, where you more or less concern yourself just to get something working enough to boot another system, or with OS kernels, where you work in a bigger system infrastructure in order to allow as much of the hardware as possible to be available for the user-land by writing drivers. Its challenging because you work on a small gear in the middle of a huge clockwork, that is mostly not written by yourself, and you will struggle to understand.
There are also many special processors you could implement stuff for like DSPs. There you will feel the language limitations but they are very interesting and odd beasts to work with.
Low-Level is mostly about reading hardware and often software documentation (and finding errors in those or the hardware itself) to figure out how to do certain things, and then if something doesn't work, printing out registers to figure out which bit is wrong. Or having to use oscilloscopes or logic analyzers to debug. This certainly isn't everyones cup of tea, but still very rewarding when you actually start to understand how the pieces fit together and things work.
1. Tools are shit sometimes
2. fixing bugs no-one else understands (and so you don't receive proper praise - because manager might not even understand what you did)
3. Literally spending weeks on insane hardware issues, that requires sifting through 1000 page documentation.
4. harder work - less pay (depending on company)
The major positive is it's extremely satisfying when you do fix the things..
That's because there's a lot more choice when you're working at a higher level. If you're working with an embedded system with an AVR, you're choices may be C/C++ and Assembly-- No need to get into an argument over the latest shiny framework.
You'll never really be isolated from office politics or poor management though. Things like feature creep or changing specs etc. also won't go away.
It's been a fun and challenging transition, but my job satisfaction has mostly been determined by the people I work with rather than the subject matter.
Specifically to your points, I haven't really seen any major difference in the ratio between routine and creative programming at the high and at the low level. The amount of politics and business requirements has also been quite similar.
I've been working mostly with Arm Cortex-M, like STM32. Give it a shot - it'll at least broaden your horizons. The Rust toolchain and peripheral access crates make getting started easier than with C toolchains.
It's easy to spin embedded into a multi-domain learning experience. Ie, firmware, PCB design and ordering, enclosure machining, PC-based update tools, part choosing etc.
It feels...liberating: If I think of an electronic device I'd like to exist, I have the tools available to make it happen. Nothing I've done in Python, Javascript, etc provided this.
When you start interacting with hardware, a lot of easy stuff can become more difficult (testing, deployment, debugging, ...) and some difficult stuff becomes easier (number of platforms to support, control over the product, scope of the project, ...). Everything is generally slower when you have to interact with hardware.
Politics is still there as others have talked about, and the complaints about the tools become more "do we really have to code in ANSI C with a proprietary toolchain?".
I'm not saying don't go for it, it's a really nice domain and IMO there's a satisfaction that's unmatched by pure software when you see your hardware project comes to fruition; but don't expect the paradise.
Low-level programming does scratch that itch of "hacking" stuff, although I have had satisfaction with glueing stuff together with Python in interesting ways.
Lots of “that’s a good idea but it’s impossible to predict all the side effects of that change so no.”
You’ll encounter managers who think you should be shipping at the velocity of a high level programmer.
Overall I enjoy it more but the politics don’t go away.
Building tools is what I truly love to do, they support business use cases, and I understand them, and I build my tools accordingly.
For me it's the most fulfilling kind of software development. You need to know your stack to the metal, but you may not need to tinker with it. For example, I do a lot of Java, but high performance Java, I understand CPU cache access times, and roughly what hotspot will generate, what the OS will do, etc. and have the tools to check my assumptions. I don't do the low level stuff unless absolutely necessary, but I use the tools I have to their limit.
You get deep knowledge of software stacks, and get to do experiments, and get to use all the cool algorithms and techniques, and even develop a few new ones.
The part I love most, is that you keep learning. There's so much to learn and it's wicked fun.
I've done high-level stuff, but I end up liking things like rendering engines or layout engines rather than building a dialog box.
I'm ranting, so TLDR: learn your stack, fully, to the transistors if you can, and pick the spot where you have most fun. Money will follow.
To rephrase the original question: it is not easy to imagine an "Agile" approach to a kernel/network or other kind of systems programming. I would be glad to know there is a chance for a more natural, R&D driven development and bit less fuss about corporate "values".
Thanks guys. Your answers provide some really good and unexpected insights!
When you are doing low level stuff it can be liberating because you don’t have to rely on so many abstractions.
I find most of programming dealing with bad abstractions or choosing a library/language that does 90% of what you want but the final 10% is extremely difficult or requires you to abandon some library/language and start from scratch.
The best way to code IMO is to find a way that allows doing low level coding but includes the possibility to jump into the high level whenever needed.
Unfortunately it's also more political, because the stakes are higher (bigger impact) and for whatever reason more difficult (but talented) personalities seem to get a pass in that world compared to application development work.