I'm sure the technical challenge would be immensely interesting, and I could tell myself that I cared more about accuracy and correctness than other potential hires... but from a moral standpoint, I don't think I could bring myself to do it.
I realise of course that the military uses all sorts of software, including line of business apps, and indeed several military organisations use the B2B security software that my microISV sells, but I think it's very different to directly working on software for killing machines.
One day, I believe during the Iraq occupation, maybe ~12 or 13 years ago, I asked him very directly how he felt about working on these killing machines and whether it bothered him. He smiled and asked if I’d rather have the war here in the U.S.. He also told me he feels like he’s saving lives by being able to so directly target the political enemies, without as much collateral damage as in the past. New technology, he truly believed was preventing innocent civilians from being killed.
It certainly made me think about it, and maybe appreciate somewhat the perspective of people who end up working on war technology, even if I wouldn’t do it. This point of view assumes we’re going to have a war anyway, and no doubt the ideal is just not to have wars, so maybe there’s some rationalization, but OTOH maybe he’s right that he is helping to make the best of a bad situation and saving lives compared to what might happen otherwise.
The US hasn’t been attacked militarily on its own soil in the modern era.
The US military monopoly hasn’t prevented horrific attacks such as 9/11 executed by groups claiming to be motivated by our foreign military campaigns.
I think there is a valid question about the moral culpability of working in this area.
Drones and missiles are definitely a step forward compared to previous technology in many regards, but I can't help but be reminded of people who argued that the development and use of napalm would reduce human suffering by putting an end to the war in Vietnam faster.
For an interesting and rather nuanced (but not 100% realistic) view on drone strikes, I'd recommend giving the 2015 movie Eye in the Sky a watch.
Another issue with drone strikes and missiles is "the bravery of being out of range": it's easier to make the decision to kill someone who you're just watching on a screen than it is to look a person in the eyes and decide to have them killed.
First, I logically agreed that the missiles were supporting our armed services and I believed that our government was generally on the right side of history and needed the best technology to continue defending our freedoms. However, a job, when executed with passion, becomes a very defining core of your identity. I didn’t want death and destruction as my core. I support and admire my college friends who did accept such jobs, but it just wasn’t for me.
Second, I had interned at a government contractor, (not the missile manufacturer), and what I saw deeply disturbed me. I came on to a project which was 5 years into a 3 year schedule, and not expected to ship for another 2 years. Shocked, I asked my team lead “Why didn’t the government just cancel the contract and assign the work to another company?”, her reply, “If they did that, the product likely wouldn’t be delivered in under two years, so they stick with us”. I understood that this mentality was pervasive, and would ultimately become part of me, if I continued to work for that company. That mentality was completely unacceptable in the competitive commercial world, and I feared the complacency which would infect me and not prepare me for the eventual time when I’d need to look for a job outside that company. As a graduating senior, I attended our college job fair, and when speaking with another (non missile) government contractor, I told the recruiter that I was hesitant working for a his company because I thought it wouldn’t keep me as competitive throughout my career. I repeated the story from my internship, and asked if I’d find the same mentality at his company. His face dropped the cheerful recruiter facade, when he pulled me aside and sternly instructed “You should never repeat that story”. I took that as an overwhelming “yes”. So, my concern was that working for this missile manufacturer, this government contractor mentality would work its way into their company (if it hadn’t already), and it would be bad for my long term career. I wanted to remain competitive on a global commercial scale, without relying upon government support.
Software for any system is complex. And it’s quite common for almost every software project to be late on schedule. The Triple Constraint — “schedule, quality, cost: pick any two” doesn’t even fit software engineering in any kind of serious endeavor because it’s mostly a “pick one” scenario.
If you’ve worked on projects where all these three were met with the initial projections, then whoever is estimating those has really made sure that they’ve added massive buffers on cost and time or the project is too trivial for a one person team to do in a month or two.
The entire reason Agile came up as a methodology was in recognizing that requirements change all the time, and that the “change is the only constant” refrain should be taken in stride to adapt how teams work.
We really need more exposure for the things that people like that want to silence..
I hope you are not writing about the US government. I don‘t think the US military can be described as protecting our freedoms after interfering and starting wars all over the world in the past. We are sadly mostly the aggressors and not the defendants.
Big projects are hard and they are frequently late. The fact that it is for the government is largely besides the point.
Well, we are the victors, so far. But the war against ourselves is going quite well.
With the exception of nuclear weapons (that's another topic), missiles are designed to destroy one particular target of strategic importance and nothing more. They are too expensive as mass killing weapons, but they are particularly appropriate for defense.
Without missiles, you may need to launch an assault, destroying everything on your way to the target, risk soldier lives, etc... Less accurate weapons mean higher yield to compensate, so more needless destruction.
War is terrible, but I'd rather have it fought with missiles than with mass produced bombs, machine guns, and worst of all, mines.
Case in point: currently the country with the best army in the world is also the one going the most at war.
I've been in a similar situation, and I think there is something important to think about: Assuming you'd be working for the defense of a country with a track record of decency (at least a good fraction of the time anyway), you have to decide what people you want taking those jobs.
Is it better that all of the people with qualms refuse to take the positions? ie Do you want that work being done by people with no qualms? Because that sounds pretty terrible too.
Yes, this is the kicker for me. My country does not have such a record. If it did, the hypothetical quandry would still exist, but would be much diminished.
At one stage in my career I had an opportunity to go work for Betfair. I knew several people there and could bypass most or all of the interview process. At the time a rapidly growing on-line gambling company, wasn't quite the major company it is now. They were paying about half as much again over my existing salary, and technology wise it would have been a good opportunity.
I ended up having quite a long conversation with a few co-workers around the morality of it. I was against it, for what I thought were pretty much obvious reasons. The house always wins, gambling is an amazing con built up on destroying lives. I don't want to be a part of that, much like I wouldn't work for a tobacco company, oil company etc. Co-workers were taking what they saw as more pragmatic perspective: Gamblers gonna gamble, doesn't matter if the site is there or not.
If you’re on a bad losing streak, they’ll send a host over to offer you tickets to a show or a buffet. The goal is to get you AWAY from gambling. They know you’re an addict, but want to keep the addiction going.
That’s where they cross the ethical line.
Unless you are an extreme pacifist (which is a perfectly reasonable thing to be), you'll acknowledge the legitimate need for the existence of an army in your country. In that case, the army better be equipped by missiles than by youths carrying bayonets. Then, there's nothing wrong in providing these missiles with technologically advanced guiding systems.
On the other hand, if I worked in "algorithmic trading" or fancy "financial instruments" I would not be able to sleep at night without a fair dose of drugs.
If they were for defense only, I might be able to do it. But instead they are sold to any government with the means to pay, regardless of their human rights record or how they will be used (e.g. Saudi). Aside from selling them on, they are used in conflicts that are hard to justify, beyond filling the coffers of the rich and powerful. Take the latest Iraq war for example: started based on falsified evidence, hundreds of thousands dead, atrocities carried out by the west, schools bombed, incidents covered up...
Given these realities, I just couldn't do it.
My original musing was more thinking along the lines of an ideal world, where I trusted my government; I'm still not sure I could do it.
The technical work was super interesting. Everyone I spoke to was plainly super sharp, and not morally bankrupt. I fielded similar moral concerns as you, but truthfully, I don't really have much of a personal ethical problem with it. I was a little more concerned at having to explain it to all of my friends, many of whom are substantially more liberal leaning in political views than I am.
Perception, and the pay cut I'd have to take from my current work, ended up being the major things that stopped me from taking it.
While it would undoubtedly be interesting from a technical standpoint, there is a serious moral conundrum - even if it was an ideal world where you trusted your government not to start wars based on flimsy or falsified evidence, start wars for profit, or sell weapons to less scrupulous governments.
Why? The more precise missiles are, the better. If no-one agreed to build missile guidance systems, we'd still have carpet bombing and artillery with 100m accuracy.
Right now, for example, Saudi Arabia is bombing Yemen with American-made bombs, and Turkey is using German tanks and Italian drones to grind Kurdish towns and villages into rubble in Syria.
That said, I think any software development which involves the government aren't fun at all for all the bureaucracies and inefficiency.
Where can someone (i.e., in my case a software engineer who's working with Kotlin but has used C++ in his past) read more about modern approaches to writing embedded software for such systems?
I'm asking for one because I'm curious by nature and additionally because I simply take the garbage collector for granted nowadays.
Thanks in advance for any pointers (no pun intended)!
I currently work on spacecraft flight software and the only real advance on this project over something like the space shuttle that I can point to is that we're trying out some continuous integration on this project. We would like to use a lot of modern C++ features, but the compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky).
The JPL coding guidelines for C [1] are an amusing, first-hand read about this stuff. Not sure if you would qualify them as "modern approaches".
[1] https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Dev...
In other words, that sounds like a system where dynamic memory management is significantly riskier and harder to test than usual!
Why not static allocation, but sharing memory between the greedy chunks of code that can't run parallel to each other? (I assume these chunks exist, because otherwise your worst-case analysis for dynamic memory would be exactly the same as for static, and it wouldn't save you anything.)
Why would you tune it? Because, if you set it high enough, then for all your short-lived actors, memory allocation will become a no-op (= bump allocation), and the actor will never experience enough memory-pressure to trigger a garbage-collection pass, before the actor exits and the entire process heap can be deallocated as a block. The actor will just “leak” memory onto its heap, and then exit, never having had to spend time accounting for it.
This is also done in many video games, where there is a per-frame temporaries heap that has its free pointer reset at the start of each frame. Rather than individually garbage-collecting these values, they can all just be invalidated at once at the end of the frame.
The usual name for such “heaps you pre-allocate to a capacity you’ve tuned to ensure you will never run out of, and then deallocate as a whole later on” is a memory arena. See https://en.wikipedia.org/wiki/Region-based_memory_management for more examples of memory arenas.
I think pools and arenas mean pretty much the same thing. https://en.wikipedia.org/wiki/Memory_pool I’ve mostly heard this discussed in terms of pools, but I wonder if there’s a subtle difference, or if there’s a historical reason arena is popular in some circles and pool in others...?
I haven’t personally see a per-frame heap while working in console games, even though games I’ve worked on probably had one, or something like it. Techniques that I did see and are super-common are fixed maximum-size allocations: just pre-allocate all the memory you’ll ever need for some feature and never let it go; stack allocations sometimes with alloca(); and helper functions/classes that put something on the stack for the lifetime of a particular scope.
A pool or slab allocator separates allocations into one of a range of fixed-size chunks to avoid fragmentation. Such allocators do support object-by-object deallocation.
Totally agreed that they aren't required for shipping great console games (and they're really hard to use effectively in C++ since you're pretty much guaranteed to have hanging references if you don't have ascetic levels of discipline). This is mainly just meant as a "here's an example of how they can be used and are by at least one shop".
I'd say that arenas are kind of a superset of both what you and I are talking about.
From that standpoint, you could also categorize arenas on a priority basis. This one is for recovery operations, this one for normal operation, and whatever is left for low priority tasks.
> A region, also called a zone, arena, area, or memory context, is a collection of allocated objects that can be efficiently deallocated all at once.
No, that's a cache miss.
Edit; ofcouse HN reacts pedantic when I claim good programmers always consider memory leaks wrong. Do I really need to specify the obvious every time?
In this case you normally want to allocate an arena yourself.
And now the compiler can no longer be embedded into another application, e.g. an IDE.
It's a reasonably pragmatic way of thinking, but beware the consequences. One benefit of working with custom allocators is that you can have the best of both worlds. Unfortunately, custom allocators are clumsy to work with.
In theory, you could have a cluster of identical nodes each handling client requests (i.e. behind a load balancer). Each node would monitor its own application memory utilization and automatically cycle itself after some threshold is hit (after draining its buffers). From the perspective of the programmer, you now get to operate in a magical domain where you can allocate whatever you want and never think about how it has to be cleaned up. Obviously, you wouldn't want to maliciously use malloc, but as long as the cycle time of each run is longer than a few minutes I feel the overhead is accounted for.
Also, the above concept could apply to a single node with multiple independent processes performing the same feat, but there may be some increased concerns with memory fragmentation at the OS-level. Worst case with the distributed cluster of nodes, you can simply power cycle the entire node to wipe memory at the physical level and then bring it back up as a clean slate.
In this case I assume that a massive amount of testing mitigates these issues however.
In this particular case, correctness was not primarily assured by a massive amount of testing (though that may have been done), but by a rigorous static analysis.
In postgres memory contexts are used extensively to manage allocations. And in quite few places we intentionally don't do individual frees, but reset the context as a whole (freeing the memory). Obviously only where the total amount of memory is limited...
It may be unwise to overide static analysis (a leak is found) with hueristics (the program won't run long enough to matter)
I agree that in general leaking resources is bad, but sometimes it is good enough by a large margin. Just a guess.
My favorite trick to optimizing some systems is to see if I can mlock() all of the data in RAM. As long as it's below 1TiB it's a no brainer - 1TiB is very cheap, much cheaper than engineer salaries that would otherwise be wasted on optimizing some database indices.
And a lot of languages, and for sure newer version of the JVM, do exactly that, they don't free memory, and doesn't run the garbage collector since the available memory gets too low. And that is fine for most applications.
For those, it's often the case that they allocate-only, and have a cleanup block for resources like file handles which must be cleaned up; any error longjmps there, and it runs at the end under normal circumstances.
This is basically using the operating system as the garbage collector, and it works fine.
https://news.ycombinator.com/item?id=8305283
https://lists.gnu.org/archive/html/coreutils/2014-08/msg0001...
It's not really kosher, but why not just keep around a fresh process that they can continually fork new handlers from?
I imagine the various long-running PHP node-ish async frameworks curse this history. Though PHP 7 cleaned up a lot of the leaks and inefficient memory structures.
My best guess is a security guarantee.
Long-running worker threads came a long time later, and were indeed intensely criticized from a security perspective at the time, given that they’d be one use-after-free away from exposing a previous user’s password to a new user. (FCGI/WSGI was criticized for the same reason, as compared to the “clean” fork+exec subprocess model of CGI.)
Note that in the context of longer-running connection-oriented protocols, servers are still built in the “accept(2) then fork(2)” model. Postgres forks a process for each connection, for example.
One lesser-thought-about benefit of the forking model, is that it allows the OS to “see” requests; and so to apply CPU/memory/IO quotas to them, that don’t leak over onto undue impacts on successive requests against the same worker. Also, the OOM killer will just kill a request, not the whole server.
In the old cgi-bin days, every web request would fork and exec a new script, whether PHP, Perl, C program, etc. That was replaced with Apache modules (or nsapi, etc), then later, with long running process pooled frameworks like fcgi, php-fpm, etc. Perl and PHP typically then didn't fork for every request. But did create a fresh interpreter context to be backward compatible, avoid memory leaks, etc. So there's still overhead, but not as heavy as fork/exec.
It's not great for maximizing performance but it's not 100s of milliseconds either, forking doesn't take long; what is slow is scripting languages loading their runtimes, but you can fork after that's loaded. If hardware is cheaper than opportunity cost of adding new features (rather than debugging leaks) it makes sense.
This whole procedure appears to be a bit unbelievable. And we're not even talking about code/system maintainability.
Not sure was it pun or no pun intended, but you gave me a good laugh.
Why? I could calculate the average amount of leaking of a program much easier than I could find all the leaks. Calculating just involves performing a typical run under valgrind and seeing how much was never freed. Do that N times and average. Finding the leaks is much more involved.
On the other hand, you can trivially calculate how many measurements you make per unit time, and multiply that by the size of the measurements to upper-bound your storage needs. Hypothetical example: you sample GPS coordinates 20 times per second, which works out to ~160 bytes/sec, 10000 bytes/min, or around 600KB for a full hour of flight. Easy to calculate - hard to fix.
Memory usage is discrete, not continuous. It's not as simple as calculating the safety factor on a rope.
a) calculating maximum leakage
b) doubling physical memory
instead of just fixing the leaks? Was it to save cycles? Prevent memory fragmentation? I feel this story misses the details that would make it more than just a cute anecdote.
No need for garbage collection, no need for "memory management". Not shoddy work. An expression of "YAGNI". The interesting thing (in my opinion), is the realization. The teller of the story went to the trouble of discovering that memory is leaking. She could have simply asked before engaging the work.
FredW
The memory is going to fragment no matter what you do.
If all software is built only to solve the problem at hand, it will take less time to develop, be less likely to have bugs, and perform better.
It isn't clear that coding for reuse is going to get you a net win, especially since computing platforms, the actual hardware, is always evolving, such that reusing code some years later can become sub-optimal for that reason alone.
IISi’s lawyers claimed on September 7, 2010 that “Netezza secretly reverse engineered IISi’s Geospatial product by, inter alia, modifying the internal installation programs of the product and using dummy programs to access its binary code [ … ] to create what Netezza’s own personnel reffered to internally as a “hack” version of Geospatial that would run, albeit very imperfectly, on Netezza’s new TwinFin machine [ … ] Netezza then delivered this “hack” version of Geospatial to a U.S. Government customer (the Central Intelligence Agency) [ … ] According to Netezza’s records, the CIA accepted this “hack” of Geospatial on October 23, 2009, and put it into operation at that time.”
Reality is always more absurd, government agencies remain inept and corrupt even when shrouded in secrecy to cover up their missteps, and by the way, Kubernetes now flies on the F16.
I think one of big problems in software development is that nobody measures the half-life of our assumptions. That is the amount of time it takes for half of the original assumptions to no longer hold.
In my limited experience assumptions half-life in software could be easily as low as around one year. Meaning that in 5 years only 1/32 of original architecture would make sense if we do not evolve it.
By design, there was no memory management. The memory was only ever allocated at the start and never de-allocated. All algorithms were implemented around the concept of everything being a static buffer of infinite lifetime.
It was not possible to spring a memory leak.
But there are whole classes of applications that are also mission critical -- an example might be software driving your car or operating dangerous chemical processes.
For automotive industry there are MISRA standards which we used to guide our development process amongst other ideas from NASA and Boeing (yeah, I know... it was some time ago)
Imagine a simple example of a webapp and number of user sessions.
Instead of the app throwing random errors or slowing down drastically, you could have a hard limit on the number of active sessions.
Whenever the app tries to allocate (find a slot) for a user session but it can't (all objects are already used), it will just throw an error.
This ensures that the application will always work correctly once you log in -- you will not experience a slowdown because too many users logged in.
Now, you also need to figure out what to do with users that received an error when trying to log in. They might receive an error and be told to log in later, they might be put on hold by UI and logged in automatically later or they might be redirected by loadbalancer to another server (maybe even started on demand).
When you start doing this for every aspect of application you get into situation where your application never really gets out of its design parameters and it is one of the important aspect to get an ultra stable operation.
To be sure, it is a unique environment, in which you know for a fact that your software does not need to run beyond a certain point in time. And in a situation like that, I think it is OK to say that we have enough of some resource to reach that point in time. (It's sort of like admitting that climate change is real, and will end life on earth, but then counting on The Rapture to excuse not caring.) But that's not what's going on here. It sounds like they weren't really sure that there would definitely be enough memory.
Static or never-reclaimed allocations are common enough in embedded code.
> they had calculated the amount of memory the application would leak in the total possible flight time for the missile and then doubled that number.
It has limited application, but there is a more common variant: let process exit clean up the heap. You can use an efficient bump allocator for `malloc` and make `free` a no-op.
(I am unable to find a link that talks about that, however).
In general, throwing away at once the set of the things together with the structures that maintain it is always faster than throwing away every item one by one while maintaining the consistency of the structures, in spite of the knowledge that all that is not needed at the end.
An example of arenas in C: "Fast Allocation and Deallocation of Memory Based on Object Lifetimes", Hanson, 1988:
ftp://ftp.cs.princeton.edu/techreports/1988/191.pdf
Windows has always been my daily drivers, and I really do like it. But I wish deleting lots of files would be much, much faster. You've got time to make a cup of coffee if you need to delete a node_modules folder...
Nobody is claiming that this was done for reasons of good software design. It's perfectly reasonable to suspect it was done for reasons of cost or plain negligence.
There's a reason tech workers protest involvement of their firms with the military. It's because all too often arms are not used as a deterrent or as a means of absolute last resort, but because they are used due to faulty intelligence, public or political pressure, as a means of aggression, without regard to collateral damage or otherwise in a careless way.
The whole point here is the blase way the technician responded, "of course it leaks". The justification given is not that it was necessary for the design, but that it doesn't matter because it's going to explode at the end of its journey!
Garbage collection makes the performance of the code much less deterministic.
A lot of embedded loops running on embedded in-order cpus without an operating system use cycle count as a timing mechanism etc.
There exist options between no reclaim and using a garbage collector which could be considered, depending on the exact technical specifications of the hardware it was running on and the era in which it happened.
But retrofitting technical reasoning about why this may have been done is superfluous. The contractor already said why they did it, and the subtext of the original post is that it was flippant and hilarious.
There are a lot of strategies to apply garbage collection and they are often used in low level systems too like per-frame temporary arenas in games or in short lived programs that just allocate and never free.
Electronics components for trajectory tracking and guidance for a particular missile weren't running fast enough, namely the older CPU that the software was targeting. The solution to this was to overclock the CPU by double, and redirect a tiny amount of the liquid oxygen that happened also to be used in the propellent system to cool down the electronics.
This apparently worked fine - by the time the missile ran out of LOX and the electronics burned themselves out, it was going so fast on a ballistic trajectory that it couldn't be reasonably steered anyway.
The telemetry for the self destruct was on a different system that wasn't overclocked, in case of problems with the missile.
I just can't stop laughing over this "ultimate in garbage collection". What a guy.
Btw we dealt a lot with Rational in the 90's. I might have even met him.
The always-leak approach to memory management can also be used in short-lived application code. The D compiler once used this approach [0] (I'm not sure whether it still does).
[0] https://www.drdobbs.com/cpp/increasing-compiler-speed-by-ove...
(Not saying that the manufacturer was necessarily wrong in this case and doubling the memory might have added a tiny manufacturing cost to something that was much more expensive)
Brilliant.
