The most likely explanation would be that there are hardly any mainframe developers. But IBM's website proclaims that "ninety percent of Fortune 500 companies rely on the IBM Mainframe" and "eighty percent of the worlds corporate data resides or originates on the mainframe." If that's true then surely there must be a reasonable number of mainframe developers out there? And if there are, then why is there so little info for them on the web? Is the knowledge proprietary and secret? Or is it just somewhere other than the usual places?
I would be really interested to hear from anyone with knowledge about this.
They weren't the types to go looking on StackOverflow or Youtube for answers.
They would also help train new joiners, and IBM offers good training too.
The kinds of systems running on mainframes don't lend themselves to copy/paste from StackOverflow type programming. It's a lot of credit-card and banking transactions where you want experienced people writing and helping on it.
Secondly, even if IBM is selling more than before, that's still an incredibly tiny fraction of the computing scene. Companies don't tend to have more than one (or possibly 2, with one for DR mainframes). So, that 80-90% of the Fortune 500 is really just 1000 mainframes total. Generously, if every Fortune 500 had a pair in each geography (roughly EU, Americas, Asia for big corps), that ends up being 6, so 3000 globally, in production. There are also some spread around academia, government labs, etc. But the numbers are dwarfed by standard x86 machines. eg. at one bank we had "the mainframe" which was actually one hot and one DR, and then 40k windows servers and 40k linux servers. Similarly, there were thousands of engineers in technology, but the mainframe team with single digits.
Given the wealth of information out there, even if there was "a lot" of information for some value of lots, it's dwarfed by the incredible amount of standard tech info.
As others have posted, your estimates are WILDLY off.
First, fortune 500 companies dont own "two" mainframes, but a LOT more.
Next, many non-F500 companies own a lot of them as well.
Years ago I worked for a local cable company in Canada.
Not a "F500" company, owned at least two mainframes for the "head office" and each different 'subdivision' owned their own mainframe.
> Really just 1000 mainframes total.
I'd be willing to bet that any major city has more within its borders.
Multiply his estimate to 1000 per F500 company ... that's only half a million
Put a pair in every major geography ... that's only 5,000,000
And, for kicks and giggles, multiply by another 10 for funzies
That 50,000,000
Yes, it's "a lot" ... but it's nothing compared to commodity hardware
According to a quick web search, Google alone had in excess of 1,000,000 x86 servers about a decade ago
That's one (large) company
I've worked with myriad companies who're running 10s or 100s of 1000s of x86 servers (let alone x86 desktops/workstations/laptops) - with nary a mainframe in sight (or maybe a couple ... which is on the order of "none" next to 100s of 1000s or millions of x86 boxes)
Per Microsoft [0], there are 1.4 billion Windows 10 and 11 devices (yes, I know that's not a server count, but it speaks to the orders of magnitude difference between x86 devices and mainframes ...even with my generous estimates posted above)
-----------
[0] https://news.microsoft.com/bythenumbers/en/windowsdevices
There are F500s that have 1k mainframes each, today, in 2022.
The equivalent from IBM is called "IBM Z Development and Test Environment" and the license for that costs $5,540 per user per year. There was (maybe still is?) a "learners edition" that you have to contact them to "see if you qualify" and it is a far more reasonable $120/year (there are no links on IBM Marketplace to even try to order it). This includes several compilers and 3 databases: CICS, DB2 & IMS.
IBM has a very different model for supporting developers than Microsoft does with MSDN.
Links:
https://en.wikipedia.org/wiki/Hercules_(emulator)
https://www.ibm.com/products/z-development-test-environment/...
https://ibm.github.io/zdt-learners-edition-about/
https://old.reddit.com/r/mainframe/
Ballmer being excited: https://www.youtube.com/watch?v=wNy2zBG79zU
After all, IBM wants you to pay IBM to do this stuff for you.
It's not like Bob from IT is going to play around with mainframe coding for a year or so and convince the boss to buy one.
AFAIR, they offer the possibility for a one year free access to a node (Linux/IFL?). I do not remember if it is part of IBM Z Xplore and I didn't find it again with a quick search, but if you contact them they for sure will tell you more.
Edit: I just found https://www.openmainframeproject.org/
"Hello World on z/OS" (2018) https://medium.com/the-technical-archaeologist/hello-world-o...
Hn discussion:
https://news.ycombinator.com/item?id=17642846
I seem to recall IBM launched z/os in the cloud as partly a production service, and partly an education/hobby resource a while back - but I'm not sure if that's what's now:
https://www.ibm.com/cloud/pricing
I think it came with access to db2 as well?
Ed: I was probably thinking of IBM LinuxOne:
https://developer.ibm.com/articles/get-started-with-ibm-linu...
"Learner's Edition is currently being updated and will return soon"
Let's hope "soon" is actually soon :)
[1] https://www.ibm.com/products/z-development-test-environment/...
Interesting comparison, because when I was teaching myself to code in the early 00s, that's how I perceived Microsoft.
MSDN was a paid subscription, and the tools you needed to write MS software were also paid.
Hence me ending up forming a career on open source software, and a generation of programmers who still turn their nose up at Microsoft languages.
StackOverflow in itself is the answer to the question "who you gonna call" -- because you might be alone, or you might be working on a small team where no one has better expertise, and usually you are working with open source and so no one is obligated to pick up the phone at the software "manufacturer". This scenario does not play in a mainframe environment: you are going to be part of a larger team, full of literal graybeards and when push comes to shove then you do have a contact number: your employer is paying absolutely ridiculous amounts of money to the software company and they will have a contact person who will pick up the phone.
Not only that but that redundancy is hidden from everything. Already decades ago the Hungarian electricity distributor had a "RAID" storage unit where one bunch of disks were at the capitol, the other bunch connected with fiber at another city -- and you talked to this thing as if it were an ordinary disk. Short of an invading army nothing could cause data loss on that thing -- simple aerial bombing wouldn't because the unit in the capitol was tens of meters underground... that's not concerns you have on your average web app.
I mean, maybe not in an old-school 1970s batch-SIMD mainframe environment with limited storage memory.
But in a modern mainframe environment, where
1. every tenant being processed is hitting their own virtualized environment locked into their own legacy versions of everything (think: separately-SCMed "Enterprise" deployments of SaaS software — just all living together on one machine cluster rather than on-prem per customer); and
2. you've got the spare storage capacity to do everything via async-queued CQRS/ES + writing secondary representations and cubes to a local data lake, rather than ever updating anything in place...
...then why not do "agile" blue-green deploys of your ACH reconciliation logic, starting with the smallest tenants first?
If you ever produce a bug in such an environment, you can always just: stop the job; emit tombstone records for all the CQRS Commands it produced so they'll be ignored on "outbox processing" rather than pushed to consumers; fix the bug; up the version number for OLAP report generation, invalidating + requiring recomputation for any reports produced from the data in the reporting period; and then start the job up again. No external system will ever have to know, because your own system should be set up such that no other external system is synchronously dependent on its outputs. (This is half the reason things like ACH are designed with the async "flex" they have.)
And, in fact, assuming you've got the spare CPU power (which you do), you can even run entire reporting / integration cycles twice — once under the old logic, once under the new logic — and examine the resulting queued commands + reports to see that they all come out the same, before allowing anything to go through. You can do this not just in production, but experimentally, against a corpus of all old input data for each tenant (which you've almost certainly kept around.) And, as such, you don't even have to stick with those virtualized legacy environments that mainframe architectures like z/OS were designed to enable — you can actively migrate clients away from them (even automatically, with CI/CD-triggered graduated rollouts et al) as soon as you can prove that their workloads (both present and historical!) won't be affected by the change. Each tenant can be re-pinned over time to the newest release that can be proven to be equivalent to the original release provided to them on all known data.
One of the key insights about mainframe programming, is that the software architectures of the kinds of systems deployed on mainframes, are designed to be resilient against individual bugs; or, one might say, against individual programmers who don't know what they're doing. Systems engineering in e.g. finance, is to bad individual-component / individual-release logic, as NASA's Mars-rover systems engineering is to bad hardware: designed under the assumption of the potential for failure, and designed with built-in remediation strategies for that failure.
* The documentation for mainframe stuff is incredibly comprehensive compared to what most people are used to, so there's less need for something like Sewer Overflow.
* The code people are working on is likely to be proprietary, so no GitHub.
* Mainframes are often used in security-conscious environments where "openness" is not a core value and might even be considered negative.
* People working in that area might like being part of a relatively closed community (lower supply = better pay) so they don't advertise.
And then there's just critical mass. Since there aren't already a lot of mainframe programmers Out There, and mainframe programmers' work has few technical "touch points" with the rest of the world (e.g. different languages, libraries), it almost makes more sense to flip the question around.
Why would there be more info on the web about IBM mainframe programming?
So even if you do see a mainframe programmer in the wild asking a question, you probably won't recognize it.
On the down side it's much harder to bootstrap an environment for doing side projects on the mainframe. The hercules emulator makes it possible locally but it's a pain [0]. However, I think IBM provides sandox's for people to mess around with and all that requires is some minimal sign up and a TN3270 terminal emulator.
IBM is also a very painful vendor. They’re in the audit and share buyback business. They struggle to innovate. (One can argue with their track record on innovation they really should just milk the existing contracts and buy back shares)
No, there are lots of companies that need to modernise their tech stack :)
I've been using a personal and business bank account with no mainframes involved for about 6 years now. Golang, microservices, Cassandra/Postgres. Progress is great to see.
During my vocational training, I did a bit of work with mainframes and found a forum back then I liked, but my work (and the forum) was mostly about sysadmin-stuff, and the little coding I did was in Perl.
There is a sizable library of RedBooks on IBM's web site somewhere, but again, the stuff I was interested in was about system administration.
A lot of the culture in the mainframe world developed a long time ago, and in relative isolation from the rest of the IT world, so I suspect by the time Stack Overflow became popular, their communities had already found other ways and places to mingle and pass knowledge on and around.
With hardware, software, and support contract being so expensive, and corporate culture being what it is, I also suspect the threshold to call up the the vendor for support instead of Stack Overflow is much lower than in most other areas.
(Most of this is speculation on my part, I only spend a few months in this fascinating corner of IT. If I'm wrong, I'd love to hear to from someone with actual experience.)
Indeed, and not just Java. zLinux runs a great many workloads on M/Fs these days, which looks a lot like a normal Linux env.
Then ofc K8S/docker/etc can run on mainframes, presenting another layer of abstraction.
Modern mainfame programming is just a question of getting to know a slightly different set of tools.
wtf why even plow $$$ into hardware at that point?
The people who operate these mainframes have tremendous access to internal IBM resources. I remember having to stifle my laughter as, during a meeting about a major outage, a seasoned programmer explained that he "took a dump and sent it to IBM" and was expecting to hear back any time.
The channels that exist for this knowledge are expansive and just aren't on the internet, though they do occasionally spill over there.
Edit: I should add that I'd bet they DO have their own web forums, and a lot of them are probably run by IBM itself. (I'm not a mainframer, in case you're wondering)
Reading Red Books has better ROI than trawling for help on the 'Net.
Everybody is welcome to watch Moshix channel on YouTube, though. It's refreshing.
Mainframe programming is not like web programming. There are not multiple frameworks and languages competing for developer attention and rising and falling in popularity every few years. It is much more "there is one way to do it, it's the same way we've done it for 15 years" and that is what you do. There are very few new questions. All the common questions are covered in training and the more uncommon ones are covered in the documentation. If there isn't already a standard way to do what you want to do, you are probably doing it wrong.
Also, I would add that most mainframe programming is flat procedural (no objects, no complex data structures - heck, COBOL doesn't have real strings). So the vendor manuals cover most of what you need to know, as things don't get too complicated and obscure. And there are always colleagues to ask too.
I think this is the equivalent - https://ibmzxplore.influitive.com/users/sign_in
I've had this bookmarked for years, never gotten around to starting it, though. Web development continues to entertain, challenge, and sustain me.
I cut my teeth on A/UX and SunOS so I'm of a certain age, but besides having school accounts for intro/101 courses using 3270 emulation I never really got hands on. For the same reasons that today I still wish Solaris had won, I think I would enjoy certain aspects of the mainframe environment.
I wonder if any of my modern stack skillset would be useful or not.
So that cultural value of keep everything close to the vest and only learn through official channels, still persists in IBM mainframe culture today. Especially since there is little to no "hacker culture" surrounding mainframes since it's not like you can download z/OS and run it on your PC (things like Hercules notwithstanding).
At least before 1980, during the IBM System/360 and IBM System/370 generations, abundant documentation about any aspect of any IBM product was available, much more detailed than any modern documentation.
Moreover, for many of the programs provided by IBM with the hardware, the source code was available, even if they were not open-source in the modern sense, because the users were not encouraged to modify and redistribute them, even if such activities happened sometimes, e.g. within the IBM SHARE user group.
When the IBM PC was introduced in 1981, this policy of providing exhaustive documentation was still active, which is why the IBM PCs were delivered with a ton of manuals, including descriptions of the hardware interfaces and the complete listing of the source code for the BIOS firmware.
Sometime later, a connection on LinkedIn pointed out the following resources:
https://www.openmainframeproject.org/
https://github.com/openmainframeproject
https://github.com/openmainframeproject/cobol-programming-co...
Also, mainframes are really good at high throughput transactional jobs. That's why you see them in banks, transportation, insurance, etc. Big Tech™ doesn't see it as "cool" and are too focused on the Next Big Thing™, so there's not a lot of attention there. Sometimes, boring just gets the job done.
As other people have stated in this thread, IBM Redbooks are what you need and they span the whole range from high level overflow to low level.
The ABC's of System Programming is probably where you want to start, there are 13 volumes I think.
https://colinpaice.blog/2020/06/22/whats-in-abcs-of-z-os-sys...
You just have to know what to look for to find low level stuff - for an example, here is the reference for IBM's High Level Assembler Language (HLASM).
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pag...
> And if there are, then why is there so little info for them on the web? Is the knowledge proprietary and secret?
IMO it is just a cultural thing. A lot of mainframe developers and especially the experts who have deep knowledge have been programming for a long time. Like, started on punch cards while they were getting a math degree.
The IBM culture has always been comprehensive documentation available from the source and predates the internet by a few decades, so the kind of public stack overflow culture we are used to never developed over there.
The book Hackers: Heroes of the Computer Revolution by Steven Levy kind of talks about this, the difference between the mainframe guys and the mini computer guys. Its a good read.
Mainframe people are descendants of the "high priests" that worked on computers before the invention of the mini computer and the culture reflects that to this day.
Example. I used to work at a massive organization that did have some core applications on mainframes. The guys who maintained it retired and they couldn't find anyone to fill the position. So they contracted with IBM to maintain the system. Guess who showed up on site wearing IBM polos? The guys who just retired. Now doing the same job but making even more while doing it.
The other thing to know is; it's really hard to learn if you don't have access to one (obviously). They are from an era when computing was not ubiquitous, there is a whole parallel universe of computing in mainframes where things have different names and conventions, while the basic concepts are the same. It can be very hard to translate.
For me, my time on mainframes was enlightening and interesting, but a major career dead end. I had to go back to entry level in a different stack to get out of it. Glad I did, those jobs are all dead end, miserable, cost centers. Every company who has mainframe applications would love to get off them, but they can't because the cost is too high.
Compare this to your typical PC/mobile device which fits on/in a desk/lap/pocket, is practically free in comparison, has a deluge of information available everywhere and is actually useful to most people in everyday life. About the only thing a mainframe really gets you is reliability in a monolithic device which from a cost/benefit standpoint isn't worthwhile for most customers and can be approximated via other means.
I've been inside many organizations that run mainframes and if you're not on one of the mainframe teams, which tend to be tiny relative to other IT groups, it's pretty much plumbing as far as the rest of the organization is concerned... which is the point.
Check out:
https://www.redbooks.ibm.com/search?query=%22system%20progra...
https://developer.ibm.com/?q=z%2Fos
Also search ibm.com for terms like "PL/I", "COBOL", "CICS", "RPG" and so on and you'll likely turn up more of the mainframe stuff. You can, of course, program mainframes in other languages, especially Java. But from an IBM perspective, references to the former batch of terms will generally turn up stuff about their mainframes, or their minicomputer series (mostly what's called iSeries these days, as far as I know. Formerly the AS/400, S/38, etc).
There are also various print books from 3rd party sources that can be found on Amazon. Ex:
https://www.amazon.com/IBM-COBOL-Complete-Guide-2020/dp/0655...
https://www.amazon.com/Quick-Start-Training-Application-Deve...
etc.
The reasons for this are, so far as I can tell, largely what other commenters have already posted. It's just a different culture and way of doing things, and the mainframe world just works a bit differently. And I think a lot of that is because that world is so insular now with IBM being more or less the only mainframe vendor left. I suppose Unisys or Hitachi or Fujitsu or somebody may still make a mainframe here and there, but the market is almost completely dominated by IBM. So the "mainframe way of doing things" is more or less "the IBM way of doing things" and in particular it's a very old skool version of "The IBM way" at that.
Much of the information is locked away in developers' heads and it was difficult for us to find many resources on getting started with COBOL. In fact, most of the COBOL code on GitHub is homework assignments but almost no real-world programs that we could analyze.
Our paper which contributes a defects benchmark: https://austinhenley.com/pubs/Lee2022ICSME_COBOL.pdf
I would also dig through Project Guttenburg https://www.gutenberg.org/ for IBM Mainframe Programming books if you're interested in the history.
https://www.ibm.com/blogs/ibm-training/free-course-announcin...
I am sure there are others out there too.
I've no knowledge of IBM mainframes - but I assume they're sold as part of a large consultancy/training package.
These are what I would call Enterprise environments, IBM, VMware, RHEL. And they all have one thing in common, lack of community documentation. Because in Enterprise environments you're expected to be trained.
And in the IRC channels people mostly talk about politics, or just everyday stuff. Very rarely do they nerd out about Linux or hypervisors. Because most people in there are trained engineers and neither need help, nor are willing to give help to someone.
If you go up to a crowd of enterprise people to ask a question, you better ask a good question. The old adage "there are no stupid questions" does not apply at all. Because these are trained people who have very specific knowledge of their Enterprise systems. They don't mess around with community projects or open source for fun. They do it to achieve a goal.
I know someone who started at IBM about a year ago and he's learning quickly how to program on IBM mainframes. By other senior engineers, and trainers.
> "ninety percent of Fortune 500 companies rely on the IBM Mainframe"
This doesn't say they own them. It's also very vague given they know the exact number.
> "eighty percent of the worlds corporate data resides or originates on the mainframe."
This doesn't say it's the primary store, or is that by volume or dataset count, or what constitutes "data". It doesn't say where the chain ends either. If the user record is created on a mainframe, did everything related to the user originate on the mainframe?
Take it with appropriate amount of salt.
There is Code in a multitude of languages even when it is not Linux, from C++ to Java, but nowadays also nodejs and whatever you want to run in that architcture.
Edit:
Basically, you should look for "System Z" which is the actual name of the machine, and "redbooks" the name of the documentation manuals.
has a lot off z howto books
https://stackoverflow.com/questions/74593657/dsnacics-stored...
Compared to other stacks/platforms, it may seem a dearth, but given the number of programmers working in that area (and as others here have pointed out, given the very different nature of mainframe programming in general), it's no so surprising that there's relatively little info that's searchable on the public internet. TBH I'm still frequently surprised by not being able to find information about a particular error I'm getting with a particular API that I would have thought was in relatively common use even in the C#/.NET world!
https://mainframenation.com/mainframe/how-to-get-a-mainframe...
As for how you'd actually play around with writing usable software.. you'd need access to a mainframe. Basically, the insane complexity of mainframes and the walled garden nature of the ecosystem makes it extremely difficult to teach or to learn.
I worked on the z/OS network stack for a few years (tech support and later some actual development) and it was the most complex thing I've ever tried to learn in my career.
https://www.redbooks.ibm.com/domains/zsystems
IBM's mainframes have some of the best technical documentation for any product line, anywhere, and if you can't find what you're looking for in the redbooks, the expensive support contract that you likely got with your z/Series comes with support that's sometimes described as... aggressive at solving high priority incidents with their internal support databases.
There's also the online manuals that come with z/OS, and you can get a free account via z Xplore: https://www.ibm.com/it-infrastructure/z/education/zxplore
You can also get a Linux LPAR on a s390x machine (LinuxONE, basically a regular mainframe with the processors fused to only run Linux) as part of the free tier on IBM Cloud.
That being said, most real-world deployments have a ton of other stuff running on top of the OS, usually from Broadcom and other close software partners, and dealing with those packages usually requires a support account with those companies as well. The applications themselves are very weird if you're not used to living in a mainframe environment. For example, you can get PKZIP for z/OS, and there is a 437 page PDF describing just the error messages alone.
Generally, learning how to get around in a mainframe environment requires working in a mainframe shop, nearly all of which are very mature and have converged on certain ways of dealing with testing, deployment, upgrades, migrations, etc. These procedures are usually not documented because they're simply well-understood by all the staff, or it would be dangerous for them to be working in the mainframe environment. Learning to use just ISPF, the closest thing z/OS has to a management shell, requires not only learning ISPF, but also all the installation-specific details for a given environment.
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pag...
There's also a z/VM library.
You can fill a large room with the printed manuals.
When I was working on the mainframe, I usually had about 20' of shelves for the manuals I used.
The trick is knowing where to look in which manual.
I characterised my job as being a professional [IBM] manual reader.
There was a gazillion times a programmer came to me with a question and I pulled out a manual.
Source code was also available for many pieces. I made several modifications and patches.
Also maybe you should know about Seattle's 'Living Computer History Museum'. It's been closed for a couple years but I'm ready to visit once they reopen.
https://www.ibm.com/docs/en/zos-basic-skills?topic=sdsf-what...
https://www.ibm.com/docs/en/zos-basic-skills?topic=zos-assem...
And on the application side you are typically dealing with databases and Runtimes(Java mostly) that also exist on Linux and have an large Linux centric community where any legacy programmers and admins kind of drown in the masses.
There is a whole bunch of companies out there with large COBOL and ALGOL code-bases but but as the problem here is less about generic mainframe skills and more about knowing how to work large COBOL or ALGOL projects.
IBM also has very advanced diagnostics tools and error tracking and those were in production use before the internet came along. It makes sense to use those resources rather than the web.
There is lots of content there.
All of the information you need is in the manuals. IBM is exceptionally good at documentation.
Also, the hardware instruction set is set in stone.
a lot of howto books
The other partial reason is that a ton of those conversations happen in old email groups that aren't indexed or index a lot worse than spam sites do. So if you're interested, searching around for mailing lists, users groups, and old forums and there's still good material out there, just buried and mostly catering to folks who know the basics, so a bit tough to parse if you're new to it.
- incredibly proprietary
- heavily customized for the environments they are running in (and boy howdy are there many)
- COBOL/Fortran/4GL not popular or desirable (for younger developers)
2. developerworks site on IBM had some info on mainframes and mainframe programming.
3. It is not "cool".
