Like they have always said, timing is everything and probably this is the project similar to Apple's Newton project that have many goods in it but just born several years too early.
Winix targets RISC architecture and with RISC-V taking off exponentially at the very moment, having RISC biased OS will definitely provide edges and advantages for the platform, similar to x86 quirks to Linux, and Linux taking advantages when x86-32 and x86-64 took off.
This year when Linus was asked for the best Linux achievement compared to other OS, Linus has pointed out the innovative Linux based lock-free filesystem [2]. Winix has built-in innovative POSIX compatible in-memory file-system (IMFS) by default. Imagine an OS with IMFS that's also natively compatible with the increasingly popular Arrow and TileDB in-memory format. With Terabyte (TB) RAM computers becoming the norm in the near future this can easily be the fastest OS with the state-of-the-art filesystem around. Fuschia is another latest OS on-the-block but by focusing on the mobile rather than desktop it will probably optimized for the former, unlike Winix.
[1]https://tianyin.github.io/misc/irrelevant.pdf
[2]https://www.tag1consulting.com/blog/interview-linus-torvalds...
[1] http://www.barrelfish.org/ [2] https://unikraft.org/ [3] https://mars-research.github.io/redleaf
Since then systems software research has been quite vital, in significant part due to Rob himself; in no particular order, relevant developments in systems software research since Rob's paper include Golang, MapReduce, HTML5 (including Web Workers, <canvas>, and WebSockets), Sawzall, Hadoop, Rust, wasm, Fuchsia (as you point out), V8, protobufs, Thrift, Docker, Xen, AWS, Azure, ZFS, btrfs, BitTorrent, Kafka, nearly all of Google's "warehouse-scale computing" stuff, memcached, OpenID, QEMU†, kvm, PyPy, SPARK, Julia and almost all the automatic differentiation stuff, Clojure, iOS, Swift, Factor, AMQP, RabbitMQ, ZeroMQ, Jupyter, the mainstream use of AJAX and Comet, QUIC and HTTP/2, Valgrind, LLVM, Kotlin, Bitcoin, Ethereum, OTR and Signal, Android, Dalvik, reproducible builds, seL4, Zig, Pony, CapnProto, Sandstorm.io, Fastly's fast-purging CDN, the Varnish cache it's based on, the fast SSDs that enabled it, TileDB as you mention (and Parquet), time-series databases like InfluxDB in general, Python 3, Racket, major new developments in ECMAScript, JSON, Z3, entity-component systems, general-purpose GPU computing, Intel ME (for better or worse, mostly worse, it's certainly relevant systems software research), OTR and Signal, Tor, Chrome, Firefox, Node.js, npm, LevelDB, record-replay for time-travel debugging, UBsan and Asan, stack canaries, epoll, io_uring, Qubes, Vulkan, CUDA, Wayland, Haskell's STM, XMPP, DTrace, the whole megillah around the shift to manycore, WPA for Wi-Fi, most of the work in making secure protocols resistant to timing and compression attacks, Git, SyncThing, ownCloud, rsync, zsync, GFS, BigTable, Cassandra, MQTT, and on and on and on. Oh yeah, and also eBPF.
Putting your filesystem on a ramdisk is a good idea but it's hardly innovative.
______
† https://web.archive.org/web/20030601085257/http://fabrice.be...
I am way late to the party, but here goes: In 2007 I started working for a company in Norway that had developed their own database, from hardware and up. It included parallell processing (a few thousand processors when I joined) and everything hosted in memory. It was fast. Boot up took a while though, since it had to load everything from disks to memory.
When I joined they were on the third iteration already, the first version went live back in 1992.
We have since retired the concept since off-the-shelf hardware caught up in terms of speed and lower cost, also it didn't scale very well.
Genode, unikernels like MirageOS, TempleOS, Singularity OS / Sing#, compiler services like Roslyn and Kotlin, MILEPOST GCC, C++ 11+, Tensorflow / TPUs, GPT-3, all of the machine learning in compilers [1] and so much more. I truly think Deep Learning Compilers will be huge.
[1] https://github.com/zwang4/awesome-machine-learning-in-compil...
PL research on the other hand, is in trouble, but that’s a different topic.
I don't know in which world you live but in mine people still mostly buy 4, sometimes 8GB of ram laptops
Not only that, but those terabytes are actually distributed systems running at most a few hundreds of megabytes of ram in each node.
Honestly I have high hopes for it, mostly because I like the colour ^^.
"The GitHub project suggests Fuchsia can run on many platforms, from embedded systems to smartphones, tablets, and personal computers." ("Google Fuchsia - Wikipedia" https://en.m.wikipedia.org/wiki/Google_Fuchsia)
Edit: fixed spelling of the name of the colour.
Also, big area of research right now is how to sandbox applications properly in this era of app stores.
Back on topic:
Systems theory seems to basically be object-oriented, meaning it sometimes has problems dealing with self-reference. This should not be a problem for most practical applications but for things like ecological dynamics the complexity is not user friendly.
Cybernetics seems to be a function-oriented approach to the same problem class. It deals with feedback loops as first-order citizens, which in complex ecosystems like desktop operating systems means it has less of a conceptual impedance mismatch.
The former is the popular discipline while the latter fell out of favor decades ago. Perhaps it is time to dust it off.
Almost all the change log in Linux are Drivers and File Systems. And anything there is new to Linux seems to have come from other OS like Solaris. You have Minix in every Intel chip or seL4 for specific uses.
I mean Windows 11 I was hoping Microsoft would share something about new kernel update or something. or Apple macOS which seems to be in the direction of moving everything to userspace.
Are there anything that is really new that didn't get any coverage?
What is really the point of computing is the interaction and capabilities for the end user which is where the commercial focus is.
The about section says "A UNIX-style Operating System for the Waikato RISC Architecture Microprocessor (WRAMP)". Did you study at Waikato?
I'm curious what you think about tech in general in NZ as well. Compared to the US it feels like there's practically no community :/.
NZ is a very small market compared to the rest of the world, so there is a lot less opportunities here in NZ in terms of number of startups and number of tech firms. The salary is also lower compared to other developed countries. Any software engineer in NZ can easily get a 30-50% salary raise by going to Australia. A lot of talents ran away from NZ because of this.
There are still some exiting companies, for instance, Serko whom I am currently working for is expanding into the international market to seize the post-pandemic opportunities. But overall it's bit stagnant compared to other OECD countries.
I'm not the author but my view of the NZ tech sector is that it's pretty much stagnant. I ended up taking my skills elsewhere as did most of the people I remain in contact with from my UC days. It seems successive governments have put all their eggs into the agricultural export market. That's a real shame.
The toolchain and simulator are also available on GitHub [2], though it looks like the author has their own fork from an older version. A few years ago the the project was ported from a custom board to an off-the-shelf FPGA dev board and the simulator was renamed from "rexsim"/"RexSimulator" to "wsim", hence the difference in name.
Been here 3.5 years now (although I still don't know if 2020 really counts) and the tech sector is great here. Maybe not as cutting edge/well funded/as high salaries as America, but I feel like the work/life balance is better in Europe.
NZ is great but I have a feeling I'll head back there once I'm a lot older; it's beautiful but there are very few opportunities when you're young.
Whereabouts did you go from NZ? I'm working remotely right now and want to leave NZ after travel opens up. Compared to other places in the world NZ seems terrible for a young, single guy in tech.
My Dad taught math at the teachers college and my Mother worked at the University as a faculty secretary, so I was in and amongst University life.
I sometimes wonder what it would have been like if I had stayed and gone to Uni there.
What was the hardest part?
What did you learn that you weren’t expecting?
This approach works better than using a debugger, even on a single-core system, because these kinds of bugs tend to be hard to reproduce and take many iterations. You don't want it to hit a breakpoint a zillion times before it finally shows itself.
And another one, tangential to what you said. Read your code line by line and ask yourself "what would break if a context switch happens right here" for each line.
One somewhat related theoretical observation about concurrency is in some article by Dijkstra (I don't remember the reference right now): he says that debugging using traces (essentially printf) does not work for concurrency, since it is projecting multidimensional data (data present at the same time in multiple processes) unnecessarily linearized onto a single dimension (a sequence of printfs) and then trying to make sense of what is happening. It may not work, even if you print timestamps.
His view was to promote theoretical proofs of correctness of concurrent code, rather than debugging, but to me at least, this is much more difficult.
Make a blog post on the experience. What does the heatmap of changes look like over 5 years? I cant imagine you burnt the candle on both ends for that whole time.
What is the final lines of code?
What blockers did you not anticipate that killed a lot of time? Hardest problem?
I think you've got a treasure of interesting retrospectives to share :)
My favorites for retro-like discussions (if you think it will be beneficial):
1. What did you worry about the most in the beginning that later turned out to be silly/inconsequential?
2. What did you intentionally put off in the beginning thinking it wasn't a big deal that you later regretted not spending more time on?
For a high-level breakdown of the files changed:
https://public-001.gitsense.com/insights/github/repos?p=focu...
681 files were changed, with the most being C/C Header files.
The file age is interesting as it shows 175 files were changed over a 2 year period. And you have about 279 files that were probably just imported and never touched again or were infrequently changed. Note renaming files/directory can skew this number.
And if you are looking for a heatmap, you can look at
https://public-001.gitsense.com/insights/github/repos?p=comm...
The superscript value <number>v besides the file/directory indicates how many versions there are. So the higher the version, the greater the focus. In this case, the kernel, include and fs directories saw most of the activity.
In the future, I will gray out files that no longer exists on the latest tree, but for now you can look at the year beside the files/directories to get an idea of when they were last touched.
And here are the top 20 most frequently changed files
path | revs
-----------------------+------
kernel/system.c | 151
kernel/proc.c | 128
include/sys/syscall.h | 124
kernel/exception.c | 120
include/kernel/proc.h | 95
Makefile | 90
kernel/main.c | 80
user/shell.c | 69
fs/inode.c | 67
winix/sys_stdio.c | 60
README.md | 53
driver/tty.c | 52
kernel/sched.c | 49
kernel/clock.c | 46
winix/wini_ipc.c | 45
fs/fs.h | 44
winix/mm.c | 44
fs/makefs.c | 42
winix/sigsend.c | 42
init/init.c | 41
I felt sad for him - he'd poured his heart into it for more than 10 years (he said) and had no idea of how to turn it into anything other than a hobby project. He hadn't open sourced it, his aspiration was that "Facebook will buy it". I haven't seen him since the pandemic kicked in.
There is nothing wrong with that. It's fun to do things just for fun.
> He hadn't open sourced it, his aspiration was that "Facebook will buy it".
It's hard to believe that's anyone's real motivation for such a project.
What there is something wrong with is criticizing others’ motivations because they seem outlandish to you. It’s fine and even healthy to be so ambitious. Not all of us want to sit around playing with legos assuming that’s the best a reasonable life has to offer.
I hope it does, I don't think it will.
The reason I ask: for the past six months I've been dipping my toes in OS development for x86 with BIOS, SVGA and ATA interfaces: https://github.com/akkartik/mu And I'm way out of my depth. I have this suspicion that x86 might be one of the more difficult environments for device drivers, so I'm curious to hear what other hardware looks like.
The OS is running in a customised RISV hardware in MIPS for educational purposes, you can read more here https://dl.acm.org/doi/pdf/10.1145/1275462.1275470 The simulator https://github.com/halfer53/rexsimulator
The BIOS has already been written for the serial ports, etc. So it is quite easy for me to interact with the devices in the OS.
Do you have a plan behind this or was it just to scratch an itch?
Keep at it, I'm really curious how what you are making will end up.
Would it affect to the speed of execution or development not to be compliant?
Well, I wrote my own operating system.
Sorry.
I did struggle to explain the significance of this project when I was applying for graduate roles, especially to HR people who only look for keywards in CV.
But there are people who appreciate my work.
I got my internship at AWS because my interviewer who had technical background was very impressed with my work, hence the reason I got hired.
Sometimes you just have to find the right audience.
Yes, good hiring managers with sharp vision will always want to hire new grades who is able to write operating systems, compilers or simulators from scratch. Having those essential CS skills are much better than having many buzz words in the resume.
PS: This is not a criticism but genuine curiocity why basic OS projects are so expensive given so much already out there. Also, Linus spent majority of hours each day for a year while this might be just a side gig for you.
Yeah it was a side gig for me, I did spend major of my days for the duration of 2 months last year to work on this.
In fact, I paused on the development of this project for 1-2 year.
But most of the time, it's just 1-2 hour per week side projects on average.
You pretty much have to seek out the knowledge regarding Operating systems in addition to taking on the challenge of learning something like C/C+ or Rust. You have to then break all the ways of thinking about your memory model after being given a GC language.
Understanding hardware is needed to debug specific CPU/IO issues that could be caused by assumptions or lack of knowledge.
Next comes learning the different algorithms needed by an OS in a new language and then implementing them. This takes a lot of time to get down correctly.
It's also a really small and niche community which means you don't have a lot of support from peers that understand the problems you encounter the community isn't an easy one to feel accepted into. It can be hard to get support because you need to be very detailed in your questions or you will be ignored.
The OS is designed for MIPS architecture at this stage, I don't think tcc supports MIPS at the moment.
If yes, then how?
It's currently using wcc to compile the kernel https://github.com/wandwramp/wcc. We can swap it with gcc for sure, but require bit of work, especially the output format and backend.
Winix assumes the output file is in srec format https://en.wikipedia.org/wiki/SREC_(file_format), so we would need to tweak gcc to support this header format and add backend support for WRAMP architecture.
ELF is quite complicated, if I have to do this, I probably just copy some codes from linux.
You have made it POSIX-ish but wouldn't one goal be to create something new ? I imagine that you just wanted to use a bunch of POSIX tools but won't that limit you in some way?
In the very first sentence of the README it states "hobbyist, educational" . That pretty much sums up the OP's motivation.
Why can't learning and exploration be their own motivation?
I'll probably never have a use for it, myself, but I am always in support of passion projects.
Good luck!
I at least got Minix 2 with the printed source code in AST's book. Minix 2 was so annoying on real PC hardware that I had to stick a keyboard repeat rate and delay patch whenever I used it. It was all straightforward and the lab assignments on it were easy because it was built for tinkering rather than efficiency.
My uni replaced FreeBSD as the teaching operating system of choice. They also go rid of Oracle DBMS# for Postgres.
# Many, many moons ago I nearly got Aaron Swartz'ed. This is because I tried browsing "proprietary" docs using a Java-based internet proxy to bypass IP ACLs from my "off-campus" location. It was clearly located in North Korea. Then, the browsing was so slow (like Tor but even worse), I figured: Why not mirror it locally instead? And, what kind of documentation "toilet-paper" police are going to sit there watching interactively or SNORT monitor what happens to "precious" corporate documentation they forced me to use?