Then I implemented a little FUSE driver in Python to read the disk image from the host system and it was wonderful to mount it the first time and see the files! https://github.com/vinc/moros-fuse
PDF link: http://www.nobius.org/practical-file-system-design.pdf
Warning: Terribly written. many hacks.
I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how?
Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments.
Even better, but unrelated question... how the heck does a x86 OS request data from the HDD?
As for how you actually request data from the hard drive: There's older ATA interfaces, and BIOS routines from them, which I suspect is what most hobbyist OSes would use.
A more modern interface is AHCI. The OSDev wiki has an overview, where you can see how the registers work: https://wiki.osdev.org/AHCI
Entirely too short summary: Use PCI to discover the various devices attached to the CPU. One or more of them are AHCI or NVMe devices. The AHCI and NVMe standards each describe sets of memory-mapped configuration registers and DMA engines. Eventually, you get to a point where you can describe linked lists of transactions to be executed that are semantically similar to preadv, pwritev, and so on.
There's tons of info on osdev.org, such as https://wiki.osdev.org/AHCI
There was a lot of discussion in the past around TFS https://github.com/redox-os/tfs, my understanding is that effort has kinda lost steam.
Actually everything around "Redox" looks like:
One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k.
A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the early 90s. With 512 bytes blocks, you get an addressing capacity of 2TB, and with 4k blocks (the AF format), you get 16TB. In fact I remember 'round the late 90s / early aught we'd reformat our drives using higher-size blocks because the base couldn't see the entire thing.
Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?
HDDs also have small "gaps" that have headers to locate the sectors (in the distant past you could do a low-level format to correct these gaps as well).
SSDs do not have these gaps but they do need the ECC.
People would even run defragmenters, that would rearrange file blocks so they were stored continously and the whole file could be read without seeeking.
So the inode says "The data for my file starts at block 72 and is 3 blocks long" (or something like that). The disk then goes there, and reads blocks 72,73,74.
Each block is 4KiB large often, so if you have a 10KiB file, you still take up ceiling(file size/block size) blocks.
That's why there is a difference between "File size" and "Size on disk" when you look at disk usage summaries.
I once made a 'file system' to mount cpio archives (read-only) in an embedded system. Cpio is an extremely simple format to generate and edit (in code) and mounting it directly was very effective.
https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf
There are definitely some significant benefits you can get from managing your own storage, rather than using a filesystem.
It isn't just about performance gains, which are substantial, it also greatly simplifies the design and code by eliminating edge cases, undesirable behaviors, and variability in behavior across different deployment environments.
It's a nice ambitious goal which can really drive language and library design.
How soon until someone builds an Operating System developed in Rust? Maybe make it microkernel-based this time.
Redox[1] has been around for almost as long as Rust has. I first heard about it 4-5 years ago.
They had an interesting competition a while back challenging people to figure out how to crash it.
What’s the progress and potential of Redox?