Writing a brand-new OS is almost impossible by now

A few months ago, I was wondering how an OS works, and instead of trying a new one from scratch, I decided to jump in on the existing one, which is Redox OS, to be exact. Redox OS is an OS that embraces microkernels and Rust as its base language, which are two things that are majorly different from Linux and interesting in their own right.

The kernel is around ~30k LoC, most bootstrapping is handled by the bootloader and drivers. The kernel itself boots in under 1 second most of the time. It’s amazing what the team has accomplished: Last year it was dynamic linking and this month they just finished Unix socket implementation.

Redox OS is 10 years old, so it should be mature now?

I think the OS kernel itself is quite mature, but the lack of supported programs hinders me a lot. As of this writing, the only compilers running inside Redox are GCC and Python (via RustPython), and most programs right now are cross-compiled. This saddens me a bit because most of the time I don’t write programs in C, but in more modern programs such as Go, Node.js, PHP, Python, and of course, Rust.

Why would these compilers have not been ported yet? That’s why I trying to jump in

And after a few attempts the problem is harder than I think:

How Porting Software Works

Porting is an attempt to make software work under a new OS. To do that, compilers use a special method known as cross-compiling. Cross-compiling in Redox OS works by using their own fork of GCC and Rust compiler so both can compile software into binaries that work within Redox kernel.

Why would they have to fork them? Why don’t we just pretend it compiles into Linux because both of them are using ELF binaries?

Theoretically, it’s possible to keep using the upstream GCC or Rust and not change anything since both emit Assembly binaries, which is a machine-specific language that makes no difference with what the OS is running with. But they have to, to introduce compiler gates such as __redox__ in C and #[cfg(target_os = "redox")] in Rust. If you look at their GCC fork changes, the changes are not that much.

The reason for these compiler gates is the difference between Linux and Redox syscall. A syscall is a special instruction available in Assembly to send messages to the kernel and may return the result, known as interrupts. The kernel prepares interrupts at bootstrapping before it launches any programs.

Syscall is so important without it the software will not even print anything to terminal. To prevent the software code from having syscall everywhere, the C language has a standard library called libc. Writing libc is the job for any OS to implement because it defines the contract between C programs and the kernel. To prevent the conflict in libc itself, the C language has two standards: the C standard library and the C POSIX library.

The C standard library defines basic things such as how to print to stdout and returning an exit code, while the POSIX standard is a superset of it which handles things such as threading, networking, etc. While C standard is a must, POSIX is not. POSIX standard exists in libc for Linux and MacOS but not Windows, which is why porting a C software to Windows is a pain for most people.

In Redox OS, libc is implemented in relibc, which is based in Rust with the help of cbindgen for C headers. It also contains additional services such as ld.so, which is the same concept in Linux for a loader for dynamically linked binaries, and some C runtime libraries.

Fortunately, Redox is almost POSIX-compliant. It’s almost because it won’t cover all of them, because either the standard itself is deprecated or it’s not available. Even though Redox can be 100% POSIX compliant, they still have to patch software with __redox__ because people are also using __linux__ for things that are not in POSIX such as parsing /proc, advanced networking features, messing with cgroups and many more.

For this reason alone, there are about 70 patches in Redox OS Cookbook as of this writing. Most of them are patches for C programs and libraries because it’s easier to patch them for themself rather than sending them upstream.

For many software that are not C, patching software is wildly different.

How Redox OS ports Rust Software

When you look at the Redox OS Cookbook patches, patches for Rust programs are almost non-existent. This is because either they have been forked or merged upstream. The exact reason for this is that Rust programs rely on libraries for OS-specific things. So creating a patch file for Rust programs is nonsense, because the library sources are downloaded from Cargo.

Cargo has [patch.crates-io] to patch dependencies, but it’s hard to create a patch because the patch must be a Git URL or a local path, and the version must be exact. So Redox resorted to fork important libraries that are important to the programs that required to make the OS usable.

For example, the winit library is important in Rust to handle low-level windowing in a GUI. The Redox OS GUI Compositor is Orbital, and Orbital uses forked winit as of this writing. Initially, they have submitted patches to winit to support Redox by adding calls to redox-syscall when #[cfg(target_os = "redox")], but as of this writing the forked winit uses libredox, and it hasn’t been merged upstream (both libraries exist for different historical reasons).

What’s libredox? It’s a complement for libc in Rust, which is a FFI binding into C standard and POSIX standard in Rust. When a program calls Rust’s libc, the call will be dynamically linked into relibc when it compiles, so Redox can change libc intricate logic without pushing changes to libc when needed. The libc functions may not cover all the things Redox can offer, so libredox exists to complement it.

Even though Rust upstream can compile most programs into Redox, it still need the forked toolchain to include the forked GCC compiler, which is needed when a Rust library need to link into external libraries such as the openssl crate.

Personally, right now I have a problem with porting tree-sitter. It’s a Rust program/library, but fails to compile with fs4 crate because it doesn’t support Redox at this time. When I look at their source code, I have to submit patches to rustix to tell them, “Hey, Redox supports statvfs now!” before submitting patches to fs4.

That’s the journey for actually porting one software. If you read the monthly news, there are a few software ports month by month. It’s not without reason that porting software is a hard challenge when it comes to a brand-new OS.

The Pursuit of a Complete Software Compiler

Porting software is not just a compiler problem, but also involves making sure that the software runs properly. There have been a lot of attempts to port important software to Redox, like sqlite3, which sometimes reports disk I/O errors, or qemu, which was reported to not run properly.

Personally, I have spent the time testing the Rust compiler in Redox. At the time of this writing, running rustc emits a Page Fault. The page fault is suspected due to extern crate not being handled properly, but there can be other reasons.

There’s also a push to the Go compiler, which is compiling, but also emits a page fault, a breakpoint trap, or a panic in relibc. The panic in relibc is suspected to be Golang binaries emitting an ELF binary that Redox currently can’t handle with, while the breakpoint trap can be something else that I don’t understand.

These two things are a problem that can happen when a compiler is self-hosting. The iterative loops of development in a self-hosting compiler result in the compiler code using the most advanced features the compiler itself can do, also known as dogfooding. But let’s think of it this way: If the compiler itself is working in Redox in the future, it’s most likely that the compiler will output the correct binaries all the time, which is an important thing to have.

I was thinking “okay, maybe try to avoid compilers that emit binaries”, something like Node.js or Deno. But Node.js uses V8, which, when I tried to port it, complained about abseil-cpp a lot. And guess what? Abseil C++ is the Google C++ Standard Library and it contains a lot of syscall and I have a feeling that I should not touch this unless I’m a Googler!

When I read the cookbook README I noticed “the porting process can take months” which now I fully believe and understand. These are just the compiler issues, let alone the libraries and software that depend on it.

Even though the Rust Compiler in Redox isn’t working yet, the compiler can cross-compile software into Redox, and it has been like that since day 1, so the Rust community accepts #[cfg(target_os = "redox")] as Redox progresses and gets more mature. Unfortunately, the same thing is not happening for other communities, like Go, Node.js, or Python communities, since the compiler itself is not supporting Redox yet.

Even if it is working now or soon, it can be complicated to convince the compiler maintainers to support another OS, and adoption of their libraries can take years to mature. It’s almost like if you want the OS to get a lot of adoption, you have to convince a lot of maintainers across the earth that it’s worth it, and maybe actually send patches to speed up the process.

Wrapping Up

For these reasons, I’m starting to think maybe “a brand-new OS is almost impossible by now” and I’m grateful that Redox started this pursuit since 10 years ago, when Rust was just released its 1.0 version, when I was in high school, barely knew low-level programming at all.

To me, it feels that pursuing a consumer-friendly, fully working OS from scratch is a lifetime goal. And if you want your own OS getting adopted, you have to convince your OS of its unique features, not just for your users, but also to developers who maintain the software and libraries you’re likely to have porting with. It’s gonna be like a chicken or egg problem.

I’m one of that person who reads wiki.osdev.org and amazed and almost going crazy with Assembly and QEMU. But I know it’s a waste of time with the advent of AI and other cool stuff, and low-level engineering is not one of them, even OSDev wiki has warned it, even their list of Abandoned Projects is kind of huge.

I’m glad that I bumped into Redox OS before, and I recommend this approach if you plan to learn low-level programming. There’s nothing wrong with trying from stratch as long as you intend it for learning, but if you want to start from existing one, there are a lot of other OS niches out there that may better suit you; maybe take a look at this and that.