My First Kernel Module: A Debugging Nightmare (opens in new tab)

Have you looked into using eBPF instead of writing a kernel module?

http://ebpf.io for some more insights.

At the very least, it'll provide some useful tooling for you to debug problems in kernel-space.

Seems like someone did try to get those functions exported, but the maintainer rejected it, saying that no driver should be poking so deep into fd internals. Makes sense. Your use case is kind of niche.

https://lore.kernel.org/lkml/20180730163256.GC27761@infradea...

By the way, C Playground is really helpful for teaching an OS course!

Here's a hack you could use to get around the functions not being exported: https://github.com/anbox/anbox-modules/blob/master/binder/de...

That’s exactly it. It’s the only way to master something. The more varied exposure over time we have to the core ideas of a discipline, the more we come to master the thought process of comprehending it’s limits and possibilities to the extent where we can make it do whatever we like.

How did you pivot from kernel programming to designing chips? Did you already have a background in embedded electronics?

>learning to think about concurrency, time, interrupts, race conditions

So what books can you recommend to understand the above subjects? I know of only UNIX Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers by Curt Schimmel.

>everything is concurrency, at the lowest levels .... after a while I came back to doing kernel stuff and found that with this new background all that hard stuff was trivial and obvious

I see a lot of HDL programmers say this. But how exactly do you map the concepts since the very language semantics between HDLs and "Standard" computer languages are different?

Concurrency is still hard for me, but I do find it getting much easier over the years :) thanks for the story!

I hadn't considered eBPF because I needed some pretty obscure information from the kernel internals (i.e. the addresses of the `struct file`s) and I didn't realize eBPF was as capable as it is. Another commenter suggested trying it, though, so I'm checking it out now!

I did use printk for debugging, but I (incorrectly) assumed it could block. Another commenter pointed out that this is not the case. TIL!

The gdb link looks very helpful and I'll try that next time. Thanks for linking that.

Where would you look for a list of what you can do with eBPF and how? (I think maybe I’m searching for a list of hook points?) I keep seeing tantalizing hints about all of the things it lets you do, but the tutorials I’ve seen only seem to cover networking and tracing.

(The project I have in mind at the moment is making a bindfs-like filesystem without FUSE, but I’ve had a few different ideas where eBPF seemed like it might have been a good fit if I could figure it out.)

This is really good to know. I had assumed it could block when allocating memory for the formatted string buffer, but the rationale explained in that article makes a lot of sense. Being able to use printk simplifes things a lot.

now that is technical leadership.

Same. I would love a job comprising solely of jumping into big hairy systems and debugging weird issues. Its much more interesting to understand how exactly things work at every level of the stack (the bottom of the stack being OS/kernel or even hardware stuff, not a backend endpoint or database) than writing code.

> I wonder if I am the only one that loves debugging difficult/weird problems.

Same here. At times, I'd prefer to just work on debugging things for colleagues versus writing rather boring code. It can give some insights when it comes to design, as well as enabling customer support to fix certain issues.

I enjoy it, but hate that it's almost always for something that needed to be figured out yesterday.

"Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community."

https://news.ycombinator.com/newsguidelines.html

I actually cannot get enough information from doing that. Crucially, I need to be able to recognize whether two file descriptors point to the same open `file_struct`. (To be clear, this isn't the same as whether they're pointing to the same file path. I need to know when the two file descriptors are sharing the same cursor.) There is no way to do this using existing APIs, because there is nothing identifying a `struct file` besides the memory address of the struct. (The "open file IDs" I mention are hashes of the `file_struct` address.)

I did spend a lot of time trying to avoid writing a kernel module, and this was the only way I could find to do it :)

Ah! This would have been really helpful!

Elixir was extremely helpful to me! It didn't always help me understand _why_ code was written the way it was (hence my incorrect use of rcu_read_lock), but it was very helpful to see some examples.

I've not done too much kernel programming, but for sure I know that looking for existing uses of code is very helpful.

It looks like the author of the piece did something similar, and noted other people doing similar things to themselves.

I wrote some modules to experiment with the Security Module API, because working with the APIs seemed like a good way to learn how they worked, and what was possible beyond just SELinux,Apparmor, etc.:

https://github.com/skx/linux-security-modules

Thank you! It's open source, and I'd love to hear if you have any suggestions for it. Would also love to see what you're building!

That's a good point. I'm hoping that this never gets hit, and if that line ever appears in the logs, then things are already broken. However, it's probably better to improve the failure mode where possible :)

[edit] and yes, since we break and don't follow the `next` pointer in the linked list, that also shouldn't cause any problems.

[edit 2] a sibling comment by cesarb pointed out that printk actually does not block, since it's important for it to be usable in critical sections to debug when the kernel gets into trouble