crt0.s suggestion

[joep]

The various invocations of crt0.s should probably build proper stack frames rather than just branching to main so debuggers can know when they reach the end of a stack trace for a PD. Cosmetic, non-breaking, and unimportant, just a 'would be nice to have some day' for those of us with OCD.

[midnightveil]

What is involved with proper stack frames? My understanding was that all you needed was the link register setup (for AArch64), but that's just set to 0 at entrypoint. This seems to work fine in GDB for us. Although I do recall warnings about something related but I didn't see what could be done to solve this.

[joep]

I should have said this doesn't happen all the time only in the right circumstances. Currently, crt0.s for aarch64 looks like this:

.global _start;
.type _start, %function;
_start:
    b main

Sometimes the GDB stack trace heuristic asks the question "who called _start?" but because _start has no call frame here it will get resolved as '???' (as I said it doesn't break anything, it's purely cosmetic). But that's why generally you will see crt0.s coded as:

.global _start;
.type _start, %function;
_start:
    mov x29, #0
    mov x30, #0
    b   main

[midnightveil]

I'm confused, though. The link register and frame pointer already start at 0, so when GDB walks the stack it sees zero. Why would this change it?

[joep]

Ah, you are right and I hadn't thought about the fact the kernel is zeroing the registers when I wrote this. I further thought a NULL value return address is what causes gdb to stop the unwind but that's really (and I mean REALLY) dated information. Instead, hitting the symbol 'main' is what causes it to stop unwinding now but ONLY if that symbol is defined in the primary executable at startup. We load all the PD symbol tables via 'add-symbol-table' command so gdb just keeps trying to unwind for the PDs, stopping one frame up from main because it detects identical stack frames. The only way to fix it is to use the .cfi_startproc/.cfi_endproc/.cfi_undefined instructions to crt0.s and it's more trouble than it's worth.

[Indanz]

Isn't that easily fixed by doing bl instead of b? If there is a valid stack at this point there is no reason not to do bl.

(This is also what I thought you meant originally.)

[joep]

You guys actually do that for the x86 version, it does a call for main. The arm and risc versions both use branches. I would agree with you the bl is the better way but our problem stems from the way gdb works internally and is not a call frame issue. To fix the problem we are seeing requires a bit more, e.g., something like:

.global _start;
.type _start, %function;
.cfi_startproc
.cfi_undefined x30
_start:
    bl main
1:  b 1b
.cfi_endproc

[joep]

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

[Indanz]

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

[Indanz]

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

It is the user space which launches the new process that determines the values, and that code cooperates with the code in _start. There is some ABI that is being followed, not relying on a random small part of it seems silly.

[joep]

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

(gdb) pd break sensor::notified
Breakpoint 1 at 0x6200408: file sensor.c, line 33.
[pd break] sensor::notified at 0x6200408
(gdb) cont
Continuing.
[Switching to Thread 4]

Thread 4 "sensor" hit Breakpoint 1, notified (ch=0) at sensor.c:33
33	{

[sensor] Breakpoint 1 at notified (sensor.c:33)
(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The issue is that gdb does not understand it needs to stop unwinding at main so it actually continues and tries to show the previous frame to main and reports a potential problem. Again, this is completely cosmetic and acceptable as is and I never would have brought it up except with our vscode extension that uses cppdebug, it actually causes it a bigger problem that sometimes causes a crash (but mostly not). We will just fix the extension to be more tolerant.

[joep]

P.S. I would also argue a more correct way is not to rely on the kernel providing zero values and crt0.s should be loading explicitly the values that are 'correct' for the machine it is running on.

It is the user space which launches the new process that determines the values, and that code cooperates with the code in _start. There is some ABI that is being followed, not relying on a random small part of it seems silly.

Ah right, I forgot there was code in between.

[Indanz]

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

What does it show with bl?

[joep]

Have you tried it? Because I think it will fix the problem of main not showing in the backtrace. _start having '???' is perfectly fine if there is a main before it.

Main does show in the backtrace with branch as is, e.g.,

What does it show with bl?

Can I ask what you are thinking here? I'm obviously missing something. I haven't done it but I would anticipate it gets a real, correctly-named _start frame which, again, I would certainly agree is better and more accurate than the current b instruction alone. But with no .cfi_undefined, gdb will still try using x30/LR to look back another frame and the same problem occurs. So I guess my question would be, if you are going to make a change anyway (b -> bl) why not go the whole nine yards and also produce the DWARF CFI?

[Indanz]

I expect an output similar to:

(gdb) bt
#0  notified (ch=0) at sensor.c:33
#1  0x0000000006200734 in handler_loop () at src/main.c:194
#2  main () at src/main.c:220
#3 _start () at crt0.s...
#4  0x0000000000000000 in ?? () (maybe)

Notably, no corrupt stack warning.

The reason I expect the warning to be gone is because I think GCC assumed that main was called like a function, while it wasn't. So the warning was correct to say that the "stack" may be corrupted (if you consider LR as part of the stack). My expectation is that GCC is smart enough to know that _start isn't a normal function but the program entry point and to stop the backtrace there without warning.

However, I'm not sure about any of this, but was curious what GDB would show for you.

So I guess my question would be, if you are going to make a change anyway (b -> bl) why not go the whole nine yards and also produce the DWARF CFI?

Well, in this case having a _start that does nothing is silly, it could just have made main the entry point directly. I don't know why Microkit does it like this. As for the dwarf CFI, it would be good to know whether it's actually needed, don't you think? Also, I don't think your proposed solution is long-term robust, what you want is a way to tell the compiler to stop unwinding when it reaches the entry point. But that info is there already.

[midnightveil]

FWIW: the same thing happens for seL4 debugging as well. I've looked into this before and I have a hard time finding a definitive reference on how this is supposed to work.

GDB is supposed to have an option that doesn't unwind past main and that is supposed to be turned off but that doesn't seem to apply here. (of course it would be nice to be correct if it did want to)