I wonder if it would be possible to use this infrastructure to write a trace command that I've always wanted, which is one that records internal function calls in the binary.
When I'm feeling overwhelmed with a new codebase, one of the things I like to do to get a handle on the code is to just run some representative command (or make a connection, if it's a server) and see a log of every function that's called with its arguments.
We have strace to do that for system calls, and ltrace to do it for libraries, but nothing that just.. traces. It's shocking to me to think about how many person-years have been lost to debugging by such a command not existing in the C world!
I've seen some hacky and unstable scripts that use gdb to set tracepoints on "*" to get this done, but having a native binary that could do it would be amazing.
Long ago, in the time before React, I’d forked and fixed an older perl program that prettied up trace output from C programs. The binaries need(ed) to be instrumented using function call hooks feature in GCC (or most any other C compiler) but it’s pretty handy for doing what you described. Faster than GDB tracing usually.
I forget the exact details but you can see the setup on my GitHub if you want to try it:
yeah I think this would be very possible! I think the hardest part would be the "with its arguments" part. But there's a really useful Rust crate called gimli for parsing DWARF (I found it a lot easier than the C DWARF libraries I looked into).
That’d be pretty nifty! Having a program that could show the arguments would be very useful.
On a related note, do you know if it’d be possible to store non-exec data in a dwarf binary and parse/load it with the rust libraries? Seems it’d be easier than doing the same in C.
>The main problem I have with ltrace is that even though there’s a -p option (“Attach to the process with the process ID pid and begin tracing”), I don’t think I’ve ever been able to get that option to work. When I run sudo ltrace -p SOME_PID, nothing happens, even though I’m pretty sure the process I’m tracing is calling library functions.
>I don’t fully understand why ltrace can’t attach to processes, but that’s not what this post is about.
I use ltrace to attach to specific PIDs all the time, across a variety of linux distributions and versions. Unsure what is going wrong here when Julia attempts to do it.
I'm on Linux. I did a tiny bit more investigation and it seems like ltrace works on some processes (like top) but not on other processes (like Firefox). I'm sure that the Firefox process in question is making library calls because when I trace the same process with `ltrace-bcc`, it shows the library calls.
thank you so much! I didn't know about that and using those files looks extremely doable. I'd assumed that ltrace used DWARF or something like that to format arguments.
eBPF is a very cool technology. LWN did a great introduction
about it [1]. The tooling is getting mature with BCC.
I for one think about how eBPF bytecode and verifier could be used outside of Linux kernel. It's a thin and easily JIT-able architecture, designed to run untrusted code.
I feel like Julia starts many of her days by saying to herself “I wonder...” and then figures out how to answer that question and for the benefit of the rest of us, she blogs about it!
I wonder if it would be possible to use this infrastructure to write a trace command that I've always wanted, which is one that records internal function calls in the binary.
When I'm feeling overwhelmed with a new codebase, one of the things I like to do to get a handle on the code is to just run some representative command (or make a connection, if it's a server) and see a log of every function that's called with its arguments.
We have strace to do that for system calls, and ltrace to do it for libraries, but nothing that just.. traces. It's shocking to me to think about how many person-years have been lost to debugging by such a command not existing in the C world!
I've seen some hacky and unstable scripts that use gdb to set tracepoints on "*" to get this done, but having a native binary that could do it would be amazing.
I forget the exact details but you can see the setup on my GitHub if you want to try it:
https://github.com/elcritch/etrace
Edit: grammar
On a related note, do you know if it’d be possible to store non-exec data in a dwarf binary and parse/load it with the rust libraries? Seems it’d be easier than doing the same in C.
P.S. Great article!
>I don’t fully understand why ltrace can’t attach to processes, but that’s not what this post is about.
I use ltrace to attach to specific PIDs all the time, across a variety of linux distributions and versions. Unsure what is going wrong here when Julia attempts to do it.
I for one think about how eBPF bytecode and verifier could be used outside of Linux kernel. It's a thin and easily JIT-able architecture, designed to run untrusted code.
[1] https://lwn.net/Articles/740157/
This topic she’s clearly extra excited about!!!
:-)