The term you are looking for in general is "reverse engineering". For software in particular you are looking at disassembly, decompilation and various forms of tracing and debugging.
As for particular software: For .NET there is ILSpy that can help you look into how things work. For native code I have used Ghidra in the past.
Native code is a lot more effort to understand. In both cases things like variable names names will be gone. Most function names will be missing (even more so for native code). Type names too. For native code the types themselves will be gone, so you will have to look at what is going on and guess if something is a struct or an array. How big is the struct and what are the fields?
Left over debug or logging lines are very valuable in figuring out what something is. Often times you have to go over a piece of disassembly or decompiled code several times as your understanding of it gradually builds.
C++ code with lots of object orientation tends to be easier to figure out the big picture of than C code, as the classes and inheritance provides a more obvious pattern.
Then there is dynamic tracing (running under some sort of debugger or call tracer to see what the software does). I have not had as much success with this.
Note that I'm absolutely an amateur at reverse engineering. I thought it was interesting enough that I wanted to learn it (and I had a small project where it was useful). But I'm mostly a programmer.
I have done a lot of low level programming (C, C++, even a small amount of assembly, in recent times a lot of Rust), and this knowledge helps when reverse engineering. You need to understand how compilers and linkers lowers code to machine code in order to have a fighting chance at reversing that.
Also note that there may be legal complications when doing reverse engineering, especially with regards to how you make use of the things you learned. I'm not a lawyer, this is not legal advice, etc. But check out the legal guidelines of Asahi Linux (who are working on reverse engineering M1 macs to run Linux on them): https://asahilinux.org/copyright/ (scroll down to "reverse engineering policy").
Now this covers (at a high level) how to figure things out. How you then patch closed source software I have no idea. Haven't looked into that, as my interest was in figuring out how hardware and drivers worked to make open source software talk to said hardware.
Due to the recent xz trouble I presume? Good idea, I was thinking about this on an ecosystem wise scale (e.g. all of crates.io or all of a Linux distro) which is a much harder problem to solve.
Not sure if the tag logic is needed though. I thought cargo embedded the commit ID in the published package?
Also I'm amazed that the name cargo-goggles was available.