I’m assuming it stays in the RAM till the time the computer shuts downs

Correct.

We know that one could, in theory, get a dump of the contents of the RAM in such a state, if done correctly.

An example of such an attack would a "cold boot attack".

Is there some way to insert the USB, decrypt the drive, and then remove the USB and all traces of the key from the system?

It sort of depends on how the underlying hardware is designed. You can create a system in which the RAM's contents are encrypted by the hardware, but at some point the data must be decrypted for use. For example, one could theoretically sniff the data-lines between the RAM, and the CPU. This is all of course ignoring the fact that the hardware, itself, could be compromised i.e. Intel M.E., backdoors/vulnerabilities in the BIOS, etc. There's lots that can be done to try to mitigate security vulnerabilites, but there is always a tradeoff between security, and convenience.

Maybe the best form of security is memorizing a private key, then manually doing the math with a pen and paper to decrypt some text, and transmit it with a carrier pigeon.