Binary speed is really the least reason to do it. Whether it's worth it or not is up to the individual, but there are a lot of little reasons Gentoo is uniquely powerful.

Benefits specific to compiling:

• fine-grained control of features and dependencies with USE flags
• very easy package maintenance (writing ebuilds)
  • much simpler to add your own custom local packages when you need them
  • less workload on the gentoo team which is good for repository health and breadth
• control of compile flags (yes speed, but more practically hardening for secure systems)
• the same gentoo is available on way more platforms and architectures than any binary distro