I don't know what is the problem with cl-lib been loaded in Emacs core. RAM is cheap nowadays. I am loading the entire cl-lib in my Emacs when I build Emacs, no problems noted (been doing this for a while):

In my loadup.el I do this:

;; This file doesn't exist when building a development version of Emacs
;; from the repository.  It is generated just after temacs is built.
(load "leim/leim-list.el" t)

(load "emacs-lisp/gv")
(load "emacs-lisp/pcase")
(load "emacs-lisp/easy-mmode")
(load "emacs-lisp/cl-lib")
(load "emacs-lisp/cl-seq")
(load "emacs-lisp/cl-macs")
(load "help-mode")
(load "emacs-lisp/cl-extra")

In conjunction with this, I have also patched cl-lib.el to remove some unnecessary loading when bootstrapping:

(provide 'cl-lib)

;; (unless (load "cl-loaddefs" 'noerror 'quiet)
;;   ;; When bootstrapping, cl-loaddefs hasn't been built yet!
;;   (require 'cl-macs)
;;   (require 'cl-seq)
;;   ;; FIXME: Arguably we should also load `cl-extra', except that this
;;   ;; currently causes more bootstrap troubles, and `cl-extra' is
;;   ;; rarely used, so instead we explicitly (require 'cl-extra) at
;;   ;; those rare places where we do need it.
;;   )

The reason is I just prefer to have it in loadup.el explicitly so I can see it and comment/uncomment if need be, instead of perhaps forgetting cl-lib.el does it on the bootstrap.

You can make those as patches and apply patches automatically when building Emacs.