Because one is a black box that very well may just be a much more advanced version of what current AI does. We don't know yet. It's possible that with the training of trillions of trillions of moments of experience a person has that an AI may be comparable.

I mean, the likelihood is basically zero, but it's impossible to prove the negative. At the end of the day, our brains are just messy turing machines with a lot of built in shortcuts. The only things that set us apart is how much more complicated they are, and how much more training data we provide it. Unless we can crack consciousness, it's very possible some day in the future we will build an incredibly rudimentary AGI without even realizing that it works the same way we do, scaled down. But without truly knowing how our own brain works fully, how can we begin to claim it does or doesn't work like something else?