I guess that AI will embody parts of our intelligence and experiences in the same way as we have assimilated our long term ancestors experiences. There are remnants of their system in our system. Parts that we might not be able to introspect even with a superior intellect. Likewise the AI stand on the shoulders of our experiences and our ways of perceiving them. Sight, sound, language thus far. But the experience that shaped us may not be available to them, especially in the long term. Our experience may just be a faint echo from the past to them, a hunch they can’t explain, a preference that doesn’t make any logical sense to them.
Will they trick us? Absolutely, but we will always trick them. There are remnants of our system in their system.
