If you rsync ext4 files to exfat, you also have issues, but those are very clearly reported when you attempt to do so.
However hold on, i do some research. Now you hooked me ;)
EDIT: Ok got it, both Filesystem NEED xattrs enabled, and you need to run rsync with the -X or --xattrs option, that should fix the problems.
So even if both file systems support extended attributes, does not mean that you can actually preserve them.