Do We Systematically Misunderstand Our Own Language?

Suppose that God had created Adam and also, in an identical space-time, an exact copy of Adam. Such a God could swap Adam with that other person, then swap them back, and do that over and over, and it would make no difference, empirically: they have identical memories and everything. One difference it would make, though, is that any direct reference to Adam in this space-time would keep failing.

And similarly, what if fundamental particles (or other substances) are constantly being replaced by identical particles (substances)? It would make no difference empirically, which is why we cannot rule it out, logically. We can of course assume that it does not occur. Consequently we can assume that reference is direct; but, surely language would work normally even if such replacements were occurring.