I agree with you and will go one step further. Your reason is an old one but still valid nonetheless. (One could postulate that it HAS happened and we haven't seen the evidence of it, but that's just a twist on "conspiracy theory" - "No evidence of the conspiracy proves it's working") or at least its antithesis.
For me, the problem is even more fundamental. We have, due to the fact that we measure "time" passing as a method of organizing our lives, fooled ourselves into the mistaken belief that the "past" or the "future" are actually tangibles that can somehow described as "whens" one can actually travel to.
We even use the relativity experiments of comparing atomic clocks that have been in motion to ones which have remained at rest and "proven" the theory that "time" slows down with velocity.
The truth is, we cannot really prove that the dilatiion is a result of two relative and different rates of time passage any more than we can prove that the clock discrepancy is not due to some effect on the atomic mechanism itself at some quantum level not yet understood. We accept it as "proof" since the measured effect is consistant with Einstein's predictions. But is this measurement really an observance of time dilation itself, or our ability to measure it? I think there is room for the "uncertainty principle" to be applied even at this level of physics. How can we be sure the conditions we are subjecting the measuring device (particularly the one in motion) to, are not themselves affecting the device itself that is doing the measuring. Perhaps THIS is the essence of Relativity in time, as opposed to any real flucuation in time itself.
I believe we make a serious presumption when we imagine time to be anything other than an artifical measurement for our convenience in describing "cause and effect" as opposed to a reality that it doesn't truly exist at all. We should contemplate the meaning of the fact that it is, always has been, and can only EVER be, Right Now. Beyond that, it is nothing more than a manifestation of concept, rather than anything real.
