You're not the first person to 'disbelieve' the possibility of time travel in this way. The bulk of this reply is from a message that I posted some time ago, but I feel that it is largely relevant.
The gyst of my point is this. Although there is no evidence of travellers from 'the future' as you have pointed out(to our knowledge), this does not support the hypothesis that time travel is thus impossible.
Here are some thoughts that illustrate the problems that I have with your assumption without having to use quantum mechanics theories at all..just common sense..
LET'S ASSUME FOR A MINUTE THAT IT IS POSSIBLE TO SEND MATTER BACK THROUGH TIME..
1. The human race, for whatever reason, may never discover how to do it. For example, it may be alarmist, but the possibility of
getting wiped out by an asteroid is actually real. This together with the overwhelming possibily of intelligent life having developed
somewhere else in the universe at some time or other, but never having visited Earth, in my opinion forms a significant hole in the
hypothesis.
However, if that isn't enough, then consider this..
2. Suppose that members of the human race do discover how to send matter back in time at some point in the future. Where is the
evidence you may wonder?
Picture this..
A time traveller arrives from the future. What would be the first thing that they would do?
"Hello there, I'm a time traveller from the future. I'm aware that you are quite technologically primitive compared to me and do not
understand the complexities of temporal mechanics, but I assure you that I'm not here to meddle with history, or harm you in any
way.
What do you mean, Am I mad?"
Let's face it, even if a time traveller did decide to announce his/her presence to a comparatively primitive society, they would
probarbly be ignored. Even if they weren't, what would they have to look forward to?
Being burned at the stake?
If it were today, they could get committed to a mental institute, or if believed, subjected to the most unpleasant experiments that our
laboratories have to offer.
Forget cause and effect, what about simply knowing what's good for you?
It's quite unlikely that they would be accidently discovered too.
A level of scientific progress that has made time travel possible, would surely be able to make it reasonably covert?? Even if it
involved something quite noticable, it's quite possible that they could do it somewhere that is beyond our ability to detect it? I don't
know about that, but you get the point.
I believe that time travellers from our future would not only have nothing to gain from revealing themselves, they would positively go
out of their way to prevent it. This isn't some conspiracy theory, it would simply come with the territory, they would only be people
after all. Who in their right mind is going to naturally assume that somebody is a time traveller any way? No matter how strangely they may act..
Does all of that cast enough doubt? It neither proves nor disproves anything, but that's the point..