Sir, you are simply repeating the invalid assumptions of the early 21st century, which are false. The false assumptions are based on an incorrect belief that Einstein was correct on every point, which of course, he was not. He was unable to unify gravity, for example. That being said, how can you assume that gravity (if not unified and properly defined and understood) can be a valid scientific constant to apply to a major theory? (such as relativity, for example).
The only logical conclusion is that basing a major theory which will then be followed by numerous other theories, all based on the previous assumptions, is invalid if a) any false constants exist, and b) if false assumptions are 'assumed' to be true.
The truth of the matter shall require a great deal of 'time' to work out, but eventually shall be realized, and "time travel" as you call it, will be accomplished. Or "is" being accomplished, depending on the observer's viewpoint. For example, you claim that if time travel were possible, "we should see them". This is incorrect, because a) you are assuming time travelers are so careless as to be noticed, when observing the 'past'. , and b) you are making a false assumption with no scientific basis whatsoever. Since we are not in physical contact, and communicating only through visible text, I am permitted to share what we call secondary information acquisition. In this type of information gathering (such as reading this text) [ I shall not become too involved in the technical details], causality is not disturbed due to a specific function within the brain which interprets visual shape information (text). It's a complex process, but basically it means that anything I type here, that you read, since not actually 'observed' (with the five senses as a combined fractal-based thinking process (as graphed), can not, and will not, affect causality in any way. This has been proven experimentally, so therefore, I am permitted to share information in this way (text). Further, a lack of full observation does not imply that you can not take in new information and use said information to conceive new thought processes, some of which will be fractal, but, not fully coalesced. As a result, no disruption of causality can occur. From there, further study must be undertaken to understand "why" that is, and this site is no place to write detailed descriptions of the processes involved.
I hope this answers your questions concerning why you do not 'see' time travelers in your era. I hope that you continue your studies and interests in science. My greatest difficulties are not the science, but the historical language studies. I hope you are able to fully understand what I have written here. Thank you.
S.Z. Waxahachie