Although the pharsing of your question suggests you have already made up your mind I will venture to attempt to address your question in a way that does not suggest an answer one way or the other but does offer one possible explaination to your question:
real time travelers please stand up? PROVE YOU ARE A ACTUAL TIME TRAVELER!!!
One theory supported by the majority of leading cosmologists and other quantum field theorists who otherwise could be loosely classified as string theorists or quantum gravitists/cosmologists is the ""many-worlds" interpretation or metatheory of quantum mechanics.
Steven Hawking is well known as a many-worlds fan and says, in an article on quantum gravity [H], that measurement of the gravitational metric tells you which branch of the wavefunction you're in and references Everett.
The Theory
If one of the systems is an observer and the interaction is an observation then the effect of the observation is to split the observer into a number of copies, each copy observing just one of the possible results of a measurement and unaware of the other results and all its observer- copies. Interactions between systems and their environments, including communication between different observers in the same world, transmits the correlations that induce local splitting or decoherence into non- interfering branches of the universal wavefunction. Thus the entire world is split, quite rapidly, into a host of mutually unobservable but equally real worlds.
According to many-worlds all the possible outcomes of a quantum interaction are realised. The wavefunction, instead of collapsing at the moment of observation, carries on evolving in a deterministic fashion, embracing all possibilities embedded within it.
All outcomes exist simultaneously but do not interfere further with each other, each single prior world having split into mutually unobservable but equally real worlds.
Consider Schrodinger's cat. A cat is placed in a sealed box with a device that releases a lethal does of cyanide if a certain radioactive decay is detected. The odds are 50/50. For simplicity we'll imagine that the box, whilst closed, completely isolates the cat from its environment. After a while an investigator opens the box to see if the cat is alive or dead.
In the MWI, two worlds are created. One world where the cat is alive when the box is open, one world where the cat is dead. The investigator is also split into two worlds. One investigator sees the cat alive when the box is open and one sees the cat dead.
Which world does the investigator end up in?
Suppose the investigator, Fred, has an operation that splits his brain in two and each half is transplanted into two different cloned bodies. Let's further suppose that each half-brain regenerates to full functionality and call the resultant individuals Fred-Left and Fred-Right.
If Fred-Left and Fred-Right are never told of the experiment and were transported to opposite ends of the earth, they would have no idea that the other half of their brain is alive and conscious in another part of the world. The same is true of MWI.
Fred-Left can ask, why did I end up as Fred-Left? Similarly Fred-Right can ask, why did I end up as Fred-Right? The only answer possible is that there was no reason. From Fred's point of view it is a subjectively random choice which individual "Fred" ends up as. To the surgeon the whole process is deterministic. To both the Freds it seems random.
Same with many-worlds. There was no reason "why" you ended up in this world, rather than another - you end up in all the quantum worlds. It is a subjectively random choice, an artifact of your brain and consciousness being split, along with the rest of the world, that makes our experiences seem random. The universe is, in effect, performing umpteen split-brain operations on us all the time. The randomness apparent in nature is a consequence of the continual splitting into mutually unobservable worlds.
Note: Split brain experiments were performed on epileptic patients (severing the corpus callosum, one of the pathways connecting the cerebral hemispheres, moderated epileptic attacks). Complete hemispherical separation was discontinued when testing of the patients revealed the presence of two distinct consciousnesses in the same skull. So this analogy is only partly imaginary.
The thermodynamic Planck-Boltzmann relationship, S = k*log(W), counts the branches of the wavefunction at each splitting, at the lowest, maximally refined level of Gell-Mann's many-histories tree. The bottom or maximally divided level consists of microstates which can be counted by the formula W = exp (S/k), where S = entropy, k = Boltzmann's constant (approx 10^-23 Joules/Kelvin) and W = number of worlds or macrostates. The number of coarser grained worlds is lower, but still increasing with entropy by the same ratio, i.e. the number of worlds a single world splits into at the site of an irreversible event, entropy dS, is exp(dS/k).
Because k is very small a great many worlds split off at each macroscopic event.
So, to answer your question. The are an infinite number of worlds to time travel to. If someone wanted to time travel to this period they would want to either record history or get something that is needed in the future but for some reason is unavailable in that time and could be transported back to the time travelers original world. That in itself would limit the number of visitors from the future.
Second, and more importantly, there are an infinite number of worlds similar to the world we occupy so the chances of a time traveler landing on this exact world line is miniscule. In fact the opposite of your statement above could be inferred. The chances of a time traveler hitting this exact world line is very small. Therefore it would be rare indeed to encounter a time traveler.
Finally, there is also the possibility that there are individuals that time travel for fun. But again, unless there are large number of them time traveling for fun the chances of them hitting this exact world line remains very, very small.
