I used to have a similar question about this paradox. What if I solved time travel, went back, and somehow convinced myself not to pursue time travel? In that instance, I might not create a time machine because I talked myself out of it. But I believe that the world is more than physical. There is also a Spiritual Realm that is the next plane. These are perhaps the visions of what a person remembers seeing in afterlife experiences that they believe to be a classic vision of heaven.
But, on this plane, time is irrelevant. There is but one time: the Here and Now. Everything else is speculative. And I believe that the coexistence of these two planes is for the purpose of third-dimensional experiences, which are impossible in the fourth dimension. But, for this experience to be practical and efficient, it would have to extract every possibility in every instance. Otherwise, our subconscious selves would be rolling dice, which I believe would be inefficient and not worthy of the effort.
So, say the man goes to his 10th birthday party and meets his younger self. Now the 10-year-old boy has a memory of meeting his older self. But, unless you yourself had this same memory, you would not. There would be two alternatives. As it should be. Something to note in this instance is that you may be surprised to find that, in this reality, your 10th birthday party might be different than you remember it being. Or it could be exactly the same, the only difference being that an older you is present.
Something that I try to observe is that there is nothing a person can do or not do that I would be incapable of, given the same situation they are in. Therefore, when another person is in joy or sadness, is dancing on stage, or is a seashore hermit, I feel that experience through a higher conscience that is not only shared but, by the most accurate definition, is one and the same.
But this is not time travel; it's dimension travel, which would also be a fascinating experience. Getting back might be a problem. But, by the time you journeyed to a handful of other worlds, your entire thinking about your own world might change. And you might not want to find your way back because of the infinite possibilities. Who knows? The experience might change you more than you change it.
You might have been destined to have met yourself at the party, which would mean that we could throw free will out the window. But I believe that, in a functional universe, free will is really an illusion determined by your relative position. But then everyone would have to accept their existence as something more than it is. Are we ready?