That is part of it, certainly. The rest? A combination of you don't remember anything of this place when you leave it and the multiversal tree, as it stands, is rather well-protected from further unnatural splitting or intrusion.

[ Still. Still. Loki knows something of desiring context regardless of the immediate usefulness of such a project. ]

The man who came out on top of all this forced the timeline he was from... I think?... to be viewed as the only true timeline. Anything that branched off and became a problem to the overarching goals and stories therein was deleted. Destroyed. Immediately dispatched with, by the Time Variance Authority, and anything that remained was shipped off to be fed to Alioth. Because he was from the future, or at least a future point than either of us, he also knew of the Avengers' time traveling in order to deal with the threat of Thanos and his, ah, passion at devising all living beings by half their numbers. A grand plan. He was very good at grand plans, I'll give him that.

When he set up the TVA, he immediately removed all their memories of him and the Time War. They believed they were created by the TVA and that they lived only in service to it.

Honestly, the list of despicable choices made by one man given completely control over the flow of time and possibility is almost impressive.