Well, first of all, know that, as you pointed out in your question, we’ve been broadcasting radio signals for about a hundred years or so, which creates a radio bubble around SOL, our sun, with a radius of about 100 light years. So first lets look at how many stars are within that bubble.
According to A stars within 100 light-years
there are about 76 stars of type “A” within that distance, which is not very many compared to the estimated 100 to 400 billion stars thought to be in our own Milky Way Galaxy.
Of those stars, there just aren’t that many candidates for habitable planets in orbit around them.
At present speed and existing technology we could reach the nearest star system proxima centauri in about 10,000 years, which is 4 light years away. We can assume the aliens would have similar problems. If we could go as fast as 4.5% the speed of light, about 140 times faster than any spacecraft we have yet to create, then we could reach our nearest neighbor in about 100 years.
I really don’t think you need to worry. We don’t even know if microscopic alien life exists anywhere other than Earth, and we don’t have any evidence that anything more advanced is anywhere near us in the Milky Way Galaxy. Anyone who wanted to travel hundreds of years to try to invade us would be nuts.