Exile is a mod that provides a lot more fun if you have a team, doesn't need to be more than people total, but it helps a lot. The game can be rather punishing at times, especially if you are new. I would also recommend that you find a group to play with if you are planning to make progress in Exile. Yes it's PvP but my experience on SG is people are pretty honest and not looking to hit the little guy. It's not productive to drive lone wolfs off.
You tend to find most groups will leave the little bases alone.
As far as I know we don't have trolls that run around and destroy others bases if they haven't targeted you on any way. So it all depends on where and how you've built. It also depends on how you've built your base - if players can get easy to the bottom of your flag, they most likely try to steal it. If you have built your base somewhere in the forest, there's a less probability that you get raided.