One of the problems with how we imagine free will is that we see as a pure representation of our best selves. We see it as the free will that angels would have. But our sorry lot is not angels, but apes. And apes are not pure.
I don't mean this to be a moral judgment (although it might be). I mean that we are not blank slates even as newborns. And we certainly are not blank as adults.
The ultimate freedom would be omnipotence, the ability to do literally anything. And I think that if you gave that to every person on earth one at a time 90% of the first acquisitions would have to do with food, sleep, freedom from pain, or sex (my guess is in that order).
The reason that I mention this is that there is a quote, that as of printing I could not source, "freedom is not just freedom from." And this is my blank slate comment, given the change to do anything, anything at all, how many people would start with world peace and how many would start with a good meal and a nap?
We are apes, and not at our best when we can do whatever we want whenever we want. Our self restraint has real limits and our desires, well less so. Imagine what would happen if you had a magic power where you could create a limitless supply of your favorite snack when ever you wanted. Just BAMF! Double Stuf! Any time of day or night. You would be free not to eat them, not to summon them, but seriously who on earth could avoid gaining a couple of pounds?
If more power and more choice leads to a certain outcome in all people then freedom doesn't always lead to free choice. And if that's true then we need to internalize this limitation into our self understanding. Cut your neighbors and friends (and yourselves) some slack.