I have had no problems. Right now I live a block away from the county building, so I am a little cautious of having my children play when everyone is going to work (something about school age children playing in the yard at 8:30-9am might raise some eyebrows).
I think it also depends on where one lives, who they know (who is out to get them), how radical they are.
Some states are more lenient with their laws than others.
Not in all cases I realize, but I think that sometimes the child's well being also raises eyebrows.
Nowadays most people have at least heard of homeschooling. They might be not know the ins and outs of it, but they have atleast heard of it.
When I go out during "school hours" most people just assume that I homeschool.