My son is allergic to the flu vaccine I need some advice?

my son has suffered from food allergies his whole life he is 4 the allergy specialist even did a skin test to see if he is allergic to the flu vaccine and it came out very bad.On top of that my son has a weak immune system he got a stomach virus a few months back and spent 3 days in the hospital.The allergist told me that everyone in the house should get the vaccine to try and stop him from catching the flu.So that is what we are doing.But I am also not comfortable taking him to places like the park or chuck e cheese.We have a couple of birthday parties coming up this weekend on my husbands side of the family and none of them have had the flu shot so im just not comfortable taking my son to their houses around like 30 different people until the flu season is over my husband does not agree he is more worried about hurting his family's feelings.I know I can't protect my son 100% but I feel I should do what I can to keep him healthy.am I being paranoid?or as parents would you do the same thing/