Child custody modification?

my ex husband has decided that my 6 year old should live with him and his girlfriend. my husband i provide well for my son and he is loved and well provided for. basically my ex has him on summer visitation and my son is saying he wants to live with him. this happens every summer, then once he gets back home he's all about me and my husband. the kid is 6 years old so i'm sure this is normal. my question is, if my ex decides to go ahead and take it to court is there any reason a judge would uproot my son from his current living situation to allow him to live with my ex? we are both good parents, income levels are about the same. ex and his gf both work full time, my husband works full time and i'm a stay at home mom. i'm just a little nervous right now and would like some answers.