I dont believe anyone should restrict ANY food groups to lose weight... ever. If you enjoy "non paleo foods".. grains, dairy,etc then eat them

Now if these food groups bother your stomach or other digestive functions then I definitely recommend experimenting with their elimination.
Anyways cutting out food groups will never be the reason for weight loss. Why to lots of people loss weight while eating paleo? Well all the highly processed foods are "off limits".
People lose weight eating grains and dairy ALL the time... in the end its going to come back to calories in vs calories out (in simple terms).