Claire, Dr. Brizman has told me repeatedly that the problem with dairy, is the pasturization. SO, when we are ready for RAW DAIRY, when it is the right time, it will probably be very beneficial for us. So if you can, look into getting access to raw dairy.
I agree that craving is a sign. I had the same issue. Commercial dairy has a opiate effect. You can read about this on line. 
I am not sure that yeast feeds on dairy. But per Dr. Jeffrey McCombs who specializes in candida yeast infections, the problem with commercial dairy, is that it is mucous forming, and fungal candida (and other bad microbes, really) have created biofilms in our bodies. This is what in part is meant by "resistant" bacteria. The bacteria "buddy up" and surround themselves with mucous to protect the little food eating symbiotic biofilm. You can read more about this if you find it interesting. So Dr. McCombs takes his patients off it for about 4 months, to help the antifungals that he uses (like SF 722) BUST up the biofilms. 
Personally because I am on SF, and OO, and have suffered from CHRONIC bloating for 2 years, I cut WAAAAY back on dairy. But I just could not cut it out. I have a splash of cream in my coffee, and a BIT of butter on a piece of toast or maybe on top the occasional veggie or rice cake. But that is it for me. NO yogurt, no milk, no cheese. I really missed it at first. Dairy supposedly has an interplay with oipiod receptors, so I had some withdrawl I think when I stopped yogurt bc I was eating it EVERY DAY. After about 2 weeks I was fine, and even tho I love it, I just don't miss it right now. 
So with the OO/SF protocol, cutting down on dairy and carbs, my bloating is reduced by 75% I am BEYOND thrilled. I really can't say it is dairy, but Dr. B says my body will like raw dairy so much better. I am planning on trying raw butter and eventually raw cream when my bladder has been good for a while. BTW heating raw dairy takes the good stuff out, so best to add it after cooking if and when you do try it.