The consensus is that anything less than 50 calories won't break a fast, but some disagree with that.

Since I lack "portion control" I figured I'd drop the milk and just go black. Unless I'm explicitly breaking my fast (or in my eating period).