Mildew is fungus that developes in moist conditions and the stains cannot be removed. Always make sure sweaters are completely dry before storing, especially if being sealed when stored.
Preventing insect damage:
Not all moths are a threat to your yarn and sweaters. The moth that infests your pantry is not the same moth that lays it's eggs in your yarn for the larvae to eat when they hatch. The culprit to look out for is the clothes moth. The moths look for dark, undisturbed locations to lay their eggs in, so you might be able to avoid them if you can store your sweaters in clear bags stored in light areas. You can also attempt to mask the odor of the yarn with herbal or cedar sachets, making it harder for the moths to detect your sweaters.
Some web sites which offer useful information about insects and care of textiles are: