The City of Toronto’s Solid Waste Management has an answer for this question. They recommend:
Flies, wasps, hornets and bees can be discouraged from invading your compost bin by covering any exposed food, with a 2.5-cm (1-in) layer of soil or by burying fresh food into the pile. Add air to your compost pile by turning it or create air channels by plunging a broomstick handle into the pile. Keep the pile slightly damp so that it will heat up. Higher temperatures and moisture will kill any fly larvae and discourage bees, wasps and hornets from nesting. If a nest has already been set up in your bin, soak the pile completely with a hose and spray nozzle and leave it damp until the colony vacates the pile. The pile can also be dismantled after freeze-up in the fall.
