You can grow vegetables in compost-alone, if it's finished compost with detectable mineral content (soil, enough to feel the grit; this soil might have come from bits of sod you've composted, or soil you've added to break up and activate thick leaf layers). Without mineral content the plants may lack for micronutrients and drainage may be slow.
Trouble with growing in compost alone, even the gritty well drained stuff, is that even finished compost keeps breaking down and settlin. So even if you fill and tamp the box at the beginning of the growing season the level may drop by late summer and the crowns of the plants may descend into poorly ventilated area prime for mildew development. We've frequently used a mixture of compost and screened (rock free) topsoil for flower and vegetable gardens, about half-anf-half. It settles, too, but not so quickly as compost alone. If you're growing an annually-replanted crop of flowers or vegetables, it'll work but for shrubs, trees and perennials (including perennial edibles like asparagus and rhubarb) you can expect to have to lift and replant down the road.