The pens do take some getting used to in order to manage the ink flow. I recommend practicing on less fancy jars first before you commit to your preferred jar. I'll probably redo the molasses sugar, pictured above, for example. It was my first jar and you can see that I hadn't quite figured it out yet. The ink can easily be removed, but only by using solvent based removers (like nail polish remover) or by carefully scraping with a razor blade, both which require extra caution.
The pen I use is a Japanese white indelible paint pen from the Kinokuniya Stationary Store in San Francisco's Japantown but most well-stocked art supply stores should carry less obscure brands like Sharpie's paint pen. Just ask for a pen that writes on glass and you should be fine.
White Paint Pens on the Web:
$3.99 per pen plus shipping Sharpie
Related: Best Jars to Organize your Pantry
(Image: Dana Velden)