It's actually very straightforward. Once I went to the trouble of looking it up, the difference was obvious.
Herbs are the leaves of the plant, while spices come from the roots, bark, and seeds. Essentially, any part of the plant that is not a leaf and can be used for seasoning would fall into the spice category.
Some plants have both: like cilantro (the leaves) and coriander (the seeds of the cilantro plant). Some only give us one or the other. As far as I know, no one uses rosemary roots or nutmeg leaves in their cooking, though there's a first time for everything!
The herb-or-spice distinction doesn't necessarily change the way we cook with any of these seasonings, but it's a handy thing to know next time it comes up while playing Trivial Pursuit.