Herbs are the leaves of the plant, while spices come from the roots, bark, and seeds. Essentially, any part of the plant that is not a leaf and can be used for seasoning would fall into the spice category.
Some plants have both: like cilantro (the leaves) and coriander (the seeds of the cilantro plant). Some only give us one or the other. As far as I know, no one uses rosemary roots or nutmeg leaves in their cooking, though there's a first time for everything!
The herb-or-spice distinction doesn't necessarily change the way we cook with any of these seasonings, but it's a handy thing to know next time it comes up while playing Trivial Pursuit.