Hey Acodispo, interesting questions, for sure.
Your first question (why HD for monsters vs levels for players?) is really a two-part question in itself.
Why is DD the way it is?
That one is easy to answer: DD is how it is largely because it emulates the way the original game is.
So more importantly then:
Why is the original game the way it is?
This one is harder to answer, and I'm sure there are others more qualified to answer.
FWIW, my own perspective is that it is unlikely there is "one true answer" to this. That aside, perhaps it's worth taking a look back at what has been printed and speculating?
So, to speculate--cos it's fun
-- I'd say it all starts with Chainmail.
CM includes normal combat (versus normal types) in which a figure's fighting capability is measured in terms of "Men". E.g., a man fights as one Man while a Hero fights as four Men, implying these throw one die and four dice in normal combat, respectively.
CM also includes fantastic combat (versus heroic/fantastic types) in which a figure's fighting capability is determined by its type. I.e., Dragons, Trolls, Giants, Heroes, etc. each have their own unique combat stats on the Fantasy Combat Table. A single throw (of 2d6) in fantastic combat can result in whatever damage is necessary to slay an opponent, be it a Wight, a Lycanthrope, a Dragon, or whatever. The implication is that CM's fantastic combat is more abstract than is normal combat; a single throw represents enough combat action to potentially slay a fantastic opponent.
In the leap from CM to D&D, Dave Arneson included something similar in his pre-D&D fantasy game insofar as Hit Dice were the number of dice you rolled for damage when you scored a hit. I.e, flunkies would deal 1 damage die on a hit, men-at-arms would deal 2 damage dice on a hit, heroes would deal 4 dice damage on a hit, etc. (or something pretty close).
When Gary Gygax and Rob Kuntz translated Dave's dozen (or so?) pages of rules/notes into Dungeons & Dragons this notion got turned around, so that hit dice became the number of dice you rolled to determine what damage could be sustained, rather than what damage could be caused. Additionally, one of the fundamental
pieces of design genius in Dave's game (and also in D&D) was that players could advance
. With this in mind, it's a relatively small step to think player attack progression should be tied to experience levels. Monsters, meanwhile, don't have experience levels. But their CM fighting-capability stat (the number of Men they fought as in CM normal combat) was translated--generally speaking--into numbers of HD they have in D&D.
Moreover, recall that CM had two distinct combat tables; one for Men attacking
(normal combat), and another for Monsters attacking
(fantastic combat). This model appears to have been carried forward into OD&D as the Alternative
Attack Matrices I & II. That is, you could use the original (CM) matrices with six-sided dice, OR
you could use the alternative attack matrices appearing in Dalluhn and in M&M--if
you happened to possess a funky-shaped 20-sided die. Remember that back in 1973-74 funky dice were not necessarily easy to come by. Perhaps this is why the d20 tables were called an Alternative
to the d6 tables?
That's all whatever it is so: getting back to that notion of fighting capability
for a moment, it's interesting to observe that D&D still provides fighting-capability stats for the players types in Dalluhn (1973), in M&M (1974), and in GH (1975). If you compare the player level bands on Attack Matrix I (Men Attacking) to these FC stats, you'll see that there's a pretty close match-up between Man FC, Hero FC, Superhero FC, and the first, second, third tiers on the attack matrix, respectively. Details on this have been posted elsewhere (see, for example here
I'm not suggesting that the original designers gave the relationship between levels-and-HD-and-FC any conscious thought. More that they just knew fighters, clerics, M-Us were meant to be "about so good"
and that this intent was subconsciously
reflected, independently, in the two combat systems. I'm not overly surprised that the two systems line up as neatly as they do.
Over to your second question: (why are monsters better at fighting than fighters?)
Once again, I suspect there is no one true "revelation" in this.
I've heard it said that the monsters need a edge to keep up with the fighters who ultimately get +3 swords, +3 shields, rings of protection, and whatnot. Which is all nice and logical, and just about works out in play too
However, the earlier version of Attack Matrix II appearing in Dalluhn (1973) has almost exactly a 5% improvement in attack capability per monster HD but only goes up to a maximum (from memory) monster size of 9 HD. In M&M the HD range is expanded to accommodate monsters of up to 16+ HD without significantly altering
the hit probabilities within the matrix. This means that M&M actually toned-down the high level monsters, and implies (to me, at least) that the numbers in the matrix are more a convenient numerical progression than they are tailored to the specific monsters of a HD range.
It's also possible that play-testing might have resulted in adjusting monster hit probabilities downward... and/or taken into consideration players having magic items... all possible
Hope all that at least gives you some ideas to kick about..?