(the following isn't fastidiously researched, just some anecdotal observations)
Right before the turn of the century, something interesting happened in the hip hop world. Hip Hop was well into the period where it was realizing its potential as a money-making force and certain entities within the culture were rising to superstar status (i'm thinking of the period well after Hammer and Vanilla Ice, even after the deaths of 2pac and Biggie, late nineties or thereabouts... the "shiny suit/hype williams/no limit hypercapitalism age).
At the same time (or a year or so later), you had the rise of what we today call "the underground," coalescing around Rawkus, Def Jux, Rhymesayers, Anticon, Quannum, Skribble Jam, a whole bunch of labels and artists and events. These were often independents, either consciously rebelling against the Puffy juggernaut or lumped in unwittingly. Though independent labels were (and are) making hip hop of all styles, the group of artists who came to represent "underground hip hop" were largely revivalists of some stripe, heavily influenced by early nineties east coast styles, the Native Tongues movement and Public Enemy.
As time passed, the split between "mainstream" and "underground," a split which wasn't always there, by the way, became more pronounced. While there were real aesthetic differences, it's more interesting to look at demographics. I think sometimes we focus way too much on the artists, who all have different motivations and opinions on this whole phenomenon, instead of the fans and the media, who really, i think, drove the split. Some artists were explicit about being indie/underground (BlackStar, Company Flow, Anticon), but many were just caught up in the shifting tides.
Many hip hop fans, especially white hip hop fans (the racial dynamics here just can't be ignored) latched onto this new movement, making stars out of artists like Jurassic Five, Mos Def, Talib Kweli, Atmosphere, Sage Francis, Aesop Rock, Company Flow, the Roots, the Quannum Camp, Common and others, artists they felt were representing "THE REAL."
Some of these fans-- not ALL of them, as some may have you believe-- became the infamous boogeymen called "BACKPACKERS." Though that term has been around for a while and has meant different things, today it most often negatively or condescendingly refers to a person with these characteristics:
1. straight white middle class male dressed in a black hoodie.
2. believes that all "mainstream" hip hop is horrible and should be avoided.
3. thinks that you're wrong about hip hop, whatever it is you believe.
4. says things like "DUDE YOU PROBABLY LIKE GAY-Z. YOU SHOULD LISTEN TO IMMORTAL TECHNIQUE BECAUSE HE TALKS ABOUT THE REAL SHIT, DUDE. DON'T BE A SHEEP, DUDE. ATMOSPHERE'S FUCKIN' DOPE TOO, DUDE-- IT'S REAL POETRY, NOT THAT BLING BLING SHIT. DUDE."
5. is a revisionist, wanting to take hip hop back to some golden age of consciousness that never really existed.
These types did (and do) exist, and that's unfortunate. Their overbearing presence, undergirded by the natural, periodic shifting of popular culture, led to the BACKPACKER BACKLASH. During the '00s, music journalists, some fans and others decided that "underground" hip hop had worn out its welcome and that the backpackers were dangerous elitists who needed to be stopped. Coupled with the rising Irony-is-King/hipster movement, this spawned a whole community of people who, while sharing many features of the backpacker (white, male, closed-minded, etc.), instead championed acts like Dipset, Clipse, Young Jeezy, TI, Lil' Wayne and other decidedly mainstream acts.
I think that another aspect of this movement is the whitening of underground hip hop. As audiences at indie hip hop shows became more and more white, the "cool kids" (who were most often white themselves) didn't want to see themselves as part of the co-option of the culture, so they rejected artists with largely white audiences in order to feel or appear to be "down" with what they now perceived to be "THE REAL."
The crazy thing is, as each new movement came into being, the old ones didn't die. Today we have overbearing backpackers AND ironic hipster coke-rap fiends AND genuine fans of mainstream rap AND people who like everything. We're at an interesting cultural point, with all these forces vying for supremacy. Add to this gumbo various race, age and class differences, and every argument about "who are the top five emcees out today" becomes a potential fistfight.
It's interesting to see how identity gets mixed up in music, particularly in hip hop, where the lines between culture and music are fuzzy. It just seems to me that a whole lot of fans, music writers and even artists hardly pay any attention to the music itself anymore, instead obsessing over how liking or writing about a certain artist will make them look to their peers. Big time hip hop bloggers and music journalists don't want to talk about indie acts anymore, because indie hip hop has become synonymous with "nerd-rap" or played out revivalism or elistist backpackery.
On the flipside of that, a lot of indie hip hop IS nerd-rap, played out revivalism or elitist backpackery, so can we blame these writers and fans?
Well, i think we can, to an extent.
Because the obvious point in all of this is that some mainstream rap is garbage and some underground rap is garbage. Some mainstream rap is good and some underground rap is good. "THE REAL" exists all around, from El-P to Ghostface to Ludacris to Lupe. Anyone on either side of the debate who dismisses the other side out-of-hand is really closing his or her mind to some great music.
And that's such an obvious, easy-to-agree-with point that i didn't even want to write it, but time and time again i see people completely ignoring it. I still run into backpackers who have never heard UGK or Lil' Wayne and never want to, and i still run into hipsters who think Lil' Wayne is the Greatest Of All Time and that Sage Francis sucks because his beats don't bang in the ride.
Sure, some of this boils down to taste and personal preference. Some fans just prefer the aesthetics of Timbaland over Ant, or vice versa. That's fine. But it rarely ends there-- these debates are never about personal preference, they're about "YOU'RE AN IDIOT BECAUSE YOU LIKE 2PAC" or "ATMOSPHERE SUCKS BECAUSE THEY AIN'T HOT IN THE STREETS" or whatever. I mean, I don't particularly like the Beatles, but you're not going to catch me saying that they're wack and that their fans are mindless sheep.
I guess the theme in damn near all the articles i've been writing lately is that hip hop is a lot bigger than most people want to admit. It contains within it a vast multitude of styles and movements and currents and identities and avenues through which to interface with it. We're all "experts" on some little aspect of the whole, but few people recognize that whole. And when another "expert" confronts us with his or her knowledge of some other side of the big picture, they come off as completely insane.
"THE REAL" will always be defined differently by different communities under the hip hop umbrella. And i'm not arguing for a cultural relativist stance on what makes a rapper a wack rapper (some things, i still believe, are somewhat objective), i just think we need to examine our own motivations and opinions before criticizing others.
I'll continue this later, maybe by reviving Colin and Jamie from the Hip Hop Panel thing i wrote. They could have a debate-- that'd be fun.