Component is a section of Aux.Out. for one-off pieces, special editorials, and lost orphans of the music discussion. Today, another cultural landslide creates the biggest Aux.Out. ever on music discovery and the services that help us navigate our musical terrain.
There’s an Elephant in the Music Room.
You may not be able to see it, but it’s there.
A few people actually see the elephant; a few more can’t see it but have noticed that something incredibly huge is taking up a lot of space in the room; yet, most people haven’t even noticed that the room is getting smaller – or that there’s this humongous, pissed-off elephant just sitting there, scowling at them.
Oh. Nobody really wants to talk about it, either.
Instead, we hear about lots of divisive discussions between musicians, labels, tech start-ups, writers, bloggers, legislators, and music advocacy groups over issues like streaming royalty rates, copyright infringement, file sharing, fair use, declines in record sales and digital downloads, termination rights, pay-for-play, venues taking larger cuts out of touring bands’ payouts, legislation to force radio stations to pay their fair share for music use on-air, and so on…
But as those battle lines get further entrenched and fortified, the damned elephant just keeps getting bigger; and unless we all deal with this elephant – pretty quickly, too – none of those problems are going to matter.
It’s a freakin’ huge elephant, too; hell, just look at the amount of real estate it covers: as of December 2013, Spotify claimed 20 million songs; in September 2012, iTunes claimed to have over 26 million songs; and in November 2013, Deezer (a service not yet in the U.S. but available in 150 more countries than Spotify) announced a library of over 30 million songs. A more staggering number comes from The Echo Nest, a leading music intelligence company: at the time of this writing, they claim to have identified well over 35 million known songs by nearly 2.7 million artists that have now generated in excess of 1.168 trillion data points. (If you have nothing better to do, you can actually go to their site and watch those numbers grow by the second…)
And yet, the most astounding figure to date appeared in the December 23rd announcement that the Tribune Company had agreed to purchase Gracenote from Sony for $170 million; hidden within their PR boilerplate was the statement that Gracenote – which started in 1998 as CDDB and now provides music identification & recommender systems for clients like Apple, Google, and Amazon – now tracks over 180 million songs.
Meanwhile, the growth in recorded music is expanding exponentially due to order-of-magnitude changes in recording technology, lowering the nut required for musicians to produce and distribute their music; and musicians are taking advantage of those changes at an astounding rate. Soundcloud just reported that people are uploading 12 hours of new material every single minute…
And yet, realistically speaking, there’s no way to accurately quantify how much music is getting released right now, nor how many musicians there really are; and if you start to include the boatloads of older catalog materials and artists that have never been released in a file-based format, it’s nearly impossible to get an accurate grasp of how much music really exists out in the wild.
But wait: that massive mountain of music isn’t the Elephant. That’s just the real estate sitting under its shadow.
Now, even if we apply Sturgeon’s Law (90% of everything is crap) to just those middle-ground Echo Nest numbers, that would mean there’s at least 3.5 million songs by 270,000 artists that have generated nearly 117 billion data points of not-crappy music, and defining crap is often just a matter of taste…
Speaking of taste definition, each day your average, dedicated music writer is besieged by hundreds of promotional e-mails/press releases/one-sheets practically begging the writer to pay attention to a new release, all while the writer struggles to find an outlet that will accept their article pitches from an ever-diminishing number of outlets – outlets that are themselves struggling to reach eyeballs in hopes of increasing advertising revenues and must therefore tighten their coverage to articles that will draw the most eyeballs: that is, what they believe will be popular to their readership.
A cursory scan from opposite ends of the music writing spectrum reveals that Pitchfork published roughly 1,300 reviews over the last year (not including features or aggregate columns), while a mainstream media outlet like the New York Times featured weekly playlists of five or six non-classical releases and three larger reviews per week – roughly around 450 to 500 reviews.
So… there’s all that music out there, just waiting for a set of ears to hear it, and yet all you ever read or hear about are maybe a few thousand releases?
Say hello to The Music Room Elephant.
Let’s call it “Lack of Discovery.”
Wait – that’s a terrible name for an elephant…
And it doesn’t sound very threatening, does it? You might even find yourself thinking, “Shit, that’s the way it’s always been!”
Match that mind-boggling volume of available music with staggering shifts in technology and listener habits, and Lack of Discovery rears its ugly head as an alarming and incredibly complex set of problems with potentially catastrophic ramifications for most musicians, and for the recorded music industry itself.
And there are no easy answers.
Why Discovery Matters
You’ve probably noticed the heightened use of the term “discovery” within just the last few months; and now that Beats Music has gone live with its “curated by trusted sources” discovery system – and with YouTube due to debut its own dedicated music service and Deezer expected to open up shop in the U.S. later this year – the discussion of music discovery and recommendation systems is about to get a hell of a lot louder.
So, really, is discovery that important?
From the business side of things, you need go no further for an answer than to look at the March 6th pre-emptive purchase of The Echo Nest by Spotify. As reported by Ben Sisario of The New York Times, this deal not only gives Spotify a programming advantage over its competition, but it also seems “to set up a possible conflict for The Echo Nest’s other clients, which include some of Spotify’s competitors” – like Rdio, Deezer, iHeartRadio, SiriusXM, Microsoft’s XBox Music, and Rhapsody – potentially depriving each of them of their primary source for discovery and recommendation data. According to the Times report, The Echo Nest’s CEO Jim Lucchese said his company “would honor its current contracts with these services, but gave no further details.” (Later in this article, you’ll see why this acquisition matters so deeply.)
What’s more interesting: back in November, Spotify raised $250 million in new funding, pegging its total funding since its formation in 2006 at $538 million; if Techcrunch is correct in reporting that Spotify paid $100 million for The Echo Nest – even if 90% of that is in Spotify equity – that’s still an enormous outlay for a company that may have a total valuation north of $4 billion… but has yet to post a profit.
Another measure of discovery’s value to business interests: on January 15th – the day before Spotify & Rdio preemptively announced that they’re now really really free in their attempts to blunt Beats Music’s initial PR push (now the only advantage in subscribing to either Spotify or Rdio is killing the ads in your streams) – Pandora quietly announced that it had added a recommendations function to the engine that runs its limited library of a little over 1 million songs; within 36 hours, Pandora’s stock price spiked up nearly 5% – reaching a market capitalization of $6.7 billion. (To give you some perspective, Universal Music Group, the world’s largest record label group, generated revenues just a little south of $6 billion in 2012.)
These services and their investors see discovery’s effect on user engagement – and more importantly, keeping users engaged – as absolutely vital to the long-term health of their business models; and if you go back though the last six months of music-service press announcements, read them in chronological order and parse them carefully, you might pick up on a sense of increased urgency for these services to emphasize their discovery features.
For musicians, how can discovery be a more important problem than the issues of low royalty rates paid for music streaming, or the claims that streaming is cannibalizing sales?
Well, using simple horse-in-front-of-cart logic, how are you going to collect royalties, or sell your music, or even have a remote chance at a sustainable life as a musician if nobody even knows you exist? Faced with the current overwhelming abundance of music, unless your potential listener has even the slightest clue to look for you, their odds of discovering your music is similar to your odds of plucking a single perfect snowflake out of a raging blizzard.
And if you’re an established artist and you think discovery won’t affect you, then you might be overlooking the long-term impact that this lack of discovery will have on the overall ecosystem of recorded music.
As a listener, discovery could become the deciding factor of whether you stay with downloads or physical product, move to a free streaming system, adopt a subscription model, dump a service entirely, or even give up on looking for anything new to listen to – all because of what or how something is recommended to you.
In his January 3rd essay, The Wall Street Journal’s John Jurgensen writes about facing this abundance of music without some form of easy and meaningful recommendation as “one reason why we fall back on the same stuff we’ve been listening to since senior year in high school,” and even coining a great new term for it: “search-bar paralysis.” Basically speaking, you are overwhelmed by choice… and yet, with so many choices available to you, you have no way of differentiating the signal from the noise. So, in the end, you don’t choose anything at all.
If what you’re listening to is what you’ve always listened to, you are much less likely to plunk down money for a new album or, in the case of streaming services, a subscription; and if you listen to a free version of a service with a narrow spectrum of choice – leaving the vast number of musicians out in the cold, looking in – and are subjected to an ever-spiraling number of ads to make up for that service’s lack of subscription revenue, then you might as well be listening to radio.
And coincidentally, the market that many of these outlets and services now seem to be targeting just happens to be the same turf mainstream radio lives within – a decision that may well be their biggest mistake. (More about that later, too.)
Yet bad recommendations can often be more problematic than search-bar paralysis; in the three months we’ve been working up this article, we’ve seen dozens upon dozens of remarkable tweets about problematic service recommendations – for example, this classic from Stephen T. Erlewine, senior editor of All Music Guide:
Or this one by writer Michele Catalano:
Clearly, there are some big problems facing music discovery and recommendation systems; but now, if it’s so damned important, how do we fix “discovery?”
But That’s the Way It’s Always Been…
If we want to tackle the problems of “discovery,” we need to clear away some long-standing preconceptions and recognize that the technological shift occurring over the last several decades represents something much more than a change of distribution formats…
So, let’s take a very brief look at music’s history as a way to gain some perspective. (Don’t worry, history-phobes. This all ties together in the end. Besides, a little history is good for you. Don’t forget to eat your peas.)
Although phonographic reproduction was invented in the late 19th century – and even at that time, a vigorous format battle was being waged between vertically grooved cylinders and laterally grooved discs – the era of recorded music didn’t really get under way until around 1901 with the introduction of a 10” disc that could play back a whopping 3 minutes of music recorded “acoustically”; i.e., a large horn collected sound, channeled it to a diaphragm that in turn vibrated a needle that etched those vibrations into a solid, wax disc spinning beneath it, thus cutting the master recording directly to the disc. (Now that’s seriously analog audio.)
So, for the sake of argument, let’s use 1901 as the true advent of The Era of Recorded Music: this means the primary method we use for listening to music – recorded audio – is a little over 110 years old.
Pretty long time, eh?
With hard evidence provided within a report presented in the June 2012 Journal of Human Evolution, after re-analyzing the carbon dating of a mammoth bone flute and a swan bone flute (one that plays a clearly-tuned series of C, D, F, and B notes), their creation now clocks in at somewhere around 42,000 to 43,000 years ago.
Why make this point?
1) Human beings have been predisposed to music for a very, very long time. You might even say it’s as if we are prewired for it. (That’s the good news.)
2) Although we face several important musical dilemmas at this time, it’s important to realize we view our dilemmas subjectively, as did our ancestors; to the cave dweller who carved that flute out of a swan bone, the possession of that instrument might have been pretty damned important at the time. (Or, then again, if they’d lost that flute while getting chased by a very hungry bear, maybe not.)
3) In a 2010 interview with The Guardian, Brian Eno stated that “the record age was a blip.” The age of these flutes demonstrates exactly just how small of a blip The Era of Recorded Music constitutes in music’s historical timeline: 0.026%.
That’s 26/1000th’s of all evidence-based music history; and music itself is thought to be much, much older. Many ethnomusicologists and archaeologists think it predates the use of language itself.
4) Therefore, “that’s the way it’s always been” is not how it’s always been. It’s just how things have been for 26/1000th’s of mankind’s total evidenced-based music history.
5) And just because something worked (somewhat) for the last 110 years out of 42,000 doesn’t mean shit. (You have to admit: that’s not much of a track record.)
6) Most importantly: Things Change. If they didn’t, we’d all still be playing C, D, F, and B notes on swan bone flutes. (And we’d still be getting chased by very hungry bears.)
Sometimes these changes occur rapidly – and sometimes, gradually. For example, the Era of Recorded Music didn’t start off with a bang but with a gradual shift away from the predominant form of popular music distribution of the time (sheet music), with recorded music not really taking off until the widespread introduction of radio – roughly 20 years after the initial introduction of the 10” disc.
Some music scholars have cued to that last analogy, equating our current societal shift to digital music technologies to the societal shift from sheet music to recorded music…
But the changes we’re experiencing today are not those of a simple change in format or distribution technology; in fact, we’d argue that the shift to digital technologies – across all creative fields – is similar to the technological and societal shift brought about in 1450 by Gutenberg’s invention of a printing press that used a moveable type system; in other words, something that may happen once or twice every thousand years or so.
Take a look at the parallels:
Prior to Gutenberg’s innovations, there were very few books; each book was either hand-copied or at best etched to wood block and block-printed, both limiting quantity and time-consuming to manufacture; control of information was in the hands of the few who could actually afford these books – the same people who also wielded power over an illiterate, information-poor populace, forcing this populace to accept their elite interpretations of knowledge…
And then things changed.
By 1500, there were over 20 million volumes, with printing presses numbering in the hundreds; by the mid 1500’s, that figure rose to 150 to 200 million volumes from thousands of presses.
With the advent of this revolutionary technology, information could be shared quickly and easily; and with this ability to distribute new ideas across borders and languages came the advent of mass communication. Education became available to many, as literacy erupted across Europe…
As such, this printing technology threatened the power that political/business and religious authorities held over the populace, as it allowed new, revolutionary ideas to spread like wildfire. There was lots of wailing and gnashing of teeth about the unwashed masses taking control (as well as book burnings, bloodshed, political and religious persecution); and yet, printed information just kept growing and spreading unabated.
Now, does any of this sound vaguely familiar to what we’re experiencing today?
Summing up The Renaissance and The Reformation in this manner is a gross over-simplification (be thankful, we could go on about this for days), but much like those who lived within those periods, it’s difficult for our society to understand the absolute enormity of the changes these new technologies have thrust upon us; it’s hard to see the forest for the trees… especially if the forest is huge and dark, the trees are massive and menacing, and there may be some hungry bears around and we can’t lull them to sleep with our swan bone flutes. (Yes, bears and flutes are a theme here.)
And massive change is never a pleasant experience to the people stuck within it; with all the wars and persecution surrounding them, we doubt many people living during the Renaissance said, “Man, this is great!” Yet changes of the magnitude we are experiencing today are not unprecedented: they just don’t happen that often.
So, in essence, we are living within an event that occurs once an eon – wrestling 21st century technologies exhibiting an exponentially accelerating rate of change as coupled to an equally exponentially enlarging volume of music and data…
… and trying to do that using barely-out-of-the-19th century methodologies.
Historically speaking, that’s a #fail.
When facing a change of this magnitude, the primary tenet of the 2,500-year-old I Ching – the Book of Changes – rings true:
We can act with change, or be a victim to it.
And since we doubt anyone involved in music really wants to be a victim, how do we “act” with this change?
Well… first off, we need to start thinking differently.
Rule No. 1: The Music Comes First.
Let’s explore three basic concepts starting off with a simple, yet often overlooked fact:
No two human beings ever hear the exact same song.
Once recorded, the song itself exists in a locked form: manipulations of sounds, silence, instrumentation, and timbre, held within time and space as an unchanging object. Viewed objectively, The Song Is The Song.
And yet, it’s not.
Think of it this way: at an arena show, even though we might be surrounded by thousands of other human beings hearing the same exact musical performance, we still view and listen to that music alone; this experience is individual to us, based upon our own unique set of life experiences up to the moment of that particular musical experience. This is why a performance may be a life-changing experience to one person, while, to another person, it was “just another show.”
The same principle applies to recorded music: each individual’s perception of that recorded object is different – not only based on their life experience up to the time the music is first heard, but also affected continually after it’s heard…
For example, think of a song you once loved that – unfortunately – you also relate to a relationship that ended in a not-too-great manner (it was a train-wreck, they ran your heart through a shredder, and so on). Your primary experience of the song has been tainted by the secondary experience you’ve associated with it; and depending on the depth of your personal disaster, it may take one hell of a long time before you can even listen to that song again – that is, disassociate the painful second experience from your initial experience of the song.
The song didn’t change: your perception of it changed.
The same principle applies to the passage of time: think of the songs you played over and over and over as a kid, and now when you hear them, you wonder, “What the hell was I thinking?”
Therefore, Basic Concept No. 1: A musical experience – and thus music discovery itself – is a subjective, not objective, experience. (One Size Does NOT Fit All.)
To demonstrate our next basic concept, let’s say you come across an album. It could appear on your stream, it could be something you rifled from a remainder bin, you could stumble across it on Bandcamp, or a vinyl copy appears at your door. You know nothing about this release; the artwork could be amazing, or it could be seriously meh-inducing, but what the hell, you decide to listen to it anyway…
And suddenly you have a HOLY FUCK experience – that moment when the music you’re hearing silences everything else in the world and the only thing you can concentrate on is how fucking good this music is and how it has already rewired each and every one of your neurons and possibly even your entire subatomic structure and then the sky rips open above you and a gigantic glowing hand descends from the heavens and gives you a BIG “thumbs-up” and you happily nod in absolute agreement and give it a BIG “thumbs-up” back and none of this exchange seems the slightest bit unusual to you since you aren’t thinking about anything at all except how this music is washing over you like a cleansing and renewing wave of cool, fresh water except it’s not water washing over you but a whole-body sweat generated from you dancing around the room ecstatically since you’ve lost control over your body to this music that now owns you and has already shaken you to the very core of your being and now you just can’t imagine living life without it and you just can’t listen to it loud enough and when it’s over all you want is more more MORE…
After you’ve recovered from this experience (or not), what’s the very first thing you instinctively want to do (other than listen to it again)?
You want to tell somebody about it.
You want to share that experience.
This Is How Music Works.
Basic Concept No. 2: Discovery is excitement; it is passion realized; it’s a HOLY FUCK experience to be shared.
And being able to tell somebody about it? That’s Word-of-Mouth.
Word-of-Mouth has always been music’s ace in the hole; it’s still the most powerful (and yet, due to Basic Concept No. 1, the most unpredictable) marketing technique known.
If a good friend with whom you share similar musical tastes were to show up at your front door dripping in sweat and clutching a phone, iPod, disc, or vinyl, spouting, “HOLY FUCK YOU HAVE GOT TO HEAR THIS SONG” – well, most likely, you’re going to be intrigued enough to give it a listen. (Or call the police, depending on the friend.)
And that sweaty friend who’s clutching their phone, iPod, disc, or vinyl spouting, “HOLY FUCK” at your front door illustrates the second part of a word-of-mouth experience: they want to receive a reciprocal outcome from sharing their experience, since they’re at your door hoping you’ll like it, too. (And also hoping you won’t call the police.)
In other words, Musical Word-of-Mouth is a two-part, shared process: a) being able to share a musical experience and, after having shared it, b) achieving a shared consensus about the music that can or will be shared again (Basic Concept No. 3, if you’re still counting).
So… when we talk about building a successful discovery or music recommendation methodology, we’re talking about building a system that can automate or facilitate a subjective word-of-mouth experience – one that actively creates excitement or engagement.
Strangely enough, it’s been built before.
The First Online Music Recommendation System
In the early 1990’s, Pattie Maes – an AI researcher at MIT’s Media Lab Software Agents Group, and one of the pioneers of software agent technology – felt frustrated when listening to Boston’s radio stations. A native of Brussels, Boston’s commercial radio scene seemed way too bland and restrictive – not nearly eclectic enough; worse, she couldn’t find any other way to discover new music to fit her tastes…
So, Maes and a group of research assistants at the Software Agents Group decided to prototype a new form of software agent named HOMR (Helpful Online Music Recommendation, later renamed “Ringo”) designed to function as a form of “electronic word-of-mouth.”
The idea was simple (but the work to build it wasn’t): create a software agent that would ask its user to scale-rate a list of artists, with users specifically advised to rate these artists for how much they liked to listen to them, thus creating a basic user profile. The software agent would then search its database, find like-minded users, compare this new user’s ratings to those of the like-minded user’s pool, and email back personalized recommendations based on that comparison.
This type of agent technology is now known as “collaborative filtering.” Limited forms of collaborative filtering existed at the time, but none had been put to use in such a novel and openly user-based manner. And it wasn’t “intelligent” software; it was an artificial intelligence built on a database of the total knowledge and choices made by its user base. As the user base grew and each user rated more albums, not only would the agent’s database grow in size – its accuracy in recommending new music would grow as well.
In 1994, Ringo went live as an email-based system – later introduced to the nascent World Wide Web using a bare-bones graphic user interface – and was enormously successful, especially for a quietly introduced academic project… but creating this agented instrumentality was only a part of Maes’ thinking: what really intrigued Maes was how a collaborative filtering agent could be used to foster and build a community.
In January 1996, Firefly – the first commercial music recommendation system – went live and blossomed immediately; but Firefly stood apart from its previous Ringo iteration in allowing its users to set up their own profile pages, write reviews, and most importantly, not only offering to connect the user to those other like-minded users to see what they were listening to, but allowing users to actually contact and communicate with each other, thus automating and fully facilitating the electronic word-of-mouth experience.
For those who were lucky enough to have used Firefly, it was an amazing experience. Talk to former users of Ringo or Firefly and they’ll tell you how much their record-buying skyrocketed as they were introduced to new bands they didn’t know existed and how it widened their musical horizons; and since the user base was made up of music geeks and active listeners (we’ll be using that term quite a bit shortly), by giving users a forum to expound on their enthusiasm, they then made new recommendations well beyond the ken of the recommender agent – all of which were fed back into the recommender engine itself. Firefly’s database grew rapidly and exponentially – as did the user base itself.
In its initial commercial iteration, Firefly worked like a charm. At its height, it boasted 3 million user profiles; again, this was pre-broadband, pre-download, and pre-Internet Bubble 1.0, so that number represented an outstanding installed user base when compared to the prime movers of that period (AOL with over 4.6 million users and Compuserve with approximately 2.6 million users)…
And, true to Maes’ thinking, it became a community – one made up entirely of new connections bonding over music they loved…
But nothing good lasts forever.
As the site boomed, the collaborative filtering technology behind Firefly caught the eyes and ears of advertisers and retailers of all sorts, and suddenly there was a mad rush to either license or develop competing collaborative filtering platforms. Firefly became a victim of its own success, diversifying the recommender engine toward other areas of interest, like appliance recommendations, causing Firefly to lose its primary identity – and many users as well.
In 1998, Firefly was sold to Microsoft during their Engulf and Devour phase for $40 million (pretty decent money pre-bubble). With the filtering technology much more meaningful to Microsoft than a hardcore gaggle of music geeks geeking out over music, Microsoft eventually shuttered the site in 1999, porting its technology into their new Passport software.
At that time, all that mattered to Microsoft was the agent technology used to aggregate and filter user data for purchasing recommendations – with the human interactive/community element that actually created that data completely removed from the equation.
And, in our opinion, that’s the exact point where automated music recommendation technology went horribly wrong.
Modern Recommendation – aka, The Mystery Grind
Recommendation and filtering technologies have changed radically over the last 20 years, and now music recommender systems are incredibly advanced iterations containing variants and hybrids of collaborative, content-based, knowledge-based or demographically-based filtering systems (amongst others) that use immense volumes of data points and multiple methodologies to replace a solitary participatory model.
For example, Pandora’s Music Genome Project is a famously “content-based” system of filtering based on a fairly small pool of music, with each song individually analyzed by a trained music analyst – a human being – against a list of 400 to 500 characteristics/data points that may or may not be inherent within the song, then combined into larger groups based on traits demonstrated in common; yet in practice, the Music Genome Project itself is only one of a core group of technologies in use by Pandora.
All of these current iterations are the result of tons of investment, extensive research, and some highly impressive work in a relatively nascent field. (According to an October 2013 RAINNews interview with Jim Lucchese, CEO of Echo Nest, only around 12 people per year graduate with advanced degrees in music information retrieval.) Beyond just the rudiments, these models are tightly held trade secrets, and the work behind them could well be considered “rocket science.”
Even though there have been incredible advances in recommendation technology, the decisions in how a music service decides to implement that technology (weighting in additional factors like existing popularity, product promotion, recommendation repetitions, and so on) ultimately becomes the weak link in the chain, affecting the overall quality of the recommendations…
And our society has become increasingly reliant on recommendations. We now lead exceedingly busy lives, and personal discovery can be time consuming; as we have grown accustomed to recommendation technology, faced with a growing mountain of information to sift through and diminishing available personal time, we have gravitated societally toward the convenience of being told what to look for or like; consciously or unconsciously, we want someone or something else to do our filtering for us, because we just don’t have the time to do it ourselves.
This could be why 75% of Netflix’s viewer activity comes from their recommendations; even if we feel that most of what is recommended to us is crap, we still continue to use this function because it beats the alternative of manually searching for things to watch (coupled with the frustrating experience of discovering that the item we’ve searched for doesn’t exist within their rights base).
The jury’s still out on the question of whether these types of recommender systems serve to fragment or homogenize choice; although there’s now a common acknowledgement that these non-participatory systems do drive consumer choice and sales, researchers know far less about how these systems actually affect markets and society as a whole.
What’s incredibly important, however, is to avoid dismissing the participatory community aspect of Firefly’s platform performance as obsolete, nor to confuse it with so-called “social music discovery.” The concept of “social community” is so much more than an app blankly spamming a line of Facebook newsfeed that says, “So and So is listening to The So So Glos…”
An active community enables a recommender to facilitate new associations without the use of a previously assumed association – that is, it allows a recommendation platform to make new, unexpected, yet completely valid and accurate associations based on individualized human interactions.
There are no music recommendation systems currently using this type of “explicit” data – that is, data derived directly from users engaging with other users – so today’s music-recommendation user has no real knowledge of what it’s like to receive active, community-derived recommendations. This experience is similar to a person who has never had coffee except for that liquid that oozes out of office coffee machines, made from mysterious no-name, pre-ground packets of an indeterminate age – or worse, “instant” coffee (a criminal blight on humanity we sincerely hope has faded into the distant past). To this individual, this coffee is the only kind of coffee they know; at best, it serves as a simple caffeine-delivery system and, as such, serves its function well (well, sort of).
This aspect also applies to recommender systems using “implicit” data: at least you get something that sort-of gets the job done.
When compared, however, to freshly brewed coffee made from freshly roasted whole beans, it’s damn hard to go back to drinking The Mystery Grind (except when you desperately need some form of caffeine, any caffeine, just to get through the day – in which case you’ll drink just about anything – or even chew the leftover grounds if necessary).
So, with very few users having “tasted” anything other than our current “Mystery Grind” forms of recommendations, their expectations are lower, and there’s no reason for them to get excited when it comes to music recommendations derived in this manner.
We are told that, in principle, within modern filtering technology, data can be implicit or explicit & derive the same results; but that explicit data is still much more powerful, as it cuts down on the “noise” factor (semi-related results that might not be accurate, like “if you like cream in your coffee, you’ll love coffee ice cream”) – and that discussion forums make far better communities than anything within current music services.
The power of this type of active community within a recommendation methodology cannot be understated: truly open, word-of-mouth interaction is having the ability to expound on and discuss something as arcane as why anyone on earth should listen to The Nutty Squirrels “Uh Oh, Part 1” (hey, it charted in 1958, and it’s a freakin’ ear-worm)…
And through this ability to actively channel recommendation, a community of like-minded individuals shares their passions and enthusiasm – that is, they get real freshly brewed coffee instead of The Mystery Grind – and they in turn get excited & engage, not just with the platform, but with other listeners. This effect creates an increased interest in the music and in the platform itself, thus generating a self-sustaining momentum from users who actively choose to stay engaged on the site longer and, more importantly, return often: the two most desirable factors for any site or platform’s success.
There’s Always a Problem
It would be an oversimplification to say that automated algorithmically derived discovery doesn’t induce excitement or engagement as effectively as models involving an active user base (although we firmly believe that); rather, all recommendation systems are affected by several factors that might be amplified from utilizing abstracted or implicit data, rather than explicit user data.
Let’s start with the problems of “cold-starting,” “weighting,” and “recommender persistence.”
Cold-starting is just that: the first time you use a service, the service needs to build your user profile from scratch; this step really hasn’t changed much since Firefly…
Yet, our newer services operate from the perspective that over the past 20 years, users have come to expect any app or platform to work immediately and therefore believe they need to grab you from the get-go – engage you or lose you – so they must throw something at you to get you going.
This type of cold-starting problem currently gets addressed by the user selecting music from a list of genres, or a service extracting a playlist from a “seed” song that the user suggests, or from asking the user for a very short list of favorite bands or musicians; and then, as the user listens to presented choices and up- or down-votes the song or album (either explicitly or by skipping the choice entirely), a more detailed user profile gets built. It’s quicker, but it doesn’t generate truly accurate results unless the recommendation service gets used heavily.
Other services start recommendation immediately using demographically or geographically derived implicit data; for example, if Spotify’s recommender could access our current iTunes database that it pulled into its desktop player (well, 68 days of it – we needed drive space, so we trimmed things down a bit), it would have gathered some actual music data as a starting point. But when we open the app, what pops up on the main discovery screen?
Gordon Lightfoot. Three Dog Night. Harper Valley P.T.A. by Jeannie C. Riley. Darius Rucker? Michael ‘Effin Bublé? TONY ORLANDO AND DAWN?
This is The Music of Our Nightmares. (Sorry. Just had to bitch about this.)
What Spotify is doing is weighting their initial recommendations demographically to relative age and the geographic area we live in. (Yes, we live in Hell.) Each recommendation is followed by a qualifier: popular in your area, popular when you were in school, huge when you were a teenager, and so on.
As you search out music to listen to, Spotify learns from this data and contours its recommendations accordingly; still, even if it had peeked at only the first song in its own player’s directory (12XU by Wire), perhaps it wouldn’t have thrown Tony Orlando And Dawn at us. (And we have used Spotify, so the agent should have known better.)
As is, allowing a service to search an existing base of songs on your device – while getting much closer to actual explicit data – still introduces a great deal of noise into recommendation data, since most people don’t actively rate their collections; but searching pre-existing libraries also runs into the wall of user privacy – the user must actively grant access to this process – and therefore could be legally problematic. Services like Rdio actually do give the user the ability to allow Rdio to import their existing iTunes database and then adds more music from artists already existing within the user’s “collection;” but at least from our experience, this data does not filter into their recommendation base, and the service instead bases its recommendations on what you have played through the service itself.
A different type of cold-starting problem also comes from the introduction of new bands or musicians: no one knows this material, so how does it get recommended?
Through analyzing the music, data points pertaining to this new music are generated; this data is then entered into the recommender database and made available… but wait: if that’s the case, why don’t you see these albums recommended to you immediately instead of music by more well-known musicians?
This is the function of “weighting.” Weighting is primarily found in the process of further recommendations: as the listener uses the service, the recommendation system gathers usage data and relates it to its database; at that point, however, certain components are combined in an effort to represent the best possible results the recommender can associate to the listener’s current choices. In other words, the recommender’s best guesses are weighted by data points such as popularity, name recognition, reviews, press, PR, or any of thousands of search-gathered factors – with name recognition and popularity often the top two factors.
Yet those weighted results may not be what the listener is looking for, thus producing a “false positive”; and depending on the service’s interests, this step can also become a point where false positives can be purposely introduced. (We’ll get to that subject very shortly.)
For weighting to work well, transparency of this process becomes vital; for example, Rdio’s weighting mechanism within their “station” function is by far the most transparent, in that it allows the user to progressively remove weighting from within their station recommendations by allowing the user to “dial up” the degree of “adventure” to the recommendations – leading to artists or tracks that have not received as many plays or are wider in relation to their initial starting point. (Kudos for that feature.)
Still, some recommender systems have no transparency at all to their weighting methodology, so you really don’t know much about why something has been recommended to you – possibly lowering trust in the recommendations and thus lowering engagement. (We’ll cover Trust shortly as well.)
Finally, repetition of a recommendation gets featured within some services; its use comes from the advertising concept of “effective frequency”: the belief that one impression never completely registers with a viewer or listener – and that it may take between 8 to 40 repetitions to register a true impression. Repetition in music services, however, can be viewed by a skeptical user as an attempt to push an artist onto a listener – who then questions, “Why the hell does (name of service) want me to listen to Chubby Checker?” – and that, too, can lower the user’s trust in the recommender system.
Trust Us: We Know What’s Best for You
Automated recommendations really have no value unless the user consciously or unconsciously trusts the recommender system. We often use “trusted sources” surrogates for this function in daily life. For example, even with our ability to click and listen for ourselves, a review from a respected writer who tells us to stick with a whole album – or that a new release will grow on you – can often tip the balance of whether you give a track or an album its full due instead of skipping ahead after 20 seconds.
This was an area where active human recommendation systems excelled, due to word-of-mouth; fully automated systems, however, do not give that depth of insight currently – usually all you’ll receive is a “Because You Listened To…” line as a reason for the recommendation.
Trusted sourcing falls one step below word-of-mouth in recommendation effectiveness; these recommendations are not personalized but can be used as a substitute methodology for word-of-mouth in two ways: 1) you trust the source’s tastes, and therefore you’ll give it a go or 2) you can’t stand the source’s tastes, so you end up using the “if they liked it, I’ll hate it” protocol – or conversely, the “if they hated it, I’ll love it” protocol. (The latter works extremely well with film criticism.)
A newer substitution for trusted sourcing is the use of “curation.” Playlisting can be a form of curation and, depending on the source, can serve recommendation well; yet it still does not carry the weight of true word-of-mouth recommendation unless the playlist is created specifically for an individual as if it were a personalized mixtape. (The nice thing about mixtapes: since everyone knew how long it took to plan and record a mixtape, they were always personal – and possibly the penultimate form of musical word-of-mouth.)
Still, “curation” moves one step further away from trusted sourcing itself, as it delegates sourcing to the generation of thousands of playlists built by known personalities, critics, genre classifications, and unknown sources…
… but if you think about it, this type of curation could possibly negate trust itself; for example, in viewing a highlighted celebrity’s curated playlist, how do we know that this person actually produced this list? Could it have been created by an assistant? Were they paid an endorsement fee to create this list? Are the tracks truly representative of what this celebrity listens to, or are they just paid placements as well? And do these lists serve to dig deeper into music’s immense catalog? If we trust curation blindly, we’re fine; if we don’t, well…
And that pretty much sums up the main problem with curation: it brings you right back to some of the same issues introduced in weighting (you have no idea why this track you have never heard of is there, how it was chosen, if there was an agenda behind its placement, and so on); but unlike weighting, with curation you are actively choosing to abdicate your choice to someone else you do not know, nor know their agenda (as we have done for the better part of the last century with radio and television). You are expected to simply accept what you are given and move on.
For this reason, curation does not serve as true recommendation; it’s simply a different approach to a “lean-back” style of casual listening as provided by services like Pandora or terrestrial radio…
And due to new factors involved within rights-base licensing, there is a distinct possibility (some would say probability) of curation eventually falling away from any true recommendation purposes to become a form of paid promotional streaming – and not true discovery.
Unlike the Firefly model – a model that had no restrictions to the music it could recommend – today’s recommendation engines are restricted to making recommendations within a finite amount of music that a particular store or service has licensed for streaming or for sale. A common misconception is that the “magic number” of 20 million songs means that all major music service rights bases are identical; but as discovered by Liam Boogar, editor of the French start-up and tech blog Rude Baguette, in doing a test of importing playlists from Spotify to Deezer, Boogar found there were songs in Spotify – the smaller rights base – that were not within Deezer’s much larger rights base.
This is an example of “fracturing” (also known as “fragmenting”). Not all services carry the same music for reasons of library size (as noted earlier, Pandora’s base is barely over 1 million songs), exclusivity (Led Zeppelin’s streaming catalog, assigned only to Spotify), rights licensed on a country-by-country basis, or limitation by some digital aggregators that exclude certain stores and services from their catalogs (or vise versa).
In essence, there is no single universal music rights base or database: the music base is therefore “fractured” – not uniform – especially when considering outlets like Bandcamp and free music service outliers whose music base never gets included in the “magic number” due to their operational models.
Yet there are two newer forms of fracturing on the horizon that are much more ominous in nature.
In November 2013, Deezer announced on its blog “a pioneering collaboration with acclaimed Icelandic newcomers, Mono Town,” giving Deezer an unstated period of exclusivity to Mono Town’s music. This is noteworthy for two reasons: 1) Mono Town is a new band, not one with a catalog and 2) this deal went far beyond exclusivity: the service actually possessed a proprietary interest in promoting an artist – as if the service were instead acting as a label – thus opening the possibility of skewing its recommendation results in favor of proprietarily monetized streams; that is, if Mono Town were to appear in your Deezer recommendations, would it be due to your love of Icelandic Pop or that Deezer has a proprietary interest in you listening to them?
Moreover, Deezer states that this agreement “is just the start of what we hope to be a music industry revolution” – that is, more agreements of this type, not only fracturing the rights base further but crossing some ethical lines in the sand.
Speaking of crossing ethical lines, the other much more ominous fracturing comes by way of Clear Channel Communications, the largest radio station group in the U.S. and creators of iHeartRadio; in facing a distinct near-future possibility of legislation that would require U.S. terrestrial radio stations to pay performance rights, Clear Channel has been quietly doing an end-around of the CRB/SoundExchange compulsory royalties process and making favorable direct-deals with small labels like Taylor Swift’s Big Machine and Glassnote…
… but the real coup occurred in September 2013 when Clear Channel made a deal with Warner Music Group to collect royalties for on-air play of WMG product, in exchange for receiving more favorable rates for their iHeartRadio streams – and better yet, to sweeten/enable the pact, WMG artists will also get extensive promotion both on-air and online.
Now, before you start screaming, “That’s payola!” In one of the niftiest sleight-of-hand movements in radio’s spotted history, as reported by Ben Sisario of The New York Times, Clear Channel has stated that its on-air promotions and play of WMG’s artists would not violate payola statutes because this promotion & play would be considered a part of its commercial inventory and not part of its regular programming. (Pretty slick, eh?)
And as for the streaming side of the equation? Music distributed through digital transmission streams is not subject to over-the-air payola statutes, so this is a clear win-win for Clear Channel.
In his comments to the New York Times, David Macias of Thirty Tigers states the WMG deal “could have a chilling effect on independent artists and what creative expression is going to look like.” But in his scathing Hypebot op-ed, Macias goes even further, writing that when radio and digital are “financially incentivized to play music only from certain sources, and you’re on the outside of that, you are less likely to have a career, whether you are an artist or working in the service of one.” Finally, he also states, “The market for pre-recorded music just got less competitive. And I respectfully suggest that you ain’t seen nothin’ yet.” (We wholeheartedly agree.)
Buses, Whales, Listeners & Logic
Okay, now we’ve come to the part of the movie where we start to see the wheels slowly falling off the speeding music discovery bus…
… y’know, the bus that’s carrying all the current economic models for the entire recorded music business. That bus.
Fred Wilson, a leading venture capitalist with Union Square Ventures (whose portfolio contains far-sighted leading investments in Kickstarter, Soundcloud, and Twitter), characterizes subscription music services as “proprietary commodity services” and points out that it’s nearly impossible to share recommendations across services due to the very nature of their business models: a missing functionality that hampers music discovery, listening, and growth…
And growth is the No. 1 factor here….
Right now, on the surface, things may appear to be growing – “wow, look at all those quickly multiplying number of streams!” – but with the very nature of these siloed business models artificially imposing limits on discovery and recommendation, it’s more likely we’re starting to stagnate instead…
Because even with the services that exemplify the very best and most progressive discovery systems available today – Bandcamp with its “fan pages” and music feed and Rdio with its ability to dial up the listener’s level of discovery – the walled-in nature of all siloed services has severely fractured the true base for listener recommendation and usability, limiting music discovery and market growth under the guise of “competitive” or “proprietary” models.
Now, as these services grow, the cost of their licensing rights grows even faster…
Ooooops. We’re getting ahead of ourselves, and we want to save the speeding, out-of-control bus swerving and then rolling over and over on its side only to crash into all those giant, flammable music storage tanks and the ensuing multiple explosions photographed from eight different angles in slow motion and THX®surround sound and all the carnage and screaming and all the burning people and mayhem – and all that other good stuff – for the end of this movie.
(By the way, that’s called a “foreshadowing tease.” Get your popcorn ready.)
Instead, let’s do a quick cutaway (with yet another piece of foreshadowing hidden inside it) and look at music’s actual audience…
On Paul Lamere’s Music Machinery blog, there’s an excellent post about his Zero Button Music Player project. Within that post, he quotes and actually graphs out David Jenning’s summary of Emap’s Project Phoenix, a pair of UK-based music studies, into behavior patterns among music listeners.
Those studies identified four basic levels of musical interest among its participants: “savants” (everything in life is tied to music), “enthusiasts” (music is key, but there are other interests, too), “casuals” (music is welcome, but other things are more important), and “indifferents” (who “would not lose much sleep if music ceased to exist”; it’s hard to imagine people like that exist, but, evidently, they make up the majority.)
We’ll group these categories one step further into two basic types – “active” or “casual” – using as our dividing line whether music was important or not to the respondent’s life; in doing so, we see that “casual” music listeners are by far the larger group (72%), while “active” listeners make up 28% of respondents in these studies.
You probably know what a casual listener is, but what constitutes an “active listener?”
Simply put: the active listener seeks out music.
In the past, they were the ones who rifled through used and remaindered sections in record and discount stores, always looking for the score everybody else had missed (sweet!); they dug through import bins and searched out obscure band catalogs; they were the people who knew Nick Drake’s entire catalog by heart long before Audi used Pink Moon in a commercial soundtrack, which finally introduced his music to a wider audience over 25 years after his death.
Active listeners are influencers: it’s their passion and enthusiasm that carries over to other less active listeners, and that momentum continues toward borderline casual listeners, who then spread the word to others; they embody the starting point for artist exposure.
For example, Lorde’s “The Love Club” EP first appeared as a free Soundcloud download in 2012, backed by very little non-traditional PR; for an unknown New Zealand artist with no track record to become recognized, somebody had to be the first listener – an active listener – who then passed their enthusiasm on to other active listeners and so on… and you know the rest of the story.
Uh oh… we used the “free” word, which means someone is bound to bring up a “dark” point about active listeners, so we’ll bring it up ourselves: yes, active listeners have also been the largest group of active file sharers.
But let’s be straight about this: active listeners are not necessarily file sharers. File-sharing behavior simply illustrates the musical appetite of active listeners; and yet, for all the noises using the recording industry’s favorite über-scary pejorative term – piracy – study after study after study has revealed that file sharers are also the most active buyers of music.
And they buy music, not just rent or lease it (as in streaming). They collect it.
Back on-point: with digitally licensed services enabling file sharers and active listeners to discover new music through legal means, it’s now essential for the recorded music world to re-embrace active listeners and develop discovery mechanisms that allow them to easily work their way through our current, massive maze of music.
The reason is simple: to use a software/hardware analogy, active listeners are music’s equivalent of “early adopters”; they’re the experimenters, the ones who will try something out when no one else will. They do all the heavy lifting.
The act of creating music is an organic process that builds upon all music that has come before it. You can even visualize the world of music creation as one self-contained ecosystem, with mainstream music and casual listening at the top of the food chain, with this ecosystem constantly fed by experimentation and change that almost always comes from unknowns, outsiders, outliers, and scenes far distant from the mainstream.
Better yet, think of the mainstream music world (and legacy artists) as blue whales: the largest mammal on the planet.
Blue whales feed almost exclusively on krill, some of the smallest creatures in the ocean; krill feed on phytoplankton, microscopic, photosynthesizing organisms that actually sustain the aquatic food base – with phytoplankton reliant on micronutrients welling up from the ocean’s deeper waters to grow and multiply.
If the ocean’s health becomes incapable of sustaining an adequate supply of micronutrients, you have no phytoplankton.
If you have no phytoplankton, you have no krill.
No krill? No blue whales.
Without care for the smallest elements within an ecosystem, stagnation or extinction occurs.
Now, we’re not making this example to endorse a movement for all mainstream musicians to Eat an Unknown Musician Today (though that would make for some radical bumper stickers). It’s just a method to illustrate how music that may seem a bit odd or off-the-wall eventually finds its way into or feeds new mainstream music movements.
And this is why discovery is vitally important to all musicians, including those at the top of the charts: in essence, the active listener is the initial nutrient base for music that’s new or different. If you do not create a hospitable environment for this nutrient base that feeds and nurtures this element of music, eventually the entire ecosystem will stagnate or collapse.
To put it bluntly: the active listener fosters music discovery and music growth. Care for them – or face the consequences.
Now, look back at those Emap Project Phoenix graphs again: 72% of listener types do not make music a priority – that is, they do NOT spend money on music…
And yet most of our current music services view the larger casual listener percentage as the true untapped potential mass audience and are directing their efforts to capitalize on that potential audience. (More about this in our upcoming Exploding Bus section.)
So, wait… doesn’t that model seem a little “out of whack?”
Why not concentrate on the 28% of listener types who DO make music a priority instead? Why not pour our efforts into revitalizing that sector?
Why not try to grow the base instead?
If we truly want our music ecosystem to thrive, then perhaps it’s time to think about building a new all-inclusive, non-weighted music recommendation platform – and at its core, an open community of active listeners generating explicit recommendation data.
Yeah, we just said that: build a new active recommendation platform.
By developing a favorable discovery environment that allows active listeners to gather and explore this immense volume of music, not only will this recommendation environment become self-sustaining, but, more importantly, we can design an environment that will actually grow the base of active listeners rather than simply try to divide the existing resources (the mainstream pie of casual listeners) into smaller and smaller pieces.
And with Spotify taking The Echo Nest out of circulation and eventually hoarding all their recommendation resources for itself, simple logic leads us to believe that now would be the ideal time for somebody to jump into this void and build a new cross-service recommendation platform, with features specifically designed to enhance continued engagement and growth.
What kind of “specifically designed” features?
We’re glad you asked.