Elastic is moving away from search

SoundCloud: Helps users find the music that moves them

How is it possible to develop a flexible and user-friendly real-time search engine for 30 million users?

Leverage Elasticsearch to intelligently search content and deliver it to millions of music lovers

Create an impressive user experience

  • Use search as navigation
  • Instantly update the search with user-generated content
  • Introducing new functions - quickly and often automatically

Develop and scale easily

  • Take advantage of the high speed of working on system-wide bugs and functions so that a full re-indexing only takes an hour
  • Scale with increasing requirements and with minimal effort
  • Trouble-free maintenance of access via APIs

"We needed a new infrastructure with which we could deliver more relevant results, easily scale them and quickly experiment with new functions and models."

- Alexander Gross, VP of Engineering, SoundCloud

Powerful and intelligent search is the key to a better user experience

The most important thing for SoundCloud is to give customers the best experience of viewing, finding, and discovering relevant content in a timely and meaningful way. As the world's leading social music platform, SoundCloud allows anyone to create, record, promote, and share music tracks in a simple, accessible, and full-featured way. SoundCloud enables music creators to immediately record or upload their own audio content, to embed music tracks in websites and blogs, to share them publicly and privately, to receive detailed analysis data and to receive feedback from the community - directly on the waveform.

Redesign of the search infrastructure to optimize the customer experience

The focus of the new, redesigned SoundCloud is intelligent search functions, which are particularly important for a first-class customer experience. SoundCloud has made the search box the central element that users see. It has also been populated with suggestions that users can use to jump directly to people, music tracks, groups, and areas of interest. The company has also added an exploration area that guides users through the vast music landscape on SoundCloud.

“The most important thing in our redesign was an overhaul of our search infrastructure,” explains Alexander Gross, VP of Engineering, SoundCloud. "We needed a new infrastructure with which we could deliver more relevant results, easily scale and quickly experiment with new functions and models."

SoundCloud is moving from Apache Solr to Elasticsearch

When SoundCloud was founded, the company relied on Apache Solr to find it. Over time, SoundCloud's data requirements grew immeasurably, their social music platform became more complex, and their business rules multiplied. Solr ran into problems with speed, scalability, flexibility, and overall search performance.

"In the world of music tracks, real time is critical," said Alexander. "It is essential that our users' tracks can be found immediately after they have been downloaded, and when we redesigned the website, it became clear that we needed a more powerful and reliable search engine to replace Solr."

After extensive research into the state of the art in search technologies, SoundCloud decided to discontinue Solr and give preference to Elasticsearch.

“The real-time aspect of Elasticsearch is very important. Before that, it took days with Solr before you could search for something, ”says Gross. “We wanted a universal search as a navigation option. With Elasticsearch we get a lot more valuable insights into our data and the system is much easier to configure and maintain. "

Benefits for SoundCloud through Elasticsearch

Shorter search times improve the user experience

The feedback on the time to searchability was very positive. Newly set sounds can be discovered within three seconds - not after days like with Solr.

Add new features more easily

Elasticsearch has greatly accelerated the implementation and configuration of new features.

Simplified cluster management

No tedious manual work is required for bootstrapping and managing the cluster - everything is set up with Elasticsearch's REST API and the index definitions are specified in flat JSON files.

Improved insights help with decision making

A single dashboard with all relevant metrics helps SoundCloud to monitor performance, both in terms of utilization and search quality.