Blending Tech and Tunes, $4 Million Riffusion Raise Lights Up AI Music Scene

Riffusion, an application born as a free and open-source tool that uses Stable Diffusion to produce music from visual cues, recently secured a $4 million investment after its creators pivoted it into a commercial enterprise.

Developers Seth Forsgren and Hayk Martiros initially built Riffusion as a hobby project. Since its inception, it has drawn interest from tech entities such as Meta, Google, and ByteDance. Part of its appeal was its simplicity.

“Users simply describe the lyrics and a musical style, and our model generates riffs complete with singing and custom artwork in a few seconds,” Forsgren told TechCrunch.

The recent funding round was led by Greycroft Partner with participation from South Park Commons and Sky9. In addition, the musical duo “The Chainsmokers” has linked up with the project as advisors.

Music is a universal medium of artistic expression, and generative AI tools like Riffusion, Suno, or Meta’s Audio Craft offer new possibilities for amateurs and professionals to compose and share their creations and interact with each other. For example, there are already several Discord servers and Youtube channels where users share their AI music.

Meanwhile, professional composers are exploring ways to incorporate AI into their creative process.

However, the blending of AI with the arts remains a sensitive topic. Some artists, like Grimes, have welcomed the technology with open arms, harnessing its capabilities to enhance their creative processes. Conversely, artists like Drake have voiced concerns about AI’s move into music.

In 2023, the fusion of AI and music has sparked several controversies. The “No Fakes Act” in the U.S. aims to curb unauthorized AI-generated reproductions of actors’ and singers’ voice and likeness to protect artists’ rights​. Universal Music Group’s concerns about unauthorized training of generative AI using their artists’ music also highlights potential violations of copyright laws​​.

The issue gained steam after the pseudonym artist Ghostwriter published the AI-generated song “Heart On My Sleeve,”​ deepfaking the style and voices of The Weenknd and Drake. The song was even considered for a Grammy’s nomination, but the idea was dismissed as AI-generated pieces cannot be copyrighted under the current legal framework.

Riffusion shouldn’t tread into such thorny pastures.

“The product isn’t built to produce deepfakes and doesn’t recognize famous artist names in its prompts,” Forsgren noted.

While details about Riffusion’s monetization strategy remain undisclosed, its collaboration with established artists suggest potential directions for the platform.

Beyond the realm of music, generative AI has also faced scrutiny in visual arts and literature. Many artists argue that training AI models using their works without consent is both unethical and potentially unlawful.