OpenAI introduces JukeBox, an open source AI for creating new music including lyrics and vocals.

The Jukebox AI is trained on vast datasets of music in almost every genre. The AI can create songs that in many cases are very similar to the artists it was trained on. Jukebox explores how it can imitate the style and genre of music. It also tries to mimic the singing style of specific singers.

The OpenAI team chose to imitate music with AI because it is difficult, the team wrote in a blog post. “We show that our models can produce songs from a wide variety of music genres such as rock, hip-hop and jazz. They can capture melody, rhythm, long-range composition and sounds for a wide variety of instruments, as well as the styles and voices of singers to be produced with the music.”

OpenAI uses raw audio to train Jukebox. To get their results, the researchers first used convolutional neural networks to encode and compress raw audio. Then, they used a transformer to generate new compressed audio which was then converted back into raw audio using upsampling.

Artists don’t have to be afraid just yet

Jukebox is very impressive, but artists don’t have to fear for their careers yet. As of now, the songs of Jukebox sound like weird, robot covers of non-existent songs. The AI, for example, is not yet able to imitate large musical structures, such as repeating choruses. It also takes about nine hours to render one minute of music with Jukebox. Improvising on the spot is still impossible.

Another potential problem with Jukebox is the violation of copyrights. Jukebox can create a new ‘song’ in the style of Frank Sinatra, and the final result comes close in terms of sound and feel. Is using musical datasets of Frank Sinatra to create new music in his likeness a copyright infringement in this case? If Jukebox evolves and can mimic more musical structures, this can have unforeseen consequences. The American rapper Jay-Z has already tried using copyright strikes to get synthesized audio of himself taken from YouTube.

OpenAI has created a website for all the samples created by Jukebox so that you can browse through them yourself.