Musician-led AI initiatives [Unfair use? series, Part 3] 🗣️

A running list of initiatives taken by individual musicians to own and drive how generative AI is used with their music and voices. (Audio; 8:14)

Much of the news and analysis about generative AI for music focuses (with good reason) on unethical, exploitative uses. However, musical artists are increasingly exploring ways to use AI under their own control.

This page lists examples of musicians who are taking AI for music into their own hands (lists are not exhaustive). It will be updated as new artists announce their experiments and experiences with AI.

Photo by Soundtrap on Unsplash

By default, we assume artist-focused initiatives like this are ethical. The artists, or the people who own the rights to their work, consent to use of their content and their name, image, and likeness. And they can control its use, credit, and compensation. That covers all 4Cs.¹

Musician-Controlled Initiatives

Most initiatives to date involve voice cloning, which we wrote about on April 29. Here are some examples of artists creating their own AI voice models.

1. Holly Herndon

The “Queen of machines” created a critically-acclaimed AI-led album called “Proto” in 2020. She now runs a site called Holly+ (via About button on holly.plus) which is “the first tool of many to allow for others to make artwork with my voice, and will distribute ownership of my digital likeness through the creation of the Holly+ DAO”

“Holly Herndon: Computers surprise you in a way that an acoustic instrument doesn’t”, by Computer Music, 2020-12-07

2. Grimes

In May 2023, experimental artist Grimes created her own Elf.Tech service to enable and allow fans to create recordings using her voice, in exchange for 50% of the royalties. Over 15,000 people experimented with the new feature in the two days following its release.

1. “Grimes invites fans to make songs with an AI-generated version of her voice”, by Vanessa Romo, 2023-04-24;

2. “Grimes on AI-generated vocals of dead musicians: “I feel like someone like Prince would probably be down””, by Tamzin Kraftman, 2023-06-12.

3. FKA Twigs

“FKA Twigs uses AI to create deepfake of herself”, Ian Youngs / BBC News, 2024-05-01. The British singer, dancer and actress used AI to create a deepfake of herself that can interact with fans and journalists, and free up her time to focus on her music. She recently testified at a US Senate Judiciary subcommittee on AI about the importance of artists controlling their name, image, likeness, and voices, as well as their content. She is quoted as stating about her deepfake: “This, however, is all under my control and I can grant or refuse consent in a way that is meaningful.”

4. Frostbite Orckings

This group is an exception to the voice cloning approach. The Frostbite Orckings were certified as Fairly Trained early in 2024. They are a “purely virtual metal band whose music is created through the use of artificial intelligence, specially trained with the style of "melodic death metal”. All training material is based on input created by the team members themselves.”

Voice Restorations

A few musicians who are no longer able to perform as they did in the past have now used AI to ‘restore’ their voices and revive their careers.

1. Randy Travis

US country musician Randy Travis is a good example. 10 years after Travis suffered a stroke that impaired his voice, an executive at Warner Music Nashville suggested he try using AI. The resulting song “Where That Came From” is here.

2. Beanie Sigel

Another example is rapper Beanie Sigel. His voice was damaged after he was shot and hospitalized in Dec. 2014. In July 2023, he announced that he was “gonna use AI on myself” to recreate his own voice for a new album (ref, YouTube ref).

Collaborations ‘with’ Deceased Musicians

In other cases, musicians have used technology to combine their own performances with the voices and music of deceased collaborators. Since the dead cannot consent, ethics of this practice can be controversial (refs 1, 2).

Some living artists have begun speaking out about their wishes on this topic in advance. “Tyler, the Creator has voiced what many others have felt about the posthumous album trend. At an April 26 concert in Los Angeles, he noted that he’s written it into his will that he does not want any unreleased music put out after his death.” (ref)

A notable example of an unauthorized collaboration in May 2023 was producer Timbaland releasing a song clip with the recreated voice of Notorious B.I.G., who had died 25 years earlier.

Here are some famous examples where the people involved appeared to have the consent of the owners of the rights to the musicians’ work.

1. Natalie Cole ‘with’ Nat King Cole

In 1991 (pre-AI), Natalie Cole worked with Joe Guercio (Elvis Presley’s musical director) and David Foster (her producer) to originate “virtual duets” (refs: 1, 2, 3). Her father, Nat King Cole, had died of lung cancer in 1965. Natalie decided to record an album of tributes to her father’s music. Her team used technology to ‘record’ a duet of her with her father of his classic “Unforgettable” for her album. In brief, it was produced by dubbing her vocals into his 1951 recording. You can listen to the duet here.

The virtual duet was well-received: it peaked at #3 in the US, #1 in Canada, #2 in Australia, and #19 in the UK. (ref) It won multiple Grammy Awards: Record of the Year, Song of the Year, Best Traditional Pop Performance, and Best Instrumental Arrangement Accompanying Vocal (for Johnny Mandel). The song also won Irving Gordon (who had written it 40 years earlier) the Song of the Year award. (ref)

Natalie died in 2016 and the Songbook Foundation acquired the rights to Natalie’s music from her estate in 2019. (ref)

2. Paul McCartney ‘with’ John Lennon

Many years ago, Yoko Ono (John Lennon’s widow) had given Paul McCartney cassette tapes of a solo home demo, which Lennon had labeled “For Paul”. The sound quality was too poor to work with, at least until newer AI-based tools became more powerful. In mid-2023, McCartney used AI to clean up Lennon’s demo. McCartney’s goal was to get it to a releasable state as a “final Beatles song”.

“The turning point came with Peter Jackson's Get Back documentary, where dialogue editor Emile de la Rey trained computers to recognise the Beatles' voices and separate them from background noises, and even their own instruments, to create "clean" audio. The same process allowed Sir Paul to "duet" with Lennon on his recent tour.”

Ringo Starr provided the drum track. “Now and Then” was released in Nov. 2023. You can listen to it here or here.

3. Kim Kwang-seok

In 2022, AI voice cloning and synthesis company Supertone used AI to recreate the voice of late South Korean folk artist Kim Kwang-seok. (This was around the time that Supertone was acquired by HYBE.) The video was released on Dec. 18, 2023.

4. Teresa Teng and Anita Mui

Tencent Music Entertainment (TME) has developed a patented AI-based “Lingyin Engine” for generating music. MBW reported in Nov. 2022 that TME had used it to develop “synthetic voices in memory of legendary artists”. For instance, their KuGou Music group recreated the voice of Taiwanese singer Teresa Teng, who died in 1995, to create song “Unsent Letter” (song in Chinese). They also recreated the voice of Hong Kong-born singer Anita Mui, who died in 2003, for a new song “May You Be Treated Kindly By This World” (song in Chinese).

Do you know of any other noteworthy virtual duets or artists using voice cloning or AI? Please share!

---

References

1. Credit for the 4 Cs (consent, control, credit, compensation) phrasing goes to the Algorithmic Justice League (led by

Dr. Joy Buolamwini

).

Credit for the original 3 Cs (consent, credit, and compensation) belongs to CIPRI (Cultural Intellectual Property Rights Initiative) for their “3Cs' Rule: Consent. Credit. Compensation©.”