ByteDance and AI-generated music
Profile on ByteDance (owner of TikTok) and their genAI music technologies (Jukedeck, Mawf, Sponge Band, and Ripple)
This post covers what’s known about ByteDance, one of the four leading Asian companies in genAI music as of June 2024. Updates will follow as more information becomes known or new product announcements are made.
This post supports Part 3 in our 8-part series on ethics of generative AI for music, announced in this INTRODUCTION post on 6 ‘P's in AI Pods1. Subscribe to be notified when new articles are published (it’s FREE!)
This article is not a substitute for legal advice. It is meant for general information only.
ByteDance (China)
History & Partnerships
ByteDance was founded in 2012 and is a privately held “unicorn”. They have raised over $9.5B in funding. They are headquartered in Beijing and incorporated in the Cayman Islands. As of mid 2024, they have approximately 150,000 employees and 15 subsidiaries, including TikTok. After acquiring Shanghai-based musical.ly for $1B in Nov. 2017, they combined it with TikTok and took TikTok global in 2018.
Since 2016, the head of the ByteDance AI Lab is Wei-Ying Ma. Before joining ByteDance, he was assistant managing director of Microsoft Research Asia for 8 years. Since 2020, he is also “Chair Professor at Tsinghua University and Chief Scientist at Institute for AI Industry Research (AIR)”.
ByteDance and Tencent Music Entertainment (TME) have been involved in litigation since 2018.
On Dec. 15, 2023, ByteDance was reported to be secretly using OpenAI’s generative AI tech to build a foundational LLM, codenamed Project Seed, to underpin competing offerings.
Other ByteDance products which may use AI include Jianyin/CapCut (video editing app; competitor to Adobe Premiere Pro), Douyin and TikTok, Lark (enterprise collaboration platform), Resso/TikTok Music, Toutiao, Xigua Video, and NuVerdes. In an April 2024 article, China Money Network opined that ByteDance was behind on AI in these apps and working to catch up.
In June 2024, ByteDance launched an image-sharing and messaging app (Instagram rival) called Whee. Unsurprisingly, given the pending US ban on TikTok in 2025, it is not yet available in the US. ByteDance has sued to stop the forced sale.
July 2019:
Acquired UK-based AI music startup Jukedeck, which was founded by
. Jukedeck specialized in creating royalty-free music for user-generated online videos.
“Ed Newton-Rex, CEO of the ethical generative AI non-profit, Fairly Trained. A veteran expert in the world of gen-AI, Newton-Rex is also the former VP Audio at Stability AI, and the founder of JukeDeck (acquired by TikTok/ByteDance in 2019).” (MBW)
April 2022:
May 2022:
Launched a music creation app in China called ‘Sponge Band’ according to Tech Planet (May 16, 2022) (ref, translated TechPlanet page).
June 2023:
Announced closed US beta for mobile app Ripple (as reported by TechCrunch).
Key Features
Jukedeck
According to this March 27, 203 MBW podcast, by the time of the 2019 acquisition by Bytedance, Jukedeck had created “one and a half million or so tracks”. Podcast host TIM INGHAM noted in the intro that “Jukedeck’s wizardry is now firmly embedded in ByteDance‘s tech stack – which, from an AI standpoint, was overseen by Newton-Rex for two years subsequent to that 2019 sale. During that period, he was Product Director of TikTok’s in-house AI Lab, gaining a unique perspective on what the world’s biggest short-form video platform plans to do with AI-generated audio.”
Sponge Band
According to tech.ifeng.com, the "Sponge Band" App has four major creative functions:
recording studio,
playing and singing optimization,
one-click remix (which, as translated, sounds like autotuning),
automatic mashup.
Sponge Band “can provide music editing services for Byte's music business "Soda Music" App and the "Douyin" App, becoming an important part of the audio ecosystem.” It’s not clear if the “Sponge Band” app is still actively being developed, at least not for iOS. The 2 latest reviews on the Apple Store app page as of April 2024 showed a want-to-be-user asking in Oct. 2022 how to get an invitation code, and a user in Aug. 2023 asking for an equalizer feature. (app page ref, translated) (reviews ref, translated) As of June 2024, the iOS app does not appear to be available in the US any more; the app page hangs at “connecting” and never loads.
Ripple
Ripple can take hummed or sung input from a user and turn it into a full instrumental song. As of June 2023, it does not yet support lyrics. Its instrumentals may be targeted for future integration into TikTok as background music.
Ripple includes a “virtual recording studio” that “lets users record, cut, trim and edit audio files with ease on their mobile device”.
Mawf
Mawf is “a machine-learning driven music-making app that analyses incoming audio signals and then “re-renders” those signals using what it says is machine learning models of musical instruments”. It is developed by the SAMI (Speech, Audio, and Music Intelligence) team at ByteDance.
In the June 2023 announcement of Ripple, Bytedance said that Mawf was a separate app from Ripple. The Mawf privacy policy page says “Mawf is an audio plugin which helps you create music within your Digital Audio Workstation.” The Mawf home page says it is a “SYNTH, CREATIVE EFFECT, MACHINE LEARNING MODEL ALL MAWFED INTO ONE”. It is compatible with many common DAWs (Ableton Live, Logic Pro, Reaper, BitWig, Cubase). They claim that “Mawf works on all kinds of inputs: singing, mono synths, midi... and yes, your cat!” As of June 2024, the site has a banner which says that beta signups have been expanded to 5000 slots.
Training Data & Technologies
According to Bloomberg Law, ByteDance has obtained more than 900 US patents since 2020 and has over 600 patents pending. Their patents, which include their AI music generation tools, are at some risk due to the pending possibility of a forced sale of TikTok in 2025 under US law. According to MBW, a new corporate entity called “Lemon Inc USA”, based in the Cayman Islands, has been the applicant on some recent patents relating to AI and music.
Jukedeck: (MBW ref) (LinkedIn ref) A patent filed in 2019 on Jukedeck’s automated music production server was granted to Bytedance in 2024. ByteDance’s research division, the Speech, Audio & Music Intelligence (SAMI) team, has been developing “cutting-edge models for music generation”, particularly in two projects called StemGen and MeLoDy. MeLoDy is a “LM-guided diffusion model … trained on an extensive dataset of 257,000 hours of music” from 6.4 million audio files. The source of that training dataset is not identified, though.
Mawf: The Mawf site says “We used machine learning (ML) to analyse recordings of professionals playing pieces of music on their instrument.” They note that “ML for audio synthesis is a new technology no one has really perfected yet.”
Ripple: The AI model for the Ripple app was “trained on music that is licensed to or owned by ByteDance”. (TechCrunch, June 30, 2024, Aisha Malik) Whether Ripple uses the MeLoDy model and / or the StemGen model is unclear.
Ownership, Usage Rights, and Pricing
Minimal information is available in English at present about ownership, usage rights, or pricing. All apps released in beta to date have been free.
What’s Next?
Our next posts in the genAI music series will cover 7 major companies outside of Asia, and many more major and minor companies. Stay tuned!
End Notes
See this “AI for Music” page for a complete set of links to all posts and company profile pages in this article series.
![[INTRO] Unfair use? The ethics of generative AI for music](https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F163f723a-f4fc-4701-84a8-3a81cad85775_4896x3264.jpeg)