This should be done at the DNS level, through your domain name provider.
I don’t think this was a DNS issue. I just deleted all the domains from the Hosting tab in my Webflow project, then added the new default domain first. This made the domain under the project name update to the new domain. Then I added the secondary domain.
Looks like Webflow always uses the first domain added to the project as the one displayed under the project name. Deleting all the domains and adding your default one first will fix the problem.
I expected the domain under the project name to update automatically if I changed the default domain to a new one. That’s why I posted this as a bug. Seems unintuitive that I have to delete all the domains and re-add them just to update the domain listed under the project.