To remove the "Try the new Outlook" button, you just need to create a new Registry entry called HideNewOutlookToggle.
Right-click or press and hold on the taskbar. Click/tap on Show Windows Ink Workspace button to toggle it to show (checked) or hide (unchecked). That’s it on the three ways to hide or show ...