A custom tracking domain is a domain or subdomain of your own that a sending tool uses for tracking: the address behind tracked links and the open-tracking pixel. Without one, every customer of the tool shares the same tracking domain.
Why it matters
Filters judge the links inside an email. A shared tracking domain carries the reputation of everyone using it, including the worst senders. Your own tracking domain carries only yours.
How it's set up
You add a CNAME record, for example track.yourdomain.com pointing to the tool's tracking server, then enter the subdomain in the tool. Use a subdomain of the domain you send from so the links match the sender.
Do you need tracking at all?
For cold email, often not. Open tracking is unreliable since Apple Mail's privacy changes, and click tracking rewrites your links. Many senders switch both off and judge campaigns by replies. In mailzy both are off by default; when you turn them on, use a tracking domain of your own.