Unitpricer products will always work with the standard tax settings you can set up globally for your shop globally. This is due to the fact that Unitpricer works with Shopifys custom sale feature, which does not support product based tax rules.
There is a possible workaround though: You can set up the shop tax rules as they should work for unit priced products (in your case: no tax) and then manually activate tax for all other products that need it activated.