There absolutely is some risk of connectivity issues and downtime with only a single IP being used as there is no possibility of failover in the case of server downtime. You probably will be fine, but that depends on your need to maintain 100% uptime.
I would recommend moving your domain, or at least DNS nameservers, to a provider that isn’t imposing such extreme limitations on DNS records - that limit is not normal, any other major provider should allow you to add as many records as you need.
Let me know if you have any more questions or if I can clarify anything!