Yes, you can remove CMS part from the site. But you have to understand that you will also lose dynamic template pages. Some elements, design and structure can be lost too, because of disconnecting from dynamic collections.
If you still sure you want to get rid of CMS, try to follow these steps:
1) For that, you will have to unbind all dynamic content from the site elements.
Open site structure in the Navigator panel, expand it and you will see some elements highlighted with purple color. This element contains dynamic content. Choose such element, go to settings panel and un-check option which connecting any dynamic content with this particular element. Repeat this step with all "purple highlighted" elements.
2) Also, you will have to clean up all dynamic template pages. Unbind dynamic content from the elements and remove everything from the page structure.
3) Remove dynamic collections.