My previous statement may not be clear. So I rewrite my question: I was granted my patent. I do have cited many prior art/patent in my patent application, one of the cited prior patent owner notified me that my patent infringes his one. Since the USPTO patent Examiers were supposedly reviewing all the prior patents that I cited in my application, and they granted me my patent, will I be safe if this prior patent owner sue me in the court?
A granted patent provides a "negative" priviledge, I mean, you can block others from manufacturing or commercializing the invention, but it does not ensure you can put your own product / service in the market.
There may be times when the product or service protected by your patent requires licensing other patents, because some specific parts of the manufacturing process or characteristics of the product have been previsuly protected by others. In those cases, third parties can block you unless you get such licence. Usually, this kind of conflict is solved by sharing licences. Anyway, the conflict appears when the product is in the market, not because of the patent itself.
However, I think you mean that the other company wants to invalidate your patent. As the examination process is not perfect, sometimes important prior art is missed, and inventive step criterion is not fully applied. I only know the patent cancellation procedure in European countries, I suppose that it is not so far away from the EEUU one. As far as I know, most of companies fight back by trying to cancel the patent of the initial suing company, it is a so common procedure that it has its own name in Spanish and other European languages. :-)
I agree with Eric Shain and DonQuiKong, I imagine that you get that notification after you patent was published. So, if you post the patent numbers of both patents (prior art and yours) perhaps we could help you better.