There won't be any issues on that. Sitemaps are for bots to crawl, for search engine indexing. Those errors, I believe, are non-existent pages. It is well explained in Google webmaster tool that those errors won't affect your google status. But if you want to improve user experience, you need to get rid of those pages. Personally, I don't want visitors looking through those non-existent pages either. What I usually do is that I use the REMOVE URL tool on google WMT and submit those error pages.
