How can the XML (or its ZIP file) for a USPTO number (e.g. 8978162) be downloaded? Can this be done programmatically (API) and/or manually?
The weekly releases (ca 6,000 entries) contain the type of information I want but not for a specific number. Thus the entry for 8978162 is found in the weekly release as I20150317/UTIL08978/US08978162-20150317/US08978162-20150317.XML.
Google is the clearing house for USPTO XML, via their Bulk Downloads site. The Grant Full Text is probably what you're looking for. The only company (that I am aware of) that currently offers full text XML on a per-patent basis via an API is Fairview Research (paid service with non-redistribution of XML source), and they are also the source of the Public Pair (legal events/status) data for Google.
As far as the reasons why there is no public API available, it is easier to paint a picture based on the historical patent services. EDS, IBM, MicroPatent and now Google have all had (at some point) free USPTO patent text available, but not in a way as to have a useful free API available. Historically, the trend is to eventually monetize, spin off, or retire the services:
EDS Shadow Patent Office -> HP -> decommissioned
IBM Patent Server -> Delphion -> Thomson Reuters (Delphion) -> decommissioned
MicroPatent -> Thomson Reuters (Aureka) -> Thomson Innovation -> Clarivate (Derwent Innovation)
Derwent World Patents (commercial) -> Thomson Reuters (Derwent) -> Clarivate (Derwent World Patents Index)
The fact is, it costs a lot of money to maintain these services (even the USPTO has budget-related maintenance issues), and it is no secret that Google has non-altruistic motives for the USPTO partnership (Jun 2010, May 2011, Aug 2012, Sep 2012, Nov 2013, Oct 2014, Feb 2015, May 2015, Jun 2015). There used to be a Google Patents "Developer API" but it was limited to integration of Google Patent content into a web site and was deprecated in 2011 (exactly 12 months after they obtained the USPTO hosting deal) - I'm not sure, but that may be what the Ask Patents sidebar is based on; you may be able to extract some amount of metadata from that service. This paragraph may seem overly cynical, but this is the story I have seen unfold over the last few years, and this is why I am not holding my breath for a developer-friendly API from Google. My interpretation of the above trend is that they have no interest in making it easy for third-party developers, but you're welcome to draw your own conclusions.
That said, before the Google-USPTO arrangement, the XML had to be purchased and distributed via tape/CD/ftp (again, in bulk form) from the USPTO, but it was cost-prohibitive (about $12,000/year IIRC), so it was fairly common practice for developers (including those at Google) to resort to screen-scraping their broken HTML and normalizing to XML or other structured formats (non-trivial) - this often crippled the USPTO servers. With the bulk downloads freely available, the situation is much better than it was a few years ago (as least there is a free "nearly Open" path to the XML), however you will still need to download the huge ZIP files, extract/parse, etc. It is possible to write a wrapper around the site to perform this; of course with the corresponding penalties of bandwidth and drive space it does not lend itself to an interactive experience for an end-user.
I believe you already have the optimal free solution worked out, and this is the approach I have taken with several of my own projects. However, under the right circumstances you might also consider the API from Fairview if you need current legal events and one-off XML retrievals.
I should also note the following resources, which provide the schemas and descriptions of the XML available from the USPTO:
Prior to 1976, only the patent metadata is available in a structured form. Everything before that relies on uncorrected OCR scans, and coverage is very spotty. It is a dark area for patent text. Also, the XML for applications is only available from 2001 forward.
You might have some luck with the European Patent Office Open Patent Services, which does provide download access (maximum volume of 2.5 GB per week free-of-charge). However, I've found their APIs to be somewhat unreliable and slow, and they discontinued several of the services I was using in 2014. I ditched OPS out of frustration (would have required a complete re-write of my code), but maybe its time to reconsider it. YMMV.
Finally, I want to note a common misconception in patent identifiers as they relate to individual documents. The number you mention ('8978162') is indeed the patent number, but a full unique identifier for the document also comprises the patent authority ('US' in this case) and the kind code (the document type), ('B2' in this case, which indicates a Patent Grant). The codes have changed several times in the past (older grants have a kind code of 'A' and new applications have a kind code of 'A1'). Therefore, to uniquely identify a document, the full identifier for the document should take a form like 'US-8978162-B2'. This will save you from refactoring, re-indexing and link construction headaches the first time you need to work with a different type of publication (e.g., application, re-issue, correction, design, plant) or patent authority (e.g., EP, CA, AU). Also refer to this write-up on patent families and an example of how the Kind Code is used to differentiate related documents.
Related resources (I will add to this as I find them):
Update: Thomson Reuters has has consolidated three of the five public APIs into a single service and spun them off into Clarivate Analytics.
You can also try USPTO Bulk Data. Enter your patent number in corresponding refine field. After filtering patents press Request download (XML).
