Anyone performed data scraping on USPTO? someone has a tool/code that can be used to extract data from USPTO and have it as SQL DB?
Here is a python project that scrapes the USPTO raw data files and parses them into CSV files, MySQL, and PostgreSQL databases.
IP Watchdog recently ran a story about recent developments in obtaining bulk file wrapper data from the USPTO. It also describes the efforts of various companies in "scraping" data from USPTO databases.
Some file wrapper data is available for bulk download from Patent Examination Data System. This contains complete file wrapper data, for those applications open to public inspection, from 1981 to the present (updated daily). The data prior to 1981 are incomplete. The downloads are in JSON and XML file formats, which can be imported into a database.
The USPTO also provides some large data sets as "Research Data Sets".
in terms of data quality and reliability, this seems the best: https://www.uspto.gov/patents/search, but without APIs.
The APIs here https://developer.uspto.gov/ibd-api/#!/bulkdata/searchPatentGrantData seem not working: i only tried grants and applications, these direct calls.
Data scraping is much harder than you think. While all the file wrappers are available in PDF, before electronic filing became common the originals were on paper, and the PDF is just an image file.
Sometimes the paper has handwriting on it, although this is rare.
The file wrappers have invaluable information on what prior art was cited against the application, and how the applicant amended the claims, and unfortunately there's no good way, AFAIK, to get all that automatically.
You think "Oh, I'll just OCR it" ? The OCR quality is really bad!