Data Scraping on USPTO

Question

Anyone performed data scraping on USPTO? someone has a tool/code that can be used to extract data from USPTO and have it as SQL DB?

Answer 1

Here is a python project that scrapes the USPTO raw data files and parses them into CSV files, MySQL, and PostgreSQL databases.

https://github.com/rippledj/uspto

Answer 2

IP Watchdog recently ran a story about recent developments in obtaining bulk file wrapper data from the USPTO. It also describes the efforts of various companies in "scraping" data from USPTO databases.

Some file wrapper data is available for bulk download from Patent Examination Data System. This contains complete file wrapper data, for those applications open to public inspection, from 1981 to the present (updated daily). The data prior to 1981 are incomplete. The downloads are in JSON and XML file formats, which can be imported into a database.

The USPTO also provides some large data sets as "Research Data Sets".

Answer 3

in terms of data quality and reliability, this seems the best: https://www.uspto.gov/patents/search, but without APIs.

The APIs here https://developer.uspto.gov/ibd-api/#!/bulkdata/searchPatentGrantData seem not working: i only tried grants and applications, these direct calls.

Answer 4

Data scraping is much harder than you think. While all the file wrappers are available in PDF, before electronic filing became common the originals were on paper, and the PDF is just an image file.

Sometimes the paper has handwriting on it, although this is rare.

The file wrappers have invaluable information on what prior art was cited against the application, and how the applicant amended the claims, and unfortunately there's no good way, AFAIK, to get all that automatically.

You think "Oh, I'll just OCR it" ? The OCR quality is really bad!

Answer 5

You should take a look at patent_client! It's a python module that searches the live USPTO and EPO databases using a Django-style API. The results from any query can then be cast into pandas DataFrames or Series with a simple .to_pandas() call. Once in Pandas, you can use its SQL ingestion functions to load it wherever you need it!

from patent_client import USApplication, Inpadoc, Patent, PublishedApplication

# USPTO databases
USApplication.objects.filter("filter criteria here")
Patent.objects.filter("filter criteria here"
PublishedApplication.objects.filter("filter criteria here")

# EPO databases
Inpadoc.objects.filter("filter criteria here")

A great place to start is the User Guide Introduction

PyPI | GitHub | Docs

(Full disclosure - I'm the author and maintainer of patent_client)