Requesting prior art on Google machine learning patent (US 2014/0180986)

Question

Google is attempting to patent well known concepts in the field of machine learning.

Can anyone provide prior art for WO 2014/105866 A1 (US 2014/0180986), "System and method for addressing overfitting in a neural network"?

Answer 1

Google is not the company that filed the application. If you examine the Legal Events, the application for this patent was originally filed by a Canadian startup company affiliated with the University of Toronto:

ASSIGNMENT OF ASSIGNORS INTEREST
ASSIGNOR:THE GOVERNING COUNCIL OF THE UNIVERSITY OF TORONTO
Effective date: 20130311
Owner name: DNNRESEARCH INC., CANADA

DNNresearch, Inc. was incorporated on November 21, 2012 and was acquired by Google on March 12, 2013. The provisional patent application was filed on December 24, 2012. Note the Christmas Eve filing date - if they had waited until after the holidays, they would have lost the ability to file a European Patent due to an early public disclosure, described below.

One additional fact to note:

Google previously awarded Hinton's research group $600,000, the University of Toronto added.

If you are looking for prior art on this application (it has not yet granted), a good place to start would be to read through the inventor's academic publications, as well as the publications of conference proceedings that members of his startup attended prior to March 11, 2013. You might specifically focus on publications from 2011 to late 2012.

Any prior art would only need to cover the three independent claims in the application:

Claim 2. A computer-implemented method comprising:

• obtaining a plurality of training cases;
• and training a neural network having a plurality of layers on the plurality of training cases,
  • each of the layers including one or more feature detectors,
  • and each of the feature detectors having a corresponding set of weights,
  • wherein training the neural network on the plurality of training cases comprises, for each of the training cases respectively:
    • determining one or more feature detectors to disable during processing of the training case,
    • disabling the one or more feature detectors in accordance with the determining,
    • and processing the training case using the neural network with the one or more feature detectors disabled to generate a predicted output for the training case.

Claim 11. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

... repeated from Claim 2

Claim 20. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

... repeated from Claim 2

The independent claims are identical (Claim 2: Method, Claim 11: System, Claim 20: Hardware). To knock out all independent claims, you simply need to focus on the clauses of Claim 2:

obtaining a plurality of training cases;
and training a neural network having a plurality of layers on the plurality of training cases,
    each of the layers including one or more feature detectors,
    and each of the feature detectors having a corresponding set of weights,
    wherein training the neural network on the plurality of training cases comprises, for each of the training cases respectively:
        determining one or more feature detectors to disable during processing of the training case,
        disabling the one or more feature detectors in accordance with the determining,
        and processing the training case using the neural network with the one or more feature detectors disabled to generate a predicted output for the training case.

The Priority Date for this patent application is December 24, 2012, so any material that is intended to serve as Prior Art must be publicly available prior to that date. I have found no fewer than 5 individual articles that should each be able to individually serve as Prior Art for all independent claims in the method, the earliest being Mozer and Smolensky (1989). These independent claims are the same as their method, which they named "skeletonization". I believe this one reference can be used as proof that Hinton was not first to invent.

---

Early Disclosure

Hinton did disclose Dropout prior to the Priority Date of the application:

Improving neural networks by preventing co-adaptation of feature detectors

Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov

The arXiV record was created on July 3, 2012 02:35:15 EDT, and the earliest document timestamp I have is 7/2/2012 11:25:24 PM.

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.

And a second version of the publication has been available since July 7, 2012 on Hinton's publications page.

Neither document mentions that the method is proprietary or subject to patent.

However, since Hinton is the inventor, I believe he still had the ability to within one year of first disclosure, which would move the literature search on his work back to December 23, 2011. Of particular note is that the European rules have a limit of 6 months for non-judicial disclosures. The time interval from the July 2, 2012 non-judicial disclosure to the December 24, 2012 Priority Date is 176 days. If anyone can find an earlier disclosure (even 5 days earlier), then the European patent application may be rejected on EPC Article 55 (Non-prejudicial disclosures).

---

The Prior Art search for those not listed as inventors on the application would still be valid to December 23, 2012. That might include the literature below:

Prior Art:

---

Submitted to USPTO as Prior Art on July 20, 2015

• M. C. Mozer and P. Smolensky. 1989. Skeletonization: A technique for trimming the fat from a network via relevance assessment in Advances in Neural Information Processing (1), D.S. Touretzky, Ed. (Denver 1988), pp. 107-115.

This paper proposes a means of using the knowledge in a network to determine the functionality or relevance of individual units, both for the purpose of understanding the network's behavior and improving its performance. The basic idea is to iteratively train the network to a certain performance criterion, compute a measure of relevance that identifies which input or hidden units are most critical to performance, and automatically trim the least relevant units. This skeletonization technique can be used to simplify networks by eliminating units that convey redundant information; to improve learning performance by first learning with spare hidden units and then trimming the unnecessary ones away, thereby constraining generalization; and to understand the behavior of networks in terms of minimal "rules."

---

Submitted to USPTO as Prior Art on July 20, 2015

• A. Stahlberger and M. Riedmiller, 1997. Fast Network Pruning and Feature Extraction by using the Unit-OBS Algorithm

Unit-OBS has two main advantages: First it is a fast algorithm to prune big nets, because whole units are removed in every step instead of slow pruning weight by weight. On the other side it can be used to do a feature extraction on the input data by removing unimportant input units. This is helpful for the understanding of unknown problems.

---

Submitted to USPTO as Prior Art on July 20, 2015

• Remove or disable some connection in Neural Netwrok October 9, 2009

I think I can initiate half of the weights of in the network to be zero, and if I can use some training methods that will tune the weight according to the initial weight (just like in every epoch, the new weights are the sum of the current weights and a factor times the current weights), then the weight that equals to 0 initially will keep being 0 finally, that may more or less like those connections are disabled.

I just want to remove some connections from a input unit to show hidden unit.

That is why i want to remove half of them so that it will be 'fair' to all this three models.

intialize and reinitialize the I1->H2 and I2->H1 weights to zero after every epoch.

---

The following have not yet been submitted to USPTO as Prior Art:

• A Basic Introduction to Feedforward Backpropagation Neural Networks (2009)

Weight values associated with individual nodes are also known as biases. Weight values are determined by the iterative flow of training data through the network (i.e., weight values are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics).

---

• Pruning Algorithms-A Survey 1993

Mozer and Smolensky [19] describe a method which estimates which units are least important and deletes them during training.

---

• CS181 Lecture 12 - Neural Network Model Selection March 1, 2009

There's a tradeoff in the number of hidden units used in a neural network. With too few hidden units, the hypothesis space might not be rich enough to represent the actual function being learned, which means that the network might underfit the data. On the other hand, more hidden units means more weights to learn, and makes it more likely that the network will overfit the data. This leads to the question: how many hidden units should we use? In addition, for a given number of hidden units, we could choose to remove some of the edges between the different layers - how many edges should we have in our network?

---

Answer 2

These guys proposed droupout in 1992. The differences are that Clay and Séquin dropped only units not inputs, and that they did not normalize the learned weights after completing training.

"Fault Tolerance Training Improves Generalization and Robustness" Reed D. Clay and Carlo H. Séquin [Proceedings 1992] IJCNN International Joint Conference on Neural Networks Date of Conference: 7-11 June 1992 Conference Location: Baltimore, MD, USA Date Added to IEEE Xplore: 06 August 2002

Abstract here: https://ieeexplore.ieee.org/document/287094

This looks like strong evidence. The authors disabled units, and showed that an application of the method was improving generalization performance of the neural network.

Answer 3

I would suggest that any good book on Model Theory, such as those by Kreisel, Krivine, and W. Hodges, from the 1970s through 1990s, would show that the size of the proof space is in general inversely related to the size of the model space. This does not directly address neural nets. However the positing of alternatives and then pruning them is an old and public academic model theoretic technique.