Prior art for using a camera in self-driving cars

Question

Toyota has filed a patent (#20120226392) for a "Driving support apparatus and driving support method". Its main claim:

A driving support apparatus that performs a driving support for causing a vehicle to travel in a lane, the driving support apparatus comprising:

• an image taking section for taking an image of a road surface which is travelled upon and is in front of the vehicle;
• an image processing section for detecting for a lane dividing line on the road surface which is travelled upon, by using the image taken by the image taking section, and setting a virtual lane dividing line in a section in which no lane dividing line is detected;
• a support section for performing the driving support by controlling the vehicle so as to prevent the vehicle from deviating from the lane dividing line or the virtual lane dividing line; and
• a warning section for issuing a warning when the vehicle is likely to deviate from the lane dividing line or the virtual lane dividing line; wherein
• in a case where the vehicle satisfies a predetermined condition,
• the warning section is actuated when the vehicle is likely to deviate from the lane dividing line, and
• the warning section stops operating when the vehicle is likely to deviate from the virtual lane dividing line.

The gist is that we are looking at a patent for a device that contains a camera and a processor which it uses to drive a car.

We've been experimenting with autonomous cars for years. Surely Toyota is not the first to use a camera (er, image taking section) to give its car a view of the road. Does anyone know of any specific prior art that exists?

Answer 1

There seem to be numerous examples of this technology at play as researchers have been working on autonomous vehicles for quite some time.

One of the best examples I found was from the 2007 DARPA Grand Urban Challenge in which a team from Tartan Racing successfully completed an urban driving challenge using a Chevy Tahoe equipped with, among other tech, cameras used for observing road markings.

That same challenge also featured a team from Stanford Racing led by Sebastian Thrun who went on to lead Google's autonomous vehicle program and likely incorporates much of the same type of technology.

Answer 2

Not every element of a claim is expected to be novel. This patent does not claim that the inventor is the first to put a camera on an autonomous vehicle. Look at all of the claim elements, taken together, and ask yourself whether you know of pre-existing art that teaches or discloses every element of the claim.

Next, look at the dependent claims that progressively narrow the focus of the independent claim. Do the same analysis with the conditions (claim elements) of the dependent claim added to those of the independent claim from which it depends. Only when you reach a "leaf claim" (i.e. one that has no dependents) have you done enough analysis to be sure that you have considered all of the possible sources of novelty in the invention.

This is not to say that all dependent claims add novelty. Sometimes a dependent claim makes explicit something that was implicit in a claim from which it depends, thereby making it clear that the claim from which it depends takes in more than just the explicitly identified scope in the dependent claim.

Answer 3

I can not find exactly which lecture it was, however Professor Andrew Ng of Stanford shows an example of a project of this very nature in his machine learning course. If I remember correctly, the machine learning algorithm discussed learned to identify the center lane, and the right curb using a camera. The system then actuated the wheel to stay in the middle. I believe there was an alarm which warned that the AI was going to give up and relinquish control to the driver.

I also believe this project significantly predates any of the commercial attempts noted here. If I remember correctly the learning algorithm was written in Fortran :)

The full course is available on ITunes Univerity:

https://itunes.apple.com/us/course/machine-learning/id495053006

Professor Ng's Page:

http://online.stanford.edu/instructors/andrew-ng

Answer 4

The gist is that we are looking at a patent for a device that contains a camera and a processor which it uses to drive a car.

We've been experimenting with autonomous cars for years. Surely Toyota is not the first to use a camera (er, image taking section) to give its car a view of the road. Does anyone know of any specific prior art that exists?

Are you looking for prior art that might anticipate or make obvious "a device that contains a camera and a processor which it uses to drive a car" or prior art that might anticipate or make obvious the actual claim you posted? They're two very different things.

In other words, if you care about this patent application in particular then, as user96 pointed out, you need to look for prior art that discloses all the elements of the claim (alone or in combination), not that has the same general gist.

Look at the link to the Tartan Racing vehicle that Alex Miller posted.
It discloses:

Boss is a Chevy Tahoe with over 500,000 lines of code to autonomously navigate in town and in traffic. Boss uses perception, planning and behavioral software to reason about traffic and take appropriate actions while proceeding safely to a destination.

Boss is equipped with more than a dozen lasers, cameras and radars to view the world. High-level route planning determines the best path through a road network. Motion planning requires consideration of the static and dynamic obstacles detected by perception, as well as lane and road boundary information, parking lot boundaries, stop lines, speed limits, and similar requirements. Boss handles surprises such as other vehicles running a stop sign or making sudden stops or turns. Defensive driving skills allow Boss to avoid crashes.

Tartan Racing technology enables Boss to:

Follow rules of the road
Detect and track other vehicles at long ranges
Find a spot and park in a parking lot
Obey intersection precedence rules
Follow vehicles at a safe distance
React to dynamic conditions like blocked roads or broken-down vehicles

Active safety can play a huge role in countering accidents from cell phone usage, drowsiness and drinking. Human payoffs for Urban Challenge technology include improved safety, and enhanced driving experience.

We can safely assume the claim's "an image taking section" corresponds to a camera so that limitation is met, but what about the next one:

an image processing section for detecting [] a lane dividing line on the road surface which is traveled upon, by using the image taken by the image taking section, and setting a virtual lane dividing line in a section in which no lane dividing line is detected;

Even though we can safely assume from the disclosure on the Tartan site that "Boss" has image processing capabilities that would correspond to "an image processing section" without any added qualifications, there's no disclosure that relates to anything corresponding to the much more specific "an image processing section for detecting a lane dividing line ... and setting a virtual lane dividing line...."

From an anticipation point of view, we're already done with the Boss website (which isn't to say the Boss doesn't have this capability - it's just not disclosed on the website and thus the website does not anticipate the claim).

And of course there's the fact that this is just a claim from an application - it hasn't been allowed and may not even be a current claim (e.g. the applicant may have amended or cancelled it since publication). Assuming it's still pending, your best source of prior art is to look at what the examiner has already cited against the claim during prosecution.

Answer 5

Might this help?

From "Autonomous Cars and Society" published by Worcester Polytechnic Institute and dated May 1, 2007.

http://www.wpi.edu/Pubs/E-project/Available/E-project-043007-205701/unrestricted/IQPOVP06B1.pdf

Section 1.3.1.1.1 Lane Departure Warning System (LDWS), Page 17.

The best way for everyday driving has been determined to be the use of image processing. This detects existing lane markers to create a virtual lane in the video camera. “Here, the predominant approach by far is the use of a monochrome video camera and image processing to extract the lane and road edge markings from the image.”

The down side to this method is that the road markings are not always clearly displayed. They can be covered by snow, worn down, or difficult for the camera to see because of highly reflective road conditions. This is counteracted by using special detection algorithms that can even detect tire lanes in the snow. There are still issues where the system may become confused or the ability for the camera to transition between drastically different lighting. Another way to detect lane departure is downward looking infrared sensors on the bottom of the car. These sensors look for the change in the reflectivity of bare pavement and lane markings. However, this system can only detect the lane departure as it is occurring and the camera system could predict lane departure from sensing the road ahead. All of these systems provide a range of warnings from audible beeps to physical feedback to the driver to alert them that they are departing the lane.