Computer monitoring — controlling a remote computer — Issued Patent — PRIOR ART REQUEST

Question

AN OVERBROAD PATENT ON COMPUTER MONITORING - This issued patent seeks to patent the idea of... controlling a remote computer! 10 minutes of your time can help narrow US patent applications before they become patents. Follow @askpatents on Twitter to help.

This patent seems like the natural (to me) extension of Expect to a GUI--and I'm guessing it wasn't the first time that principle was applied. It seems like it covers any sort of remote programmatic interaction with a machine too, which sure seems broad to me.

Patent: 7,870,504

Title Method for monitoring a graphical user interface on a second computer display from a first computer

Claim 1 requires each and every step below:

A method for using a first computer system to remotely monitor and interact with the operation of a second computer system through a graphical user interface of said second computer system, comprising the steps of:

1. receiving a bitmap image of said second computer system graphical user interface at said first computer system;

2. searching said bitmap image of said second computer system graphical user interface for a first graphical element contained within and comprising less than said bitmap image through an automated execution of said first computer system commands; responsive to said receiving step and results of said searching step, generating a user peripheral input device input action within said second computer system graphical user interface as interpreted by said second computer by automatically creating and passing a signal through a communications channel from said first computer system to said second computer system graphical user interface;

3. monitoring said bitmap image of said second computer system graphical user interface automatically from said first computer system for an expected second graphical element contained within and comprising less than said bitmap image within a predetermined time interval; and

4. signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected second graphical element

What is good prior art? Please see our FAQ.

Want to help? Please vote or comment on submissions below. We welcome you to post your own request for prior art on other questionable US Patent Applications.

Answer 1

GUI Capture and replay testing using bitmaps is mentioned here as problematic(1997): http://www.imbus.de/forschung/pie-gui-test/how-to-automate-testing-of-graphical-user-interfaces/

Model based GUI Testing (Apr 15, 2003): Uses bitmaps if no other option is available http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3983&rep=rep1&type=pdf

Automated Testing of SAS System GUI Applications (From SUGI22 conference in 1997 but google says 2002): Actually uses image comparison http://www2.sas.com/proceedings/sugi22/APPDEVEL/PAPER10.PDF

Code from the paper:

/* Read the test result and baseline images. */
testname=’grtest1’;
pic=’graph1’;
call send(imgtest,’_read_catalog_’,
’testlib.’||tstname||’.’||trim(pic));

call send(imgbase,’_read_catalog_’,
’baselib.’||tstname||’.’||trim(pic));

/* Compare the images using _DIFF_IMAGE_. */
call send (imgdiff,’_diff_image_’,imgbase,imgtest);

/* Check return code, and output appropriate message.*/
rc=sysrc();

if rc=0 then
    put ’Images compared identically:’ testname pic;
else
    put ’Images had differences:’ testname pic;

Testing on a remote machine is done in link 2. Testing using bitmaps and steps is done by link 3. So I think there is some prior art showing all techniques

Answer 2

It really looks like the Sikuli project.

Sikuli Script automates anything you see on the screen. It uses image recognition to identify and control GUI components. It is useful when there is no easy access to a GUI's internal or source code.

They have two academic papers that can be useful to represent Prior Art:

• Sikuli: Using GUI Screenshots for Search and Automation, UIST 2009.
• GUI Testing Using Computer Vision, CHI 2010.

Also, some academic references:

• L. S. Zettlemoyer and R. St. Amant. A visual medium for programmatic control of interactive applications. In CHI ’99, pages 199–206, New York, NY, USA, 1999. ACM.
• T. Ostrand, A. Anodide, H. Foster, and T. Goradia. A visual test development environment for GUI systems. SIGSOFT Softw. Eng. Notes, 23(2):82–92, 1998.

Answer 3

1) us5600789, FIG. 15 (1992)

Automated GUI testing in which machine 1 executes a test which monitors and controls a GUI running on machine 2. Looks pretty solid.

2) us5634002

Citations on front page look helpful.

3) US 2002/0029259 (1991)

At least the key features, if not all.

4) User Interface Softbots (Sept 2003)

"This thesis focuses on the concept of ibots, interface agents that interact with software applications through the graphical user interface, in the same way that human users do. As a part of this work, we intend to develop a system that supports the control of an application through its graphical user interface, bypassing its API."

Other literature is cited in the thesis.

5) EP0890898 "Screen remote control with periodic bitmap image comparison and transmission" (1999)

6) http://www.itaas.com/company/documents/ecs_support/6.pdf (2000)

"Content Validation. Apart from how a WebSite responds dynamically, the content should be checkable either exactly or approximately. Here are some ways that content validation could be accomplished: ... Selected Images/Fragments. The tester should have the option to rubber band sections of an image and require that the selection image match later during a subsequent rendition of it. This ought to be possible for several images or image fragments."

Answer 4

I don't agree that this is overly broad. I also don't think this is simply VNC or other remote desktop tools. Those don't do bitmap analysis/pattern recognition on a remote PC. This sounds very specific to testing, and a poor way to do it.

Leading solutions in this field typically run an agent on the test target, and access the UI API directly rather than relying on bitmap pattern recognition techniques which are far more complicated and probably less reliable.

If this were about using the host UI API to interact with the monitored system, I would agree this is like Expect for the UI, and there are many old and well established systems to do so.

Answer 5

There is an industry leading test automation product that has been doing this for at least 10 years. http://www.testplant.com/eggplant/testing-tools/

Answer 6

The closest implementation to something like this I can think of is something like AutoIT - http://www.autoitscript.com/site/autoit/

As far as I know, it doesn't use bitmaps. Using bitmaps for something like this would be stupid, and as far as I know, nobody has done the 'screen scrape and comparison' for general control.

The closest thing I can think of to implementing the patent has been possible for a long time; this use case would meet the patents definition:

1) Have a script run on system A through SSH to automatically update an asset (eg. a web page) 2) Monitor said asset from system A using eg. VNC or RDP and automatically capture data 3) On system A, perform comparative analysis of said content 4) update system B from system A using script in #1 based on determination of

4 is where the juice is, as in the patent. Screen scraping and doing something useful with the results that isn't highly specific is Hard.

I would not be surprised if this method has been used in online game bots to get around anti-botting methods, or in Captcha-cracking attempts.

Answer 7

First post! Found this place from /.

From as early as 2007: http://www.macrogoblin.com/Featured_Bots/Bot_LotRO_Fighting_Healer_Master.aspx

Note that for comparison's sake, the Tank can be considered the First Computer and the Healer can be considered Second Computer.

Healer's Healthbar Pixel #1: When health is low it will trigger the LIGHT HEAL routine. By default it does this when Green Value is below 100 at this pixel.

and:

This is a Fighting Healer that is meant to autofollow your tank. The healer plays on a second computer, and you play the tank yourself on the main computer. The Healer is controlled through your Local Area Network (LAN) by the Host computer, which is your main computer where the tank is being played.

and:

All pixels are read from your Tank's game window. So, everything is really being done on the host's side (Main Computer). Commands that you want your healer to do are sent through the Network live from routines on the host.

In other words, the First Computer scans a 1x1 bitmap (pixel) from graphical display of the Second Computer, and if it finds that the pixel is no longer green (failure condition, as the pixel fails to be green), it will activate failure routine (cast heal spell).

This type of manipulation is common with bots that operate in a multiboxing environment, especially for buddy bots like the one described here.

Answer 8

Back in the 90's Microsoft made extensive use of a tool called MSTest (they probably still use it).

among other things, it captured screen images of the program under test and analyzed them to identify controls and fields. It could make decisions based on the presence or absence of graphical values. It also simulated mouse and keyboard interactions in order to exert control over the program being tested.

MSTest itself was actually a specialized programming language (syntax similar to VB), you could write programs to perform any desired monitoring and control sequence.

as I recall, this was eventually turned into a commercial product and made publicly available. This was not a unique product there were several other similar products also available in the 90's.

The use of VNC or similar in conjunction with MSTest would be trivial.

Answer 9

This is a particular way of accomplishing the remote monitoring of the contents of a GUI interface involving the remote system analyzing the bit map image of the system being monitored. I think other systems get a much higher level representation of the screen and don't need to do pattern matching to know what is going on at the other end.

Answer 10

This patent reminded me of when people screen-scrape video games and remotely play them...

http://www.joystiq.com/2009/08/13/computer-script-plays-super-mario-world-by-itself/

Answer 11

I'm having trouble deciding whether this is too broad or actually too narrow to be worth attention.

You take the bitmap of computer 2's screen and send it to computer 1.

You search the bitmap on computer 1 for a given graphical element (without displaying the bitmap!).

The first computer then sends a UI event to the second computer and waits for graphical confirmation, flagging an error on computer 1 if said graphical confirmation is not found within a specified time interval.

Isn't this a case of taking a few well-established techniques and chaining them together? That can't possible be patentable?

Answer 12

It would seem to me that this is obvious based on simple graphics programs from the 1990s over X11.

X11 handles the machine to machine communication. The graphics software searches and handles the bitmap. X11 itself is a bitmapped GUI interface when used over a network.

http://en.wikipedia.org/wiki/X_Window_System (1984)

http://www.handmadesw.com/Products/Alchemy_Administrator.htm (pre-2000)

http://en.wikipedia.org/wiki/Xv_%28software%29 (pre-1994)

Answer 13

This (broad) patent seems to be VNC, originally released by ORL in 1998. Information about this can be found in IEEE Internet Computing Volume 2, Number 1, a pdf of which can be found http://www.qandr.org/files/vnc-ieee.pdf

Answer 14

It is hard to follow all of the claims, but I think Timbuktu invalidates a bunch of them in the 1980s. http://en.wikipedia.org/wiki/Timbuktu_%28software%29 Also, PCAnywahere from Norton, and McAfee had one as well who's name escapes me. These all came to prominence in the 90s so well within the scope.

Going through the Wikipida List of RD software by year may be able to knock out a lot more. http://en.wikipedia.org/wiki/Comparison_of_remote_desktop_software

Answer 15

This is also similar to http://www.bostonsoftwaresystems.com/Products/Boston-WorkStation, although it operates remotely rather than running on the same computer as the software that is being automated.

Answer 16

I would suggest Looking at X-windows as well. http://en.wikipedia.org/wiki/X-windows This has been around since 1984.

Answer 17

These claims are too broad. They would cover the functionality of HP's iLO, RILOE, and RIB products, that have been commercially available for more than 15 years. Other vendors also offer remote management hardware.

All such solutions collect remote system video, presents it through a webpage or application, and captures mouse/keyboard movements which are provided up to the SUT.