I am curious is it legal to compare performance of a patented algorithm with a non-patented algorithm? I guess it would be because it is purely non-commercial and educational even if that kind of research would be conducted by a commercial company. Is there any law case that would support this thesis?
U.S. Patent law does not say infringement requires a financial gain. It says you infringe if you make or use a patented invention. It also says you can't sell, offer for sale or import which are commercial activities, but use and make do not require any explicit commercial exploitation or any diminution in the patent owner's commercial prospects.
It is not the comparison that is the problem, it is the performance of a patented set of steps, presumably needed to have something to compare to. If you can do a comparison on paper without executing the patented method then no infringement has occurred.
In the U.S. a research exemption is very narrow other than spelled out in the Hatch–Waxman Act which only applies to certain drug development situations.
An article published by the National Library of Medicine says -
The judicially created exemption is a narrow exemption, reliance upon which is not likely to be fruitful."
and in a drug related case (Roche v. Bolar, ) that did not fall under the safe harbor of Hatch-Waxman -
The court ruled that whenever experiments were performed for business purposes, those experiments were not exempt. “[T]ests, demonstrations, and experiments which are in keeping with the legitimate business of the alleged infringer are infringements for which experimental use is not a defense.”
What about a not-for-profit university? see the case Madey (307 F.3d 1351) about the fact that a university's reputation enhancement leads to financial gain.
Although a university may fund or allow research projects with arguably no commercial application, these activities further the legitimate business objectives of the university. Research projects may increase the stature of universities and help attract grants, students, and faculty. “[S]o long as the act is in furtherance of the alleged infringer’s legitimate business and is not solely for amusement, to satisfy idle curiosity, or for strictly philosophical inquiry, the act does not qualify for the very narrow and strictly limited experimental use defense.”
I think the bottom line in the U.S. is that executing the steps of an algorithm that is patented unless it is for purely "philosophical inquiry" will be infringement without permission of the patent owner.
Do it in the EU
Also published by National Library of Medicine this article covers the laws in a variety of EU countries, many of which have explicit laws allowing experimental use.
effects of a patent shall not extend to “acts done for experimental purposes relating to the subject matter of the patented invention.”
An act that would constitute an infringement of a patent for an invention shall not do so if it is done for experimental purposes relating to the subject matter of the invention (§60(5)(b) UKPA).
Do it in Canada
New section 55.3 of the Patent Act, which came into force on December 13, 2018, provides:
55.3 (1) An act committed for the purpose of experimentation relating to the subject-matter of a patent is not an infringement of the patent.
Or better yet, do it in a country where the algorithm is not patented.
Patents are territorial. No commercial entity finds it cost effective to file for patents everywhere.
Let me preface this by pointing out that I am not a lawyer. So this isn't legal advice. There is a Research Exemption, but I've been advised that this mainly covers generic drug development. That said, I do have a number of patents on algorithms related to real time PCR analysis and have published papers comparing their performance against other algorithms even after retiring from my company who owns the patents. My old company seems to have no problem with this, but I can't guarantee this is true for other patent owners.
In my old companies case, the algorithm is part of an entire instrument system and isn't sold as a software product. A company selling their algorithm as commercial software might not be so understanding so I do believe there is some risk of infringement. After all, educational institutions do purchase products so reproducing the algorithm might represent a lost sale. You could, perhaps, reach out to the patent holder and obtain permission to utilize the algorithm for your non-commercial purposes.