NOTE: This answer between the bars was written prior to edits to the question that clarified and changed some of the considerations and assumptions I made for this response.
This response originally assumed that the inequality was creating a predetermined range as inferred from the Question Title, however, the inequality is a relationship between two inputs, not a range.
Yes, the competitor's invention likely infringes upon the first claim. (However, keep in mind that during litigation, Terms of the first claim will be defined and Expert Witnesses will be called to opine on what everything means. It is somewhat unlikely that the claims as interpreted prior to litigation will mean the same as the same claims during the trial.)
The first claim includes the condition that the machine executes Y when A is equal to B.
For this, we can look towards an Obviousness determination for overlapping ranges (which, while "obviousness" is different from an infringement analysis, whether portions of overlapping ranges teach other ranges is illustrative of how various ranges may be interpreted for infringement).
The Courts have held:
In the case where the claimed ranges "overlap or lie inside ranges disclosed by the prior art" a prima facie case of obviousness exists. In re Wertheim, 541 F.2d 257, 191 USPQ 90 (CCPA 1976).
In re Woodruff, 919 F.2d 1575, 16 USPQ2d 1934 (Fed. Cir. 1990):
The prior art taught carbon monoxide concentrations of "about 1-5%" while the claim was limited to "more than 5%." The court held that "about 1-5%" allowed for concentrations slightly above 5% thus the ranges overlapped.
In re Geisler, 116 F.3d 1465, 1469-71, 43 USPQ2d 1362, 1365-66 (Fed. Cir. 1997):
Claim reciting thickness of a protective layer as falling within a range of "50 to 100 Angstroms" considered prima facie obvious in view of prior art reference teaching that "for suitable protection, the thickness of the protective layer should be not less than about 10 nm [i.e., 100 Angstroms]." The court stated that "by stating that ‘suitable protection’ is provided if the protective layer is ‘about’ 100 Angstroms thick, [the prior art reference] directly teaches the use of a thickness within [applicant’s] claimed range."
In response to the Second Question in the most recent edit:
Let's suppose that the judge rule in favor of the competitor, deciding that it doesn't infringe.
At this point, would the original inventor be able to file a continued application to add a claim that covers the competitor's machine? It's clear that the invention that allows the inventor's original claim would also allow a claim that would cover the competitor's workaround?
That really depends:
If there is no currently pending family in the original patent family, then "No."
If the specification does not include the specific limitation of the "only" part, then "Probably not" since the "only" language would likely be viewed as new matter.
If during prosecution, the Applicant made an argument that the 'less than' part is essential to the invention, then "Most Probably Not". That is prosecutorial estoppel.
If there is currently pending family, and the specification includes the restriction that machine that executes Y only when A is equal to B, and while getting the prior application, the applicant did not create prosecutorial estoppel in choosing that specific condition, then the patentee may be able to get another patent that has that claim.
BUT- it would almost certainly require a Terminal Disclaimer for Obviousness-type Double Patenting. HOWEVER, there are currently some cases going up that have this situation and it really isn't looked too fondly on. I think there was a case from earlier this week or maybe last week that addressed this issue. This situation also may bring up some Inequitable Conduct issues if the litigation is not properly disclosed to the USPTO.
