Patent litigation is often described as "ruinously expensive" to pursue, thus there generally must be some financial incentive.
My take on the advice given by your legal experts is that because you are a non-profit, there is no profit in trying to sue you for damages, or even spend the money to try to block you.
That said, there are many variables.
For instance, what if your device cuts into the bottom-line of the patent holders by costing them sales of similar products? In this case, there is a financial incentive to bar you from being able to make your products available.
If you're going after a market they don't care about, say poor people who can't afford their products in the first place, and for whom no 3rd parties will subsidize the products, you might not be cutting into their bottom line...
The prior art may be shaky if it is overly broad. Certain patents issued prior to the Alice (2014) and Mayo (2012) rulings are particularly vulnerable, and could be overturned.
The problem regarding tyranny goes back to the high cost of patent litigation, which is generally hundreds of thousands of dollars on the low end, and which can easily rise into the millions. Thus a "deep pocket player" can sue at will, regardless of the merit of their claims, and if the target cannot afford the legal fees required to defend themselves, the plaintiff will prevail by default.
In general, patent trolls are in the business of shaking down big companies, who will settle rather than defend b/c it's generally cheaper to do so. (This is the patent troll strategy.)
By contrast, if a giant company infringes on an individual's IP, there are plenty of attorneys who will take on the case for the individual on a contingency basis, because the target (the giant company) has deep pockets and represents a "prize" worth going after.
Another critical question to ask is do you or other individual members of the non-profit have personal liability? This will likely depend on the type of corporation and its structure.
I also want to give the disclaimer that I am not an attorney, and this answer should not be taken to constitute advice, but merely a perspective on the issue.