If I produce an open source product with an idea, lets say; an algorithm or protocol that nobody did before, can a company register a patent with my idea and sue me?
One of the basic sections of the patent law is 35 USC 102. It describes some of the requierements for getting a patent, especially novelty. While the law is mostly short and doesn't offer a lot of explanation, there is a manual for patent examination (MPEP). The relevant section is this one.
A claimed invention may be rejected under 35 U.S.C. 102 when the invention is anticipated (or is “not novel”) over a disclosure that is available as prior art. To anticipate a claim, the disclosure must teach every element of the claim.
Almost every publication in any language impedes patents (for the same thing) all over the world. To some extent oral publications don't meet that criteria, but something which is written and in English makes it impossible for anyone to get a valid patent on the disclosed invention.
But there is one issue - examiners don't always find all the relevant prior art and sometimes, sources are not traceable back to their publication date. You could file for a patent or write a paper to have more certainty, but in this case I would advise you to just make sure you can prove at what point in time you had what part of your open source project available to the public. If really you got sued by someone who was able to patent your invention after your disclosure (note: not likely), you could still easily defend yourself with that prove.
For some more reading on novelty and publications, check this answer.
You may also find that filing the algorithm with the copyright office is useful because it makes the content a matter of public record. Copyright isn't going to protect a procedural mechanism because mechanics are the domain of patents, but as a prophylactic, it may be quite effective, and provides iron-clad evidence of the date of the public disclosure.
Submissions to the copyright office in the US cost $35, and the process can be done online in a few minutes, and does not require an attorney.
This technique alone may not help, because as DonQuiKong points out, an examiner may not find the documentation that shows the malefactor's claims to be non-novel.
Thus, you should probably also heavily publicize the algorithm, what it does, and links to documentation. (Publicization in this case may be simply a matter of posting on forums, etc., using basic SEO strategies such as relevant key-words, and utilizing high-traffic forums so that the posts show up on search engines.)