Edited for below comments.
MPEP 2163.07(b):
Incorporation by Reference [R-11.2013] Instead of repeating some information contained in another document, an application may attempt to incorporate the content of another document or part thereof by reference to the document in the text of the specification. The information incorporated is as much a part of the application as filed as if the text was repeated in the application, and should be treated as part of the text of the application as filed.
Note, this does not explicitly reference figures/drawings but I assume they would be included.
Every IBR I have seen has been when a continuation/CIP is filed and the original application text is used but with new claims and the IBR portion tacked on to the beginning to preserve the priority date. Perhaps the USPTO assumes that omitting the actual text/figures would be in error and they would likely ask you to amend to include the reference text.
However... One might interpret the CFR to not hold that a requirement for later amendment is not conditional on the omission being unintentional, but that "inadvertent" is mere decorative text operating under the assumption that such omission would only occur accidentally. When read in light of the other portions of MPEP 608, I think there is a strong argument for this interpretation (implying that 2163.07(b) is purely for the limited purpose of overcoming 'new matter' issues). Bad analogy, but it gets to what I mean: "If you inadvertently dial the wrong number, you must hang up and try again."
In summary, I think it would be interesting to try leaving it all out and purely relying on IBR, but I think the examiner on the case would likely ask you to amend it in.
37 C.F.R. 1.57:
(b) Subject to the conditions and requirements of this paragraph, if all or a portion of the specification or drawing(s) is inadvertently omitted from an application, but the application contains a claim under § 1.55 for priority of a prior-filed foreign application or a claim under § 1.78 for the benefit of a prior-filed provisional, nonprovisional, international application, or international design application, that was present on the filing date of the application, and the inadvertently omitted portion of the specification or drawing(s) is completely contained in the prior-filed application, the claim under § 1.55 or 1.78 shall also be considered an incorporation by reference of the prior-filed application as to the inadvertently omitted portion of the specification or drawing(s).
(1) The application must be amended to include the inadvertently omitted portion of the specification or drawing(s) within any time period set by the Office, but in no case later than the close of prosecution as defined by § 1.114(b), or abandonment of the application, whichever occurs earlier. The applicant is also required to:
(i) Supply a copy of the prior-filed application, except where the prior-filed application is an application filed under 35 U.S.C. 111;
(ii) Supply an English language translation of any prior-filed application that is in a language other than English; and
(iii) Identify where the inadvertently omitted portion of the specification or drawings can be found in the prior-filed application.