Some questions about: "how could a mathematical theory ever found mathematics? What would then found Set Theory itself? Its axioms? They are all of a mathematical nature, so that would not work. We would need non-mathematical axioms to found Set Theory, before it ever could found Mathematics."  1) The term "foundations" as used by logicians often means a set of axioms for a theory such that all the propositions of the theory (which include the axioms) are logical consequences of the theory. In this sense, the foundations for set theory must be mathematical.  Do you think this meaning of foundations is illegitimate? If so, why?
2) Suppose that someone says, "Here's a non-mathematical set of axioms for Set Theory".  That would be an interesting claim.  How would the someone prove his claim or even make it plausible? Would those non-mathematical axioms, if they are a foundation, reduce Set Theory to something non-mathematical?