What are sets made of?
That numbers are ubiquitous in Set Theory will be apparent from this example taken from Felix Hausdorff "Set Theory", 1924/2013:
"Let A = {5, 6, 7, . ..} be the set of all the natural numbers from five on, В = {1, 2, 3, ...} the set of all the natural numbers, and C={1, 2, 3, 4, 5,6} the set of the first six natural numbers. 
Then we have В - A = {1, 2, 3, 4}, С - (В - A) = {5, 6}." (p.15)
First, it would seem that we could easily replace those numbers with variables that could designate any set of objects. Let us do it.

i)   A={a,b,c,...},
ii)  B={d,e,f,..,}, 
iii) C={a,b,c,d,e,f}
iv)  B-A={a,b,c}
v)   C-(B-A)={e,f}.

Still, those elements need to have at least those two properties:
1) They must be ordered, otherwise we would not know how to subtract one set from the other (unless the sets are finite and easily overseen).
2) each element is distinct (and distinguishable) from the other.

If those elements were tennis balls, we would need a distinct color for each ball. Also, we would need a rule that tells us which color precedes or follows each other color, and that for all colors. That is the only way we can be allowed to use open expressions like (i) and (ii).
In other words, we may be able to work out this kind of examples when dealing with small finite sets, but it would soon become practically impossible to do that without numbers.
Those color schemes are therefore nothing else but a, very impractical, numeral system. As is a series like

0,{0},{{0}},{{{0}}},...

Mathematicians may prefer to think that Set Theory is the foundation of Mathematics because it explains numbers at a more primitive level, a closer analysis shows that its concepts are at least as sophisticated as the rules of arithmetics in particular, and Mathematics in general. And that it would be very difficult to distinguish between them.