First is Peter Aczel's work with the Anti-foundation Axiom which allows you to create sets with "infinite regress". Instead of blanket disallowing these kinds of set like ZFC does, he enlarges ZFC by allowing any set which is a unique solution to a group of equations that may be self-referential.
I was introduced to this stuff by a book "Vicious Circles" by Barwise and Moss (http://www.amazon.com/Vicious-Circles-Center-Language-Inform...).
The important point is that the Anti-Foundation Axiom lets us model infinite, streaming structures similar to those modeled in computer programs. It also motivates the somewhat unpopular idea of bisimulation which is very necessary for creating a kind of equality in an AFA world... and also for proving the equality of streaming algorithms.
Second is Lawvere's introduction to thinking about the foundations of mathematics from a Category Theoretic perspective. Lawvere is a proponent of thinking of Set theory as simply one, somewhat interesting Category which can be generated by a more foundational theory and set of axioms. I'm not personally anywhere nearly well-knowing enough to say whether that works, but his book Conceptual Mathematics (http://fef.ogu.edu.tr/matbil/eilgaz/kategori.pdf) gives a very interesting POV on how to work from the Category Theoretical basis to answer some normal Set theoretic questions.
"... natural language abounds with directly or indirectly self-referential yet apparently harmless expressions—all of which are excluded from the Tarskian framework"
used in this article to motivate the concept of inconsistency-tolerant logic is actually much better support for non-well-founded set theories  allowing hypersets (i.e. sets which contain themselves), e.g. by assuming Aczel's anti-foundation axiom . Since these set theories are compatible with standard the set theory, extending it, rather than replacing it, they don't throw out the notion of consistency, but have no problem dealing with self-referential sets and such. "Vicious Circles"  is a great exposition of the subject of hypersets and their applications to computer science and logic, among other things and is quote accessible for such a theoretical subject.
Fresh book recommendations delivered straight to your inbox every Thursday.