I won’t pretend to have been in on what was going on, but (I WAS aware of one thing...)

Not that I was in the know or anything/privy to all the ins and outs, (but one thing I DID know is that...)

It’s not like I knew what was going on/had all the details, but (I DID know one thing...)