If you're having trouble, this book can help you: https://www.amazon.com/Zero-Bugs-Program-Faster-Thompson/dp/...

Here's another good book I recommend while we're at it: http://www.amazon.com/dp/0996193308

(disclaimer: I wrote it :)

  > How did you get into such issues? Were there classes where you got started, or was it all on the job?

I took the usual college classes dealing with concurrency (in my school it was in the OS class), but it took several years in the industry to really feel confident with threads (it took less time to feel confident with networking). I wrote down my knowledge (for what it's worth) in this book: http://www.amazon.com/dp/0996193308

  >  I'm not sure how you would move lock/unlock pairs to the top of a file from different functions/scopes.

That solution might not work in every case. It worked in the particular case I was referring to.

  > It's more about a codebase that's almost perfectly right, but where I'm scared to change anything because I don't have a big-picture model of the concurrency in my head..

Hmmmmm, that's an interesting question. Usually when the codebase is written, the person who wrote it had an idea in his head that, "this is how things will be locked to avoid problems." I try to figure out what that idea was.

Sometimes there is like a critical 'zone,' where a thread acquires the lock when it enters, and releases when it leaves. For example, it could acquire the lock when it enters a class method, and releases it when it leaves the class. Then the class becomes the critical zone.

Maybe learning to think of 'critical zones' is the most important skill to understanding the big picture?