I decided to drive over to Waltham tonight to listen to Brent Ozar talk out of his ass about virtualization. The place was packed and @BrentO put on a good show, as expected. He gave us the four hour slide deck but condensed it down to about 60 minutes.
I chatted with Brent a bit before the talk about some of the issues I had come across with virtualization but it was during the talk that I was struck by a sense of déjà vu. Brent was talking about various ways that certain configurations could affect a host which will then affect all the guests. In my mind I stopped to ponder where I had come across that situation before.
It was with a SAN.
See, when you build yourself a SAN, the idea is to have a crapload (that’s the technical term) of disks carved out into logical units (LUNs). Your servers end up getting assigned to the LUNs which are presented to the server as a disk. This makes it easy for your SAN admin to expand disks quickly when more storage is needed. It also makes it easy for your SAN admin to choke every last bit of performance from your database server, especially if they treat it like it is a file server. But that’s a post for a different day. The point here is this: your stuff is shared.
So, what does that mean? It means that if you have a performance issue on one server, which can cause issues with your SAN, then you could see the effects manifested on other servers. You could roll into work one morning and see that there were problems on 20 servers and not think it was related in any way. Worse yet, you could see a problem on just one server and the culprit was a different server sharing the same spindles as yours.
But with virtualization this problem goes away, right? WRONG! In fact, you not only still have the SAN issue but now you have a possible issue with guests and hosts. And we can throw vMotion into the discussion just because we haven’t had enough things to possibly go wrong. There are so many possible reasons for you to see a performance degradation on your guest that it will really make the forensics challenging.
For those folks that like checklists, you should add in items for checking to see if servers are sharing the same SAN storage, if guests are tied to the same hosts, and if your guest was migrated during the night. It will make your troubleshooting easier. When you go virtual your servers are more co-dependent on each other than ever before. Awareness of this fact will be crucial in how quickly you can isolate problems.
After the talk was done I had Brent call my cell phone to verify that his ringtone would come up (“Bad Romance”) and it did.
I heart technology.
