It was a simple enough question, or so I thought. One I felt should be either a simple “yes” or “no”.
“Do we block remote desktop connections here?”
Sure enough, I got back the quick and simple answer I expected, along with a question for myself.
“Nope. We don’t block any RDP sessions. Maybe you configured your server wrong?”
I was in the SolarWinds Austin office trying to connect to one of my virtual machines running inside of Microsoft Azure. The remote desktop (RDP) session had worked fine from my home for weeks, and again from the hotel the night before. But now it didn’t work.
Knowing how to diagnose an issue is a skill you acquire with experience. I thought about all the possible ways this connection could be failing and the only difference I could find was the network. But here I was, being told that nothing was blocked, despite the evidence to the contrary.
Being the good DBA that I am I double-checked my work. I looked at the current port settings for this server in the Azure Portal. These ports are randomly assigned when the VM is created. For remote sessions the private port is 3389, but the public port was set to 54630:
And I checked the port number being used in my RDP connection:
But the result stayed the same. After a minute or so I would get this message:
This appears to be a generic error message. There are no details, no links to documentation on how to troubleshoot possible connectivity errors. This error message is less than helpful. We are left to fend for ourselves at this point.
Instead of giving up I put my troubleshooting skills to work by breaking down each sentence.
“This computer can’t connect to the remote computer.”
Well, OK. But that doesn’t tell me if the issue is with me, is with the remote computer, or with something in between. We are using a cloud service (Azure) so it is always possible that communication failures may happen. I move on to look at the second sentence.
“Try connecting again.”
Definition of insanity: doing the same thing over and over and expecting a different result. Right now this error message is Groundhog Day for cloud admins. I move on to the last sentence.
“If the problem continues, contact the owner of the remote computer or your network administrator.”
Well, I’m the owner of this remote computer, and I already know I won’t be of much use to me. Unless you consider Microsoft to be the owner. And, in a way, they are the owner, but I don’t have the number for the data center handy.
But what about that network administrator? That’s a good, and as it turns out only, clue here. Could it be that there is, indeed, an issue blocking the port despite my being told that was not the case?
I went back to my colleague to ask more questions about the network. (That’s my way of politely writing “I went back to my colleague to blame the network”.) And this time I was greeted with the most brilliant of replies:
“We don’t block any ports here. Show me the error message.”
OK, maybe I call that brilliant because I’ve written before about showing me the error message. A picture is worth a thousand support tickets, so we went to my machine, I launched my RDP session, and it failed. The response at that point was this:
“Oh, why are you using that port? I doubt we are allowing non-default ports. Just use the default 3389 and see if that works.”
I was happy, confused, and frustrated all at the same time. Yeah, I was a typical user, the one with a case of PEBKAC.
“But you just said you weren’t blocking any-“
“And you said RDP wasn’t working. You never said you were using a different port. RDP works fine with the default port of 3389. So try 3389 and let’s see what happens.”
So, back to the Azure portal I went, updating the public port to be 3389, matching the private port. And then, trying RDP again, we see success:
Which then led to this exchange:
“I thought you said we didn’t block any ports!”
“What I meant was we don’t block the correct ports. Use the correct ports and you’ll be fine.”
This, dear reader, is what you call experience.
I’ve lost time before due to a firewall of one kind or another. My favorite all-time firewall issue was at TechEd in New Orleans in 2013 when the convention center was blocking port 1433. Ask Grant Fritchey (blog | @gfritchey) or me about that someday. Good times.
A few months from that trip to Austin I found myself at SQLBits, delivering a precon with Karen López (blog | @datachick). We’ve built out some VMs in Azure so that our attendees can put their hands on something because that’s what makes for a proper training experience.
Everything worked fine from the hotel the day before. Our scripts built and configured all the VMs in a matter of minutes. We could RDP to the machines without any trouble. Everything was working as expected.
Until it wasn’t.
When we got to the event the next day we were no longer able to RDP to our Azure VMs.
I was concerned I had somehow made a mistake with the port numbers. I set about double-checking them when an attendee approached me and suggested we should check the ports again. I was confused at first (probably because he was speaking British) but then I immediately understood what was being suggested: the conference center was blocking the non-default ports! Same as with Austin, if we switched to 3389, then RDP would work as expected.
So we set about manually updating each VM through the portal. And as I was updating each one it occurred to me that I should have a script for this in the future, should I find myself needing to quickly make changes to the RDP ports (any endpoints, really) on many VMs at the same time.
So, here is the script I cobbled together after SQLBits to help me for next time. You’re welcome. As always, here is the usual disclaimer:
Script disclaimer, for people who need to be told this sort of thing:
DISCLAIMER: Do not run code you find on the internet in your production environment without testing it first. Do not use this code if your vision becomes blurred. Seek medical attention if this code runs longer than four hours. On rare occasions this code has been known to cause one or more of the following: nausea, headaches, high blood pressure, popcorn cravings, and the impulse to reformat tabs into spaces. If this code causes your servers to smoke, seek shelter. Do not taunt this code.
You can also download a copy of the script here.
<##############################################
File: AlterEndpoints.ps1
Author: Thomas LaRock, https://thomaslarock.com/contact-me/
https://thomaslarock.com/2015/04/troubleshooting-azure-connectivity-ports-and-endpoints
Summary: This script will loop through all the virtual machines
in an Azure subscription. You can modify the script below
to add, modify, or remove endpoints as needed.
Date: April 28th, 2015
You may alter this code for your own purposes. You may republish
altered code as long as you give due credit.
THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT
LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR
FITNESS FOR A PARTICULAR PURPOSE.
##############################################>
<# We are going to loop through all VM's in this subscription. However, if you want to filter for a subset, perhaps by name, you could use something like: #$VMlist = Get-AzureVM | Where-Object { ($_.Name -ilike "something") } But we don't want to filter for our example, so we just grab all VMs and build an array #>
$VMlist = Get-AzureVM
<# We will now loop through each VM in the array #>
foreach ($VMServiceName in $VMlist) {
Get-AzureVM -ServiceName $VMServiceName.ServiceName –Name $VMServiceName.Name | Set-AzureEndpoint -Name "Remote Desktop" -PublicPort 3389 -LocalPort 3389 -Protocol "tcp" | Update-AzureVM
}
If you wanted to add an endpoint to all your VMs that’s easy, you just use the following syntax:
Get-AzureVM -ServiceName $VMServiceName.ServiceName –Name $VMServiceName.Name | Add-AzureEndpoint -Name "Remote Desktop" -Protocol "tcp" -PublicPort 3389 -LocalPort 3389 | Update-AzureVM
If you wanted to remove an endpoint on all your VMs that’s easy too, you just use the following syntax:
Get-AzureVM –ServiceName $VMServiceName.ServiceName –Name $VMServiceName.Name | Remove-AzureEndpoint –Name "Remote Desktop" | Update-AzureVM
I even have a version of this script that can remove all endpoints from all VMs, but I won’t post it here because I’d be concerned someone ran that unwittingly. I would rather not be the enabler for someone bringing down hundreds of servers. But Denny Cherry (blog | @mrdenny) needed it one night so I put it together for him, and I know others may want it as well. If you want a copy of the code snippet, just drop me an email and I’ll send it to you.
Lesson here is that when working remotely you need to consider things like firewalls and blocked ports and be ready to quickly troubleshoot, diagnose, and remedy Azure connectivity issues.
And always blame the network.
Port & firewall fighting at third-party locations like hotels, conference centres, airports, training facilities are such fun. It’s like skateboarding blindfolded and without a helmet.
Something like that, and alcohol is often a factor.
My goto troubleshooting methodology in instances like this is to ping the host, telnet to the host on the port desired and see if it connects and then port scan the host with nmap. I find that wastes the least amount of time because you can respond to your colleague, the help desk or whoever with “I tried the following and I got exactly this far”
Also, since a port scan will find al lthe open ports, and nmap -sV will even do a basic service fingerprint) it helps you realize if you are totally connecting to the wrong server.