Thursday, September 20, 2012

Configuring Apache as a Reverse Proxy for Lync Server

Introduction

As stated in this blog: http://blogs.technet.com/b/server-cloud/archive/2012/09/12/important-changes-to-forefront-product-roadmaps.aspx, a number of Forefront-branded solutions are discontinued from further releases.

For us, as Lync administrators, including TMG 2010 in this group is extremely painful, having in mind the ease of which Lync external web services were published. This is not the end of the road, however. There are third party software which can be sucessfuly utilized for web publishing.



Today I will install and configure Apache 2.2 in my lab as Reverse Proxy for Lync 2013.
I will be using Windows Server 2008 R2 for this exercise.The server where I will install Apache is not member of domain, because it will be exposed on internet and I want to enhance the security. OS was patched with the latest Windows updates prior to Apache installation.

I will use two virtual network interfaces:
“LAN” – 10.255.2.52/24 and “DMZ” – 192.168.1.52/24
Only DMZ interface have Default Gateway. Routing to the internal subnets via LAN interface was added manually. I will use the same procedure used when configured the Edge server.
LAN interface:

DMZ interface:

Current routing table:


***In this case, Reverse Proxy's LAN interface is on the same subnet where Lync servers are. I can afford it, because this is lab. In production, however, I would consider placing the LAN interface on "internal DMZ subnet" in order to have a phisical separation.

To add static route to internal subnet, ROUTE command was used:
 ROUTE ADD -p 10.255.1.0 MASK 255.255.255.0 10.255.2.1

Now my routing table looks like this:



The DMZ IP address was mapped to Public IP address and the External Firewall was configured with ACL to allow connections from any source to this IP address on port TCP/443. Visit to http://www.whatsmyip.info confirmed my configuration

The first step is to download Apache Win32 Binary including OpenSSL from this link: http://httpd.apache.org/download.cgi where I will download httpd-2.2.22-win32-x86-openssl-0.9.8t.msi. ***See Aditional Resourses at the end of this article.
The installation of Apache is pretty straightforward. Run the .msi and follow the screens…
















I did not change the default installation path and so, the bits were installed to “C:\Program Files (x86)\Apache Software Foundation”.
Before I test my installation, I want to make sure the service will bind to the correct (DMZ) IP address. To do so, I will modify “httpd.conf” file located in “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf” directory. Actually, most of the configuration is done in this file and so, I created shortcut on my desktop for easy access.
Once the file open in Notepad, locate the line “#Listen 12.34.56.78:80” and add below:
Listen 192.168.1.52:80


***Note that at this point I am configuring Apache to serve requests for HTTP traffic only. This is because I want to test the general functionality. Also, I have not deployed certificate yet, thus HTTPS protocol will not work at this point. Because Reverse Proxy will be used to serve requsts from Public Internet, I bound the service to the DMZ interface only.
Once the file is modified and saved, restart Apache service.

 
From within the Apache server, open Internet Explorer and navigate to http://192.168.1.52 (the IP address we configured Apache to listen to).

 


Before test from Public internet, we must add Firewall exception for the desired ports and protocols in the Windows Firewall (local machine). I will use PowerShell to do so.



netsh advfirewall firewall add rule name="Apache Port 80" dir=in action=allow protocol=TCP localport=80

netsh advfirewall firewall add rule name="Apache Port 443" dir=in action=allow protocol=TCP localport=443


 


Quick test from a computer on Public Internet shows that my configuration is now operational. At this point, we've accomplished a couple of very important tasks:








  1. Our Apache server is listening on the internal interface and is responding to simple HTTP queries
  2. Our Apache server is also listening and responding to queries for the Internet.
Now we can proceed with some configuration tasks.

Because we will be using Apache as Reverse Proxy and will serve SSL requests, the corresponding modules must be allowed to load at startup. To do so, some lines in “httpd.conf” file must be uncommented.

***Unix folks are familiar with text file configuration. For those who come from Windows world - Apache for Windows is basically a "port" of Unix binaries to Windows. The text file configuration structiure is preserved thus we do configuration by "uncommenting" (allowing), or "commenting" (disallowing) modules to be loaded and run. For example:


Here "mod_auth_basic.so" will be loaded, and "mod_auth_digest.so" will not be loaded.
  1. Locate and uncomment “LoadModule proxy_module modules/mod_proxy.so” line.
  2. Locate and uncomment “LoadModule proxy_http_module modules/mod_proxy_http.so” line.
  3. Locate and uncomment “LoadModule ssl_module modules/mod_ssl.so” line.
  4. Locate and uncomment “Include conf/extra/httpd-ssl.conf” line.
  5. Save the file.
***If you bounce Apache at this point, the service will not start. This is because we configured Apache in SSL mode, but no certificate was assigned yet.
I have already requested and received certificate from DigiCert, which I will use with my test setup. The certificate have CN=webext.lynclog.com and includes meet.lynclog.com, dialin.lynclog.com and lyncdiscover.lynclog.com in the list of Certificate Alternative Name list.

***The above certificate was requested during deployment of my Lync 2010 lab. There are, however, new certificate requirements for Lync 2013 publishing listed here http://technet.microsoft.com/en-us/library/jj205381(v=ocs.15).aspx and, as usual, it is highly recomended to follow Microsoft's best practices to assure sucessful deploiment.

The first step is to export the certificate with the private key to .pfx file.

Locate the certificate, right click and go to All Tasks, Export








We have to use OpenSSL to convert our .pfx file two files in format Apache can interpret.

***In the steps below I will use some custom folders for file location. Those folders are not mandatory and you can use any location.
Create folder “Certificates” on C:\ and move the exported (.pfx) certificate there.
Run Command Prompt as Administrator, and navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin” (OpenSSL executable is located there).

 
First, use this command to extract the Private Ket from the certificate:
openssl pkcs12 -in c:\Certificates\rp_cert.pfx -nocerts -out c:\Certificates\encr_lynclog.key.pem
 



IMPORTANT: Always type the commands and not copy and paste. Web browsers tend to replace characters and you might run to problems.

Use this command to extract the certificate from the .pfx file:

openssl pkcs12 -in c:\Certificates\rp_cert.pfx -clcerts -nokeys -out lynclog.cert.pem


Lastly, use this command to convert your key file in RCA format:

openssl rsa -in c:\Certificates\encr_lynclog.key.pem -out c:\Certificates\lynclog.key


Navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra” and create new folder named “ssl”. Copy the files "lynclog.cert.pem" and "lynclog.key" from "Certificates" to "ssl" folder.



Next step is to configure the SSL properties of Apache.

Navigate to “C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra”, locate “httpd-ssl.conf” file and open it with Notepad.

Locate the line "" and add the following line below:

SSLProxyEngine On


Uncomment the line “SSLSessionCache         "dbm:C:/Program Files (x86)/Apache Software Foundation/Apache2.2/logs/ssl_scache"

Comment out the line “#SSLSessionCache        "shmcb:C:/Program Files (x86)/Apache Software Foundation/Apache2.2/logs/ssl_scache(512000)"


Locate the tag “” and make sure the line “SSLEngine on” is not commented.

Now we will specify the location of our certificate and key files. Locate the line “SSLCertificateFile” and modify the path accordingly.


***As stated above, I decided to use " C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\extra\ssl" as store for my certificates. If you use different location, make sure the correct path is used.
Locate the line “SSLCertificateKeyFile” and modify the path accordingly.


Save httpd-ssl.conf (the file we were working on) and restart Apache service.
From public Internet, visit the url (https://meet.lynclog.com in this case) and make sure our server work via SSL.



The last step is configure our Apache server to act as Reverse Proxy.
Open file httpd.conf with Notepad. Locate the line “# 'Main' server configuration” and add bellow the following:
ProxyRequests Off
Order Deny,Allow
Allow from all
ProxyReceiveBufferSize 4096
ProxyPass / https://uspool.lynclog.com:4443/
ProxyPassReverse / https://uspool.lynclog.com:4443/
ProxyPreserveHost On
KeepAlive On


...where uspool.lynclog.com resolves to the VIP of my HLB handling the web services of my EE pool.

***Above we instructed Apache to proxy the requests arriving on port 443 from internet to port 4443 on our Lync server. This is equal to "Bridging" term in TMG:



Restart the service again. If everything was configured correctly, we should now see our Dialin and meet pages:









The above configuration was tested with Lync Mobile support (iPhone) and worked as well.


Summary:
This article describes how to install and configure Apache 2.2 to act as Reverse Proxy for Lync 2010 and Lync 2013 web services publishing.
 
Additional resources:


Apache direct download link: http://mirror.metrocast.net/apache//httpd/binaries/win32/httpd-2.2.22-win32-x86-openssl-0.9.8t.msi

***As of this moment, the donload link to the latest (2.2.23) bunaries is broken. You can, however, check here: http://mirror.metrocast.net/apache//httpd/binaries/win32/ for availability. Always use "Latest stable" version. Beta builds should not be used in Production.

Apache WIKI: http://wiki.apache.org/httpd/

Reverse Proxy Publishing (Lync 2010): http://technet.microsoft.com/en-us/library/gg398872.aspx

Certificate Summary for Reverse Proxy (lync 2013): http://technet.microsoft.com/en-us/library/jj205381(v=ocs.15).aspx

48 comments:

Luca Vitali said...

GREAT article! Thank you.

Luca Vitali said...

I think we can go further: do you think is it possibile to install Apache on the Lync EDGE, with 4 IP on the DMZ interface (3 for EDGE, 1 for Apache)? This could be a very optimized solution in small Lync installation.

Drago said...

I don't see any reason why having Apache on the Edge server will not work. However, I bet Microsoft will categorize such approach as "unsupported scenario" and furthermore the Administrator must consider the possible implications when or if support case is open.

Anonymous said...

Make sure the reverse proxy is correctly supporting NTLM. This is an issue with Lync 2010 on mobile clients. Connections in NTLM are authenticated rather than requests. A reverse proxy that is not NTLM friendly will cause users to find they are being identified as somebody else who happens to be using the proxy at the same time.

Thomas Wenzl said...

A Lync Reverse Proxy should be configured for not authentication. In this case it doesn't handle NTLM authentication requests, but lets the application, i.e. Lync Server 2010, do the direct authentication for those users.

Any sort of authentication configured on the reverse proxy will cause problems with Lync.

Drago said...

Excellent point, Thomas. Fact is - Apache can be used to publish Exchange as well when Basic-Auth is forced. I might blog about it too, later when Exchange 2013 go RTM.

Anonymous said...

NTLM is a connection oriented auth mechanism. The reverse proxy breaks a single connection into two. You have the connection between the client and the proxy and you have the connection between the proxy and the Lync server. If the proxy is not ensuring messages from a given client connection are only used on a given connection with the server, you end up with the server misidentifying the user. So it is not so much that the proxy is doing any type of authentication, but is maintaining the perception the server has about the connection being affiliated with a specific client. I have heard of customers using Apache as a reverse proxy hitting this exact problem, but I don't know what the work around is.

Harald Steindl said...

Perfect timing for this great post, THANKS!
Any info if this also is the same for the upcoming Lync2013?

Drago said...

Actually, for this test, I've published Lync 2013 external web services.

Unknown said...

Microsoft says: "For customers using TMG for reverse proxy, transitioning to Forefront UAG is an option. Most web publishing scenarios that are supported by TMG can be published by UAG, though specific functionality may not be identical. For customers who do not want to transition to Forefront UAG, customers should plan on transitioning to an alternative vendor solution prior to April 14, 2020" . We must work with alternative vendor!!! So, Is Apache supported?

Drago said...

As on now, I am not aware of supportability statement when comes to Apache. If one is (ever) issued, this would mean Microsoft thoroughly tested the computability and (perhaps) offer guidelines how to configure it properly.

Anonymous said...

@Allesandro - One other thng... Sadly, today, UAG won't work completely. We are still waiting on an SP to correct some behaviors in the Autodiscover. One would THINK it would work, given that the uderlying engine is TMG. But, alas... Stay tuned. I'd love to provide the guidance on how to do UAG with Lync with Mobility, to be sure.

Rick Kingslan / Lync Server UA

Anonymous said...

@Allesandro: (Apparently the picture / word verification confused me... Trying again)

'Support' and reverse proxy aren't even in the same language when talking Lync Server. If your RP can publish the Web services, deal with web tickets, can publish Autodiscover and take in the correct return, as well as deal with Simple URLs and the URIs that we need, it's as supported as anything else. For reverse proxies, we are not some concerned with WHAT it is as we are with what it can DO. If it does what we expect in a predictable manner, we would accept it. This means that, by all rights, Squid should work, as would IIS with ARR, and - naturally - Apache. And, I should know... I wrote the docs (Edge Planning, Deployment, Mobility, among others). :o)

Rick Kingslan / Lync Server UA

Unknown said...

So this worked great until I realized that mobile users were authenticating wrong. I would login and get someone else's credentials. This is with lync 2013. Does someone have a lync to fix the NTLM issue?

Drago said...

Interesting. Tim, can you provide more information the issue?

"I would login and get someone else's credentials"...

Drago

Unknown said...

basically the same as stated earlier I think it is an NTLM issue. From my mobile client if Mark R signs in before me, then I sign in, we both show up as Mark R instead of Tim L and Mark R

Unknown said...

I wonder if my topology is wrong. Should external web services be pointed at the reverse proxy server? That is how I have it now. I have lyncdiscover, meet, webext, and webservices pointed at RP. SIP, XMPP, webcon, and av point at the edge server.

Unknown said...

I found a reverse proxy DNS settings site and followed it and was missing one entry for edgepool externally. I made that change and things have been good so far today. Still nervous to mass roll out but looking good for now. Earlier posters mentioning properly configuring for NTLM but no mentions how. Thanks again, if this works will save us money :)

Unknown said...

so internally now the problem is back. If I sign in, then another user signs in right after me, we both end up signing in as me. I have lyncdiscover pointed to RP internally, lyncdiscoverinternal pointed to the Edge server internally, and webservices external pointed to lync or the RP. Is this correct?

Drago said...

Tim,

This is not correct. lyncdiscoverinternal must point to the IP address of your SE front end or the VIP of HLB if you have EE.

When mobile client from inside connect to lyncdiscoverinternal, after authentication, the client will receive the sign in URL - the EXTERNAL web services of your pool.

Let's say you have SE with FQDN lync01.domain.com where the external web service have FQDN extweb.domain.com

lyncdiscoverinternal would point to the IP address of the Lync server. After your mobile client authenticate, it will receive the URL - External web services (extweb.doamin.com in this example). From LAN, extweb.domain.com should resolve to the DMZ IP of your RP.

Remember - the goal is no matter where the client is located (Private Wi-Fi or Public internet), the sign in to appear as coming from Public internet. Only in this case persistent session can be maintained.

Unknown said...

Hi,

I am also having issues with wrong identity while coming from IOS devices, its this same for internal and external users. My dns is configured correctly (i believe) - lyncdiscoverinternal points to FE server, lyncdiscover to external interface of RP and lyncexternal to external interface of RP.
I believe the issue is with IOS implementation of NTLM authentication.
Does anybody know how to fix it?
PLEASE HELP!

Drago said...

I have seen this before. One way to resolve it is to remove the app and add it again.

However, the new client is just around the corner :-), it will use the edge server to sign just like the regular Lync client and all those troubles will go away...

Drago

Unknown said...

Hi,

thanks for the response. I am afraid removing and adding the app is not helping, the issue is with the apache proxy and i cant find a solution to resolve it. Any other ideas??

thanks,

Daniel said...

Thanks for this great article, it helped me get a reverse proxy working on linux. It works well with Lync 2010, but I'm having issues with Lync 2013. It keeps redirecting to my internal Lync 2013 FE server. Have you been able to get this working with Lync 2013?

Drago said...

Daniel,

Actually, for this post I published Lync 2013 pool.

Where you see the redirect? This is - what published URL.

Drago

smiller said...

Having an issue with the Lync 2013 client. Users are able to log in to the client and use instant messaging. They are unable to use video or voice. It seems like the connection times out on the apache side. Tried applying a Proxytimeout condition but that did not work. I am not seeing any errors on the apache side or edge. On the lync monitoring, I see the call go through without errors. Any ideas

smiller said...

Let me clarify, it is the new Lync Mobile 2013 app. Tested both on WP and IPhone, same issues.

Drago said...

Smiller,

The new mobile clients utilize UCWA (and not MCX) for "signaling". In case of audio and video calls, AV Edge is used for media just as any other client.

You are saying "It seems like the connection times out on the apache side", but fact is RP have nothing to do with media.

Re-visit the logs for more clues.

smiller said...

Nothing in the logs, does instant messaging also travel through the edge? That aspect is working.

Drago said...

From what I can find until now, when mobile client initiates communication, this is done via RP with:
POST /ucwa/v1/applications/31366441603/communication/messagingInvitations HTTP/1.1

The server then generates INVITE to the remote party and the communication is just like any other client.

When we talk about media, and endpoints are internal, media will flow in P2P manner.

If one is external and the other internal, edge will be involved.

Daniel said...

Drago,

I ended up getting it to work with Lync 2013. I was messing around with the hosts file a lot at the time since I hadn't put any records on our external DNS. I had a wrong IP address for the dialin/meet URLs and that's what made me think it was redirecting to the wrong server.

Anonymous said...

There's something fishy with the Lync Mobile 2013 client indeed, I couldn't get the Lync Mobile 2013 to work when I used Apache as a reverse proxy. I could log on to Lync Mobile 2013 but video or voice wouldn't work.

I deployed a TMG reverse proxy and everything started working, so I'm thinking something was not right in the Apache configuration. If someone can get Apache to work successfully with Lync Mobile 2013 / iOS client, please post Apache configuration...

Anonymous said...

I can confirm the same thing. Spent a week trying to get apache to work with Lync 2013. Everything but Voice and Video would work. I suspect it's something related to content type. I replaced the apache reverse proxies with IIS ARR and everything is now working.

Daniel said...

I can confirm that I got voice and video working on the Lync 2013 iOS client using Apache as the reverse proxy.

However, I have an issue where I keep getting notifications telling me that the "server configuration has changed". I've posted a thread about it: http://social.technet.microsoft.com/Forums/en-US/lyncprofile/thread/4ed779a8-457d-4a58-bcec-7587b4f95c19/

Daniel said...

I used the same configuration as Drago for both Lync 2010 and Lync 2013. The main difference is that Apache is running on an Ubuntu 12.04 LTS virtual machine on a Server 2012 host.

tatankam said...

In a Windows environment and Apache we weren't able to work with Lync 2013 mobile (voice and video didn't work).

The problem was due to content type.

In a windows environment, you see the entry:
DefaultType text/plain
in httpd.conf and this is the problem.

In an Ubuntu environment instead, you see the entry:
DefaultType None

So, this is reason that Lync 2013 mobile works on Ubuntu.

We set
DefaultType None
in our windows environment, and it works!

Best regards,
Mario

Unknown said...

We are using Apache on a Windows environemnt and came across the problem that audio and video did not work. After setting the cotnent type to DefaultType we got Andorid andiOs working, but we still cannot login in on a windows phone! In the IIS logs i can see 401 http errors.. Anyone came across the same problem?

Unknown said...

I am going to try the correct DNS settings to get rid of the wrong users authenticating. But I also had AV issues and the pesky message about config changes. Making the type none and adding these timeouts fixed both

ProxyPass / https://kisd-lync-1.domain.local:4443/ retry=1 acquire=3000 timeout=600 keepalive=On

Anonymous said...

It's greate. By the way, how to configure this reverse proxy for external users using internal office web apps?

Drago said...

It should not be any different form configuration for the other web services. Just make sure your farm is configured with external URL.

Gerald said...

Can anyone confirm or confute that this procedure also works if you use an internal certificate on your Lync FE Server.
With TMG you can use different certificates on the listener and on the FE Server.

Anonymous said...

Can anyone give me a hint why i get access denied of the iis when calling the external dialin and meet adresses?

regards
martin

Neeraj said...

Anybody having issues with Windows Phone devices?

All my iOS, Andriod, Fat clients, LWA clients work fine, but Windows Phone fails to sign in.

Using Apache on Ubuntu.

Error on device - ERROR UTILITIES CHttpConnection.cpp/1019:Request failed with the WININET errorCode (UcwaAutoDiscoveryRequest): -2147467260

Any ideas?

Thanks

LA said...

Thank you for a great post.
We are using Apache as a Reverse Proxy on a Ubuntu server. Everything is working fine except from external connection to Exchange. This happened after setting up the Apache server. I get a pop-up window asking for credentials for retrieving calendar data from Outlook. When entering credentials it just pops up again asking for credentials again. if I choose cancel I get Outlook integration error. The "Configuration information" switches between "EWS is partially and not deployed". It also switches between showing the "EWS External and Internal URL" when accessing externally. Internally the URL s are listed. If I enter the EWS External URL in a browser I must authenticate my user and a website is shown..
any suggestion/tips?

Drago said...

Do you use Apache as RP for exchange as well? Lync client contacts EWS via HTTPS indeed, but the call is directly to CAS, where Lync reverse proxy is not involved at all.

LA said...

Thanks for your answer.
We are using TMG as RP for Exchange.

Anonymous said...

I need to install and configure in centos. it is possible?

Nimsik said...

خدمات تخصصی تعمیر و سرویس لپ تاپ در تهران

https://2sottamir.ir/category/%d8%aa%d8%b9%d9%85%db%8c%d8%b1-%d9%84%d9%be-%d8%aa%d8%a7%d9%be/