If you’ve ever worked in environments requiring a proxy, reverse proxy, or caching system, you’ve likely heard of Squid proxy. Squid is one of the leading open-source proxy tools with an extensive community and available plugin library.
As is the case with many large, open source projects, there are many different ways to accomplish a goal and just as many opinions from folks in the community on how to best accomplish that goal.
In the case of Anitian’s Platform Team, our goal was to provide an authenticated proxy for secure environments that was integrated with Active Directory (AD) domain-joined hosts in the cloud. This meets an important FedRAMP requirement that all traffic leaving a compliant environment is authorized and secured via HTTPS. The approach we chose increased our security posture beyond FedRAMP requirements while simplifying management for our SecOps Teams and customers by requiring a user to be in a specific “AllowedSquidUsers” AD group in order to use the proxy.
From a user experience perspective, our approach meant having a user log in to a workstation with their AD credentials, make a request to the internet, and have their AD login credentials verified by the proxy. The proxy would also need to verify that the authenticated user exists in our “AllowedSquidUsers” AD group.
Configuring Squid for LDAP Authentication
Before jumping into how we accomplished this, let’s look at another common approach – Lightweight Directory Access Protocol (LDAP). If you run a Google search for “Squid AD Authentication,” you’ll mostly see results explaining how to accomplish this with LDAP. LDAP or LDAPS (the “S” is for secure) is fairly simple to configure and you can find many community guides on how to do it.
There’s just one major issue that made this approach a non-starter for us. Because of how LDAP works, it required a username and password to be left on the Squid proxy in order to allow it to make requests against AD to confirm credentials. However, leaving usernames and passwords on the proxy introduces major risks in case of a breach. More specifically, if an attacker gained access to the Squid proxy, they would have no problem swiping the AD user credentials for the LDAP binding account, thereby gaining access to all the data in the AD. Because of this, we needed a more secure solution that would exceed FedRAMP standards and meet the internal bar for security engineering at Anitian.
Configuring Squid for NTLM Authentication
Enter NTLM! Windows New Technology LAN Manager (NTLM) is an approach to this problem that doesn’t require a set of user credentials. As noted in Microsoft’s documentation:
“NTLM credentials are based on data obtained during the interactive logon process and consist of a domain name, a username, and a one-way hash of the user’s password. NTLM uses an encrypted challenge/response protocol to authenticate a user without sending the user’s password over the wire.”
Because of this, we decided to use this approach, which is also supported by Squid.
Documentation for Configuring Squid for NTLM can be found here. If you’re not a domain administrator, there are a lot of new tools referenced on this page like Samba and WinBind. I’ve broken down the useful pieces below.
Domain Joining
As a cloud security company, Anitian leverages cloud-native tooling wherever we can. If you’re building in AWS, there are some great tools to speed along the process of domain joining. We won’t cover these in detail here but using AWS Directory Services along with SSM makes domain joining a breeze. You can check out the guide here.
Verify Your Domain Join
I worked on this project with two other engineers, and this got us a few times. The SSM domain join is great but can silently fail. Because of this, I recommend that you verify your domain join status by navigating to the SSM part of the AWS portal and checking the “State Manager” panel. This should show you the associations and their status as well as the command output and errors from the domain join process.
Alternatively, you can log into the instance and run “realm list” (Linux). If you don’t see your expected domain, something is wrong, and you’ll need to troubleshoot the SSM domain joining process. Make sure your instance has a clear network path to the domain and allows the required ports to talk to it.
Configuring Squid
By default, the Squid configuration is located on your instance at /etc/squid/squid.conf. Open the file in your editor of choice. From personal experience, I highly recommend adding debug output to the top of your configuration file. More specifically, add “debug_options 29,5” for NTLM authentication logs. When you test your proxy, these logs can be viewed with “tail -f /var/log/squid/cache.log”. More details about the Squid Debug and Logging Reference can be found here.
How to Configure Squid for NTLM Authentication
Now you’re ready to implement the authentication for Squid. To get started, we’ll set the authorization helper configuration.
“””
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --require-membership-of={{ yourdomain }}\AllowedSquidUsers
auth_param ntlm children 30
auth_param ntlm keep_alive on
“””
Here we’ve established that the NTLM program can use the NTLM helper protocol and requires users to be a member of the AllowedSquidUsers group. The “children” and “keep_alive” settings are both set to their defaults, per Squid documentation, and don’t need to be edited.
Next, we’ll set the basic authorization protocol to allow for the NTLM program to have a fallback protocol.
“””
auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic --require-membership-of={{ yourdomain }}\AllowedSquidUsers
auth_param basic children 10
“””
Now we’ll set the realm setting to give users a friendly message when they’re missing authentication and create the Access Control Lists (ACL) called “proxy_auth” in order to set the authentication to required.
“””
auth_param basic realm "{{ domain }} Proxy - Authentication required"
acl ntlmauth proxy_auth REQUIRED
http_access deny !ntlmauth
“””
Almost there! We have our NTLM authentication set up, but we recommend testing it to ensure that it’s working properly before adding the group requirements. In order to allow us to test, add the http allow rule for traffic originating from the localhost. We’ll also set the listening port for Squid.
“”” http_access allow localhost http_port 0.0.0.0:3128 “””
Save the configuration and restart the service with the following:
“”” sudo systemctl restart squid.service “””
It’ll take a few seconds to restart the Squid service and reload the configuration file. Then we can run a test to ensure that everything is working as intended. Note that you’ll need a domain users’ credentials for authentication.
“”” # Your domain user needs to be in the AllowedSquidUsers group! curl https://google.com -L --proxy http://<domain_user_username>:<domain_user_password>@localhost:3128 “””
If you see “HTTP Error code 407”, that means the authentication didn’t work as expected, or your user is not in the group. Double check the status of the domain join. If you’re still seeing errors, check the “/var/log/squid/failures.log” file or add additional debug output at the top of the configuration to help troubleshoot.
While it is possible to use an “external acl” with winbind_group_helper, we recommend against this in favor of using the “–require-membership-of” flag. The external ACLs caused inconsistent behavior.
Now, if you’re like me, you’ll probably try to do something over-engineered for your test. Something like this:
“””
while true; do curl https://google.com -L --proxy 'http://localhost:3128' --proxy-user '{{domain}}\{{domain_user_username}}:{{domain_user_password}}' -s -I -o /dev/null -w "%{http_code}"; echo; sleep 3; done;
“””
Then add or remove the user from that group and watch the status code go from 407 to 200 depending on group membership status. You’ll see that this won’t work! That’s because each request resets the credential time-to-live (TTL) timer. So, for example, let’s say you’re in the group, you run the curl loop script, and start seeing 200 codes as expected. Then, while the script is still running, you remove the user from the group. You’ll continue getting 200 status codes. Each request is resetting the timer for the credential TTL instead of re-validating that the user is in that group.
Here’s the fix: just extend the test loop sleep timer to be longer than the TTL for your credentials and you’ll see the expected results.
You may remember that at the beginning of this blog, we recommended enabling debug logs. If you run “tail -f /var/log/squid/cache.log” while you’re testing, you’ll see some helpful output that will clarify what Squid is doing behind the scenes with regard to caching credentials.
Conclusion
Once explained, this behavior makes sense but can be tricky to figure out on your own. I also found that this oddity was not well documented online which was the impetus behind writing this blog in the first place. My hope is that the next engineer to tackle this problem will be able to leverage what we’ve learned and shared in this blog post.
As an extra resource for you, I created an example Squid Configuration file for NTLM Authentication in Anitian’s GitHub account which you can view here.
Happy Squid-ing!
