A Certificate Upgrade Dilemma
Last year, a customer of ours faced an almost impossible security challenge regarding certificate management. They were using SCCM 2007 with a PKI infrastructure and all the certificates required for the various functions of SCCM: IIS, PXE, Client Authentication, Site Server Signing Certificates, etc. Every 12 months, the Site Server Signing Certificate needed renewing, which was easy: renew the certificate before updating site mode settings within the SCCM console so the policies all get re-signed and clients pick up the change with no loss of service.
But this time (queue ominous music), the root CA had had an in-place upgrade—a catch-22! If they updated the server before the 12,000 clients, the clients would fail authentication and the server would not be able to deploy the new certificates to them. If they updated the clients first, the clients would instantly fail authentication once the certificate was upgraded. They couldn’t use SCCM to push out maintenance jobs. As you can see in Microsoft’s official guidance in this TechNet article, there was no easy solution.
Wait, why was this so Tricky?
The change in the root CA meant all clients could no longer get policy from the management point because of a mismatch in certificate security related to the AllowedRootCAHashCode value data. So, per the TechNet article the options at their disposal were:
- Edit the client registry
- Uninstall and reinstall the SCCM client
- Reinstall the client using command line options to provide a copy of the new site server signing certificate
None of the above are very attractive, especially when you are limited with management options to achieve this. GPO was considered. There are various ways to achieve this—such as using GPO preferences, login or computer start-up scripts—each with their own benefits and drawbacks. Had the solution been to just blank the data for the defined registry value as defined in the article (even leaving computers to then reboot naturally) using GPO preferences might have been an option. However, during testing it was not as simple as this. In the end they needed to:
- Stop the ccmexec service
- Insert the new correct value (it would not auto populate reliably)
- Restart the service
Further, they needed to cater for x64 and x86 systems, and also wanted to ensure if the job hit the same machine twice that it wouldn’t break the SCCM client again. So some decent logic was also needed to be incorporated into the script. Now, I know this doesn’t sound too tricky; however, consider the time available to test the various GPO deployment options, and the fact that if the company was in this situation for much over 24 hours the service desk (and wider business) would have started to have been impacted. In real life, it would have been a challenge using traditional tools. Initially issues would have been reported regarding application delivery and eventually with the client and server patching cycle. At that point the profile of the problem gets raised quickly as you’d expect!
An Unbelievable Solution
The customer had recently deployed Adaptiva OneSite to simplify their SCCM 2007 environment and make the migration to SCCM 2012 easier and faster. Part of the Adaptiva suite—specifically part of the content push policies—is the ability to deliver and decompress an SCCM content package to clients before ultimately executing a command line such as the aforementioned batch file. This can all be done independently of the SCCM infrastructure.
The client was aware of this functionality, and had been planning to use it to push out the SCCM 2012 client across the estate when the time came. They called Adaptiva UK and I quickly showed them what do to. In essence, the solution consisted of:
- Creating the package containing the script in SCCM (no programs required)
- Having that publish into OneSite content format (this is configured to happen automatically)
- Creating the content push policy to deliver this package to the client estate
- Configuring the advanced options to decompress the package and execute the script
They ran a small pilot and then deployed the script and fixed this issue across the entire estate within 24hrs of the problem being realised. Furthermore, because of the way OneSite works, this job was processed across the estate very quickly. All online machines received the content push policy and executed the post push command line within minutes, and any offline machines processed the job as soon as they came back onto the network.
You’ve got Bigger Problems to Solve
This security certificate upgrade dilemma is not not a nice situation to be in for any SCCM System Administrator. However, having a tool like this in your arsenal is almost priceless, even though this is probably one of the least known parts of the functionality that the Adaptiva product suite provides. It also eliminates the need for SCCM infrastructure (DPs, PXE points, SMPs, etc.) and cuts Windows 10 OSD timelines to a fraction of what they would otherwise be. Most importantly, Adaptiva has put intelligence into everything, so it dramatically lightens your workload as an admin.
If you made it this far in this blog, you’re pretty technical. So if you want more information, I’ll skip the basic intro, and send you directly to our OneSite technical deep dive. However, if you really want to begin to understand OneSite, why not download a free trial or schedule a demo.