Tuesday, November 21, 2017

winload.exe 0xc000000e Error restoring from Shadow Protect

In this post I'm going to address the winload.exe 0xc000000e error when restoring an image with StorageCraft ShadowProtect Recovery Environment.  I experienced this error when attempting to restore a Windows 10 1709 image to a number of Dell workstations.

Please note this error was blogged by MVP Philip Elder.  I found however his workaround for Windows Vista modifying the boot file on the image prior to capturing did not work for Windows 10.  His original post can be found here:

http://blog.mpecsinc.ca/2009/04/first-successful-windows-vista.html

Below is an error that you receive after restoring a Windows 10 image with StorageCraft ShadowProtect Recovery Environment on an MBR Disk.

File: \Windows\system32\winload.exe
Status: 0xc000000e
Info: The application or operating system couldn't be loaded because a required file is missing or contains errors.

 
If your computer runs a Legacy BIOS, you have no option but to setup your bootable hard drive as an MBR disk.  This is because Legacy BIOS mode does not support GPT Disks for booting.  Microsoft only supports booting GPT Disks with a UEFI compabitlbe BIOS.

Source: https://www.aioboot.com/en/gpt-legacy/

The Dirty Workaround for MBR Disks with Legacy BIOS

After liaising with StorageCraft support, the only way to get your restored Windows 10 image booting if you have a Legacy BIOS is to restore the MBR disk and partition.  This is done by setting up your drive which your recovering to as MBR and restoring the boot sector in the recovery image as shown below.


After you restore the MBR and reboot, you will get the winload.exe error shown above.  The dirty work around to fix this works as follows.  After your MBR disk restore job finishes, Go to Tools, Boot Configuration Utility.  In the Boot Configuration Utility you will see a repair button which will repair the restored image making it bootable.

 
The reason I say "Dirty Workaround" is because this utility restores the Windows Vista boot application which happens to work with Windows 10 resolving the issue, but still its not a clean restore.  Your essentially using the Windows Vista boot application to boot Windows 10 on your freshly imaged workstation.
 
Restoring as GPT Disk with UEFI

Only if your computer has a UEFI compatible BIOS (which is most modern computers these days) can you use this method.  Before proceeding with the restore, you need to configure the system as UEFI with "Secure Boot" turned off.

I have laid out the steps below for completing a Restore on a GPT disk.  First start the restore wizard in Recovery Environment.

 
Next convert the volume to GPT Disk.  If it is already a GPT Disk, you may need to convert to MBR and then back to GPT.  This is because we want it to create the 3 required partitions for us automatically.
 
 
On the largest unallocated space, right click and create a primary partition using all unallocated space.


Click next at this screen.


As we are not using an MBR type disk, these options are grayed out.  Simply click next.

 
After the volume is restored, we can go to the Boot Configuration Data and see that it is in a Bootable state.  As we are using a GPT disk, we can see there is no boot loader on the hard disk.  With GPT Disks it is the UEFI BIOS responsibility to perform this functionality and boot the operating system.
 
 
Final Thoughts
 
UEFI is the way things are going and there is a supported clean restore method with StorageCraft and UEFI for the latest Microsoft Operating Systems.  MBR Disks have a maximum capacity of 2TB and as Hard Drives at the time of this writing are expanding beyond 12TB in size, we will see MBR Disks becoming a thing of the past.
 
Many people however still utilise MBR disks for their operating system and it would be nice for StorageCraft to provide a working restore option for MBR disks on Legacy BIOS mode instead of forcing people to use a dirty workaround restoring a legacy Windows Vista boot partition onto modern operating systems.
 
Hope this post has been helpful.

Wednesday, September 27, 2017

An error occured while using SSL configuration for endpoint 0.0.0.0:444

A customer of mine contacted me complaining Exchange was down, OWA failed, and all Outlook clients could not connect.  They were running Exchange 2013 CU13.

In the system logs, the following error was experienced throughout:

Log Name:      System
Source:        Microsoft-Windows-HttpEvent
Date:          28/09/2017 7:24:32 AM
Event ID:      15021
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      exchange.domain.local
Description:
An error occurred while using SSL configuration for endpoint 0.0.0.0:444.  The error status code is contained within the returned data.


When attempting to open Exchange Management Shell, the following error was experienced.

VERBOSE: Connecting to exchange.domain.local.
New-PSSession : [exchange.domain.local] Connecting to remote server exchange.domain.local failed with the following error message : [ClientAccessServer=EXCHANGE,BackEndServer=exchange.domain.local,RequestId=3c48101b-0062-4799-b902-07b2b634321c,TimeStamp=28/09/2017 12:47:18 AM] [FailureCategory=Cafe-SendFailure]  For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Microsoft.Excha ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108477,PSSessionOpenFailed



If we look at the error, it was complaining about SSL on 0.0.0.0:444.  This is the Exchange Backend Website, the frontend website listens on 0.0.0.0:443.

We fired up IIS Manager, checked the backend website and looked at the SSL Binding.  There was no certificate attached for some reason.

I just re-attached the backend certificate, the default self signed certificate generated by the Exchange installation process is fine.
 
 
Then ran an "iisreset" on the Exchange server.
 
All was fine again.

Wednesday, September 20, 2017

Modern Public Folders and Mailbox Database Quotas

There is a very good documented procedure for migrating public folders to modern public folders on Exchange 2013 or Exchange 2016.  This article can be found under the following TechNet article:

https://technet.microsoft.com/en-us/library/dn912663(v=exchg.160).aspx

One thing that is not mentioned in this article as a pre-requisite but should always be checked prior to proceeding with a migration is the Mailbox Database quotas that are in place.  Public Folder mailboxes adhere to any Mailbox Database quotas in place for the mailbox database they reside under.  You want to ensure the public folder content being migrated to each public folder mailbox does not exceed your mailbox database quota limit or you will run into issues during the migration.

In the event the public folder data being migrated does exceed your mailbox database quote, like with standard mailboxes you can increase the quota to the public folder mailbox to overwrite the database defaults with the following command.

Set-Mailbox PublicFolderMailbox -PublicFolder -IssueWarningQuota 1000MB -ProhibitSendQuota 1100MB -ProhibitSendReceiveQuota 1200MB -UseDatabaseQuotaDefaults $False

Make sure you check this prior to migration.

Outlook Clients continuously prompt for username and password - Exchange 2016 Migration

In a recent Exchange 2010 SP3 Update Rollup 18 to Exchange 2016 CU5 migration for a client, we had issues with some users prompting for username and password continuously after their mailbox was moved to Exchange 2016.

After looking into this further, I noticed the HTTP§1§1§§§§§§ and OWA§1 were missing from the protocolSettings when comparing a problematic user to my lab.

 
The settings in protocolSettings are controlled by the user CASMailbox settings "Set-CASMailbox".  As you see this user has IMAP4 and POP3 disabled as it has a 0 on it.

We want HTTP and OWA enabled with a 1.

After adding the following to the users protocolSettings and performing an iisreset this resolved the issue for the users in question.

HTTP§1§1§§§§§§
OWA§1

 
Hope this has been helpful.

Monday, September 18, 2017

Preparing Windows 10 Enterprise Edition 1703 for Enterprise Deployment with SCCM 1702

This blog post goes through what is required to get Windows 10 Enterprise Edition 1703 ready for deployment in an enterprise environment with Internet Explorer.

In Windows 10 Enterprise Edition 1607, a common practice to prepare the image for deployment was to create a custom default profile.  This was done by creating a temporary user account on the base image and customizing it such as removing edge, store and windows mail from task bar and removing modern apps from start menu tiles.   The profile of this temporary user would then be used to create a new "Default profile" under C:\Users and the old profile would generally be renamed to something like Default.old or deleted entirely.

In Windows 10 Enterprise Edition 1703 however, customizing the default profile results in Sysprep failing with an error.  This error results in an infinite loop of restarts after the first boot with the following text:

Why did my PC restart?

There's a problem that's keeping us from getting your PC ready to use, but we think an update will help get things working again.
Here's how to update:
  1. Make sure your PC is plugged in
  2. If this PC uses Wi-Fi, select Next to following instructions to connect to a Wi-Fi network.
  3. If this PC does not use Wi-Fi, insert a network cable to connect to a wired network, and then select Next.
  4. Once you're connected, select Next, and the update will install.

As a result we must perform all modifications to the image without modifying the default profile on the base image as a work around.  After leasing with Microsoft, we also do not want the Windows 10 1703 image to ever touch the Internet as it will download additional bloatware and updates during the installation process which can also cause Sysprep to fail.

Below is the documented steps for creating an Enterprise Ready Windows 10 Enterprise 1703 build with the bloatware stripped out and Internet Explorer as standard browser so your legacy Enterprise web applications continue to function.

Step 1 - Create a new Virtual Machine with no Internet

You want to create your image on a virtual machine, not a physical workstation.  Do not install VMware Tools or HyperV Integration Services as we want to keep the image clean.  The image will eventually be deployed to physical hardware and as a result we do not want such software on the Windows 10 Enterprise build.

Make sure you use all generic virtual hardware, for example on VMware make sure you use E1000E Virtual NIC, not VMXNET3 as this requires custom drivers from VMware Tools.

Install Windows 10 from the latest Windows 10 Enterprise 1703 ISO.  Make sure the VM is disconnected from the Internet during the build process to ensure it cannot download updates.

Step 2 - Enable Sysprep Audit Mode

Immediately after the install finishes, enable Sysprep in Audit Mode.  You use audit mode to setup the default profile which will affect all users that log into the computer.

Do not generalize the image and simply select reboot.


Step 3 - Unpin Applications from Start Menu and Taskbar

Whilst in Audit mode, go through and unpin all the modern apps from the Start Menu.  Also unpin anything you want from the task bar such as store, windows mail etc.


Step 4 - Remove Bloatware

Next we want to go through and remove all bloatware from the image.  In Windows 10 Enterprise 1607 we could simply achieve this with the following command:

Get-AppxPackage -AllUsers | Remove-AppxPackage

On Windows 10 Enterprise 1703 however we cannot do this or it will break sysprep.  As a result we need to specify the individual bloatware applications we wish to remove.  Here is the list I used on my image, tailor it for your needs.

Get-AppxPackage -allusers *Adobe*
Get-AppxPackage -allusers *EclipseManager*
Get-AppxPackage -allusers *WindowsFeedbackHub*
Get-AppxPackage -allusers *MicrosoftOfficeHub*
Get-AppxPackage -allusers *GetStarted*
Get-AppxPackage -allusers *zune*
Get-AppxPackage -allusers *messaging*
Get-AppxPackage -allusers *solitaire*
Get-AppxPackage -allusers *bingnews*
Get-AppxPackage -allusers *bingweather*
Get-AppxPackage -allusers *skypeapp*
Get-AppxPackage -allusers *stickynotes*
Get-AppxPackage -allusers *xboxapp*
Get-AppxPackage -allusers *windowscommunicationsapps*
Get-AppxPackage -allusers *OneConnect*
Get-AppxPackage -allusers *3DBuilder*
Get-AppxPackage -allusers *3DViewer*
Get-AppxPackage -allusers *Pandora*
Get-AppxPackage -allusers *PowerBI*
Get-AppxPackage -allusers *CandyCrush*
Get-AppxPackage -allusers *speedtest*
Get-AppxPackage -allusers *QuickAssist*
Get-AppxPackage -allusers *Office.Sway*
Get-AppxPackage -allusers *Twitter*
Get-AppxPackage -allusers *bingsports*
Get-AppxPackage -allusers *Duolingo*
Get-AppxPackage -allusers *ActiproSoftwareLLC*
Get-AppxPackage -allusers *RemoteDesktop*


Also run the following commands so the applications are no longer available for the next user:

Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*Adobe*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*EclipseManager*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*WindowsFeedbackHub*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*MicrosoftOfficeHub*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*GetStarted*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*zune*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*messaging*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*solitaire*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*bingnews*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*bingweather*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*skypeapp*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*stickynotes*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*stickynotes*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*xboxapp*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*windowscommunicationsapps*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*OneConnect*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*3DBuilder*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*3DViewer*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*Pandora*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*PowerBI*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*CandyCrush*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*speedtest*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*QuickAssist*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*Office.Sway*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*Twitter*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*bingsports*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*Duolingo*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*ActiproSoftwareLLC*"} | Remove-AppxProvisionedPackage -online
Get-appxprovisionedpackage –online | where-object {$_.packagename -like "*RemoteDesktop*"} | Remove-AppxProvisionedPackage -online


Step 5 - Prevent the Image from downloading more Bloatware

To prevent the image from downloading more bloatware when we connect it to the Internet, we need to add the following registry key.  This stops it from downloading additional non essential applications considered by many as "bloatware".

reg add HKLM\Software\Policies\Microsoft\Windows\CloudContent /v DisableWindowsConsumerFeatures /t REG_DWORD /d 1 /f

Step 6 - Create an Unattended xml file

Next create an unattended xml file.  I placed this on the image under C:\Windows\System32\Sysprep.

CopyProfile = $true in the XML file instructs to make the changes made in Audit Mode the default profile on the image.

<?xml version="1.0" encoding="utf-8"?><unattend xmlns="urn:schemas-microsoft-com:unattend">
<settings pass="specialize">
<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
<CopyProfile>true</CopyProfile>
</component>
</settings>
<cpi:offlineImage cpi:source="wim:D:/sources/install.wim#Windows 10 Enterprise" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

Also mount your Windows 10 Enterprise 1703 DVD to your image under D:\ to match the offlineImage path of the unattend XML file.

Name the XML file anything other then unattend.xml as this is the default file Windows 10 uses.

Step 7 - Run Sysprep

Next run Sysprep from an elevated command prompt.

sysprep.exe /generalize /oobe /shutdown /unattend:c:\windows\system32\sysprep\Win10unattendanswer.xml

Snapshot the image after it is shutdown and confirm that it boots correctly and runs sysprep without errors.  Once you have confirmed this, roll it back to the snapshot ready to capture the image.

Step 8 - Capture the Image with DISM

Next capture the image with DISM using a command similar to the following:

Dism /Capture-Image /ImageFile:c:\my-windows-partition.wim /CaptureDir:C:\ /Name:"My Windows partition"

For more information on capturing with DISM please refer to the following website:

https://technet.microsoft.com/en-us/library/hh825072.aspx

Step 9 - Removing Edge and Pinning Internet Explorer with SCCM

Despite removing the Edge icon from the image in the default profile, the CopyProfile part of sysprep does not bring the change across.  Other start menu changes all stay in place.

Microsoft MVP J├Ârgen Nilsson has created a script to use in an SCCM task sequence to ensure Edge stays removed.  He published this here:

http://ccmexec.com/2015/12/removing-the-edge-icon-from-the-taskbar-during-osd/

This script has also been published to TechNet Gallery under the following location:

https://gallery.technet.microsoft.com/Manage-the-taskbar-remove-c3024e40

This script however whilst it removes Edge, does not pin Internet Explorer in its place.  Here is 2.0 of this script which pin's Internet Explorer in the place of Edge.  Please download from the following link:

https://sites.google.com/site/cbblogspotfiles/ManageTaskbar%202.0.zip

Step 10 - Create an SCCM Package for the Script

This procedure assumes your using SCCM 1702 to deploy your Windows 10 image.

For this process we want to create a new SCCM Package, not a Application.

 
Navigate to the path on the network to where the Zip file was extracted.  If you didn't see the link above, you can download it from:
 


Select "Do not create a program" and click Finish.


And as always with SCCM, distribute the package to the distribution points.

 
Step 11 - Modify the SCCM Win10 Deployment Task Sequence

Next we want to configure the SCCM task sequence to run the batch script we imported to a package.  This batch script simply imports a registry key to the default profile and configures a "runonce" to ensure all new users that login to the image run the PowerShell script to modify the task bar.

To run the batch file we want to add a "Run Command Line" option at the end of the task sequence usually as the last step.  Simply select the package and add in the command line area "TaskBar.cmd".


This will ensure after new machines are deployed Edge will be removed and Internet Explorer will be put in their place.

Extra Steps

I recommend considering to deploy the following group policy settings to your Windows 10 computers:

Disable the Windows Store:

Computer Configuration --> Administrative Templates --> Windows Components --> Store --> "Turn of Windows Store"

Disable the OneDrive:

Computer Configuration --> Administrative Templates --> Windows Components --> One Drive--> "Prevent the usage of OneDrive for file storage"

Disable Cortana:

Computer Configuration --> Administrative Templates --> Windows Components --> Search --> "Disable Cortana"

Set the default applications to use IE instead of Edge.  This requires you create a xml AppAssoc file with DISM and deploying it with Group Policy.  See the following web page:

https://docs.microsoft.com/en-us/internet-explorer/ie11-deploy-guide/set-the-default-browser-using-group-policy

To create the AppAssoc XML file use the following DISM command:

dism /online /export-defaultappassociations:\\server\share\\AppAssoc.xml

Shoutout:
I would like to provide a huge thanks to Amit Anand from Microsoft for working with me and attending numerous skype meetings on creating this solution.

Hopefully this post has been helpful. 

Friday, September 15, 2017

Outlook Error - There is a problem with the proxy server's security certificate (Error Code 80000000)

During an Exchange 2010 SP3 to Exchange 2016 CU5 upgrade, some Outlook clients intermittently received the following certificate error message:

There is a problem with the proxy server's security certificate.
Outlook is unable to connect to the proxy server mail.domain.com (Error Code 80000000).
 


All Outlook Clients were running 2013 SP1.

The error was only experienced after moving a mailbox from Exchange 2010 to Exchange 2016.  The issue was also intermittent, not all workstations received the error message.

On the Exchange 2016 server the following was configured:
  • A valid digital certificate from RapidSSL was installed on the Exchange 2016 server along with the intermediate certificate and root certificate.
  • All names for Internal and External URLs were set to a name that matched the digital certificate for Outlook Anywhere, MAPI over HTTPS, Exchange Web Services, OAB Distribution and Autodiscover.
  • DNS was configured to direct all connections to the new Exchange 2016 servers.
  • We had a proxy exclusion in place on all workstations to ensure data to the Exchange server "mail.domain.com" was sent direct, not through the proxy.
This error was a bogus error and had nothing to do with a proxy server!

The server was configured with MAPI over HTTPS enabled on the organization configuration and all Outlook clients were creating MAPI over HTTPS connections to the server.

After much troubleshooting, we set the InternalClientsRequireSSL to $true and did an "iisreset" on the servers Outlook Anywhere configuration.

After changing InternalClientsRequreSsl to $true the issue did not occur.

 
What is interesting about this, I set InternalClientsRequireSSL to $false in my Exchange environment running Exchange 2016 CU4 and restarted IIS, I was not able to reproduce the issue with Outlook 2016.
 
I also have other clients that have InternalClientsRequireSSL set to $false that are doing SSL Offloading via a load balancer running Exchange 2016 that do not experience this issue.
 
It is an interesting issue as it was intermittently occurring across workstations at my client site.
 
The issue was definitely related to InternalClientsRequireSSL as I tested setting it back to $false, as soon as changing it back to $false the issue immediately reoccurred.
 
Perhaps a bug between Outlook 2013 SP1 and Exchange 2016 CU5?  InternalClientsRequreSsl is a valid setting and there are many cases where this must be set to $false.
 
Hope this post has been helpful.

Thursday, September 7, 2017

Autodiscover: Outlook Provider - 6001 Error for one user only

We had issues with Autodiscover with only one mailbox in our environment.

When we ran the Test-OutlookWebServices against the problematic mailbox, we got an error.

Test-OutlookWebServices -Identity "journal@domain.com" -MailboxCredential (get-credential domain\journal)


Looking at the full report with the format list option "| fl" we get:


Test-OutlookWebServices -Identity "journal@domain.com" -MailboxCredential (get-credential domain\journal) | fl

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     : mail.domain.com
Scenario            : AutoDiscoverOutlookProvider
ScenarioDescription : Autodiscover: Outlook Provider
Result              : Failure
Latency             : 16
Error               : System.Net.WebException: The remote server returned an error: (401) Unauthorized.
                         at System.Net.HttpWebRequest.GetResponse()
                         at
                      Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
                         at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
Verbose             : [2017-09-08 04:38:10Z] Autodiscover connecting to
                      'https://mail.domain.com/Autodiscover/Autodiscover.xml'.
                      [2017-09-08 04:38:10Z] Test account: journal Password: ******
                      [2017-09-08 04:38:10Z] Autodiscover request:
                      User-Agent:
exchangeserver/Test-OutlookWebServices/journal@domain.com
                      Content-Type: text/xml; charset=utf-8
                      Host: mail.domain.com
                      Cookie: X-BackEndCookie=S-1-5-21-2167321796-859855631-2145623002-1367=rJqNiZqNgayprdK6p7y3vrG4utG
                      SnpGVlpKKj4yX0ZOQnJ6Tgc7GzMjGxsjGy8iBzc/OyNLPxtLPx6vPy8XLx8XOzw==
                      [2017-09-08 04:38:10Z] Autodiscover request:
                     
                      http://www.w3.org/2001/XMLSchema
"
                      xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"
                      xmlns="
http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
                       

                          journal@domain.com
                         
http://schemas.microsoft.com/exchange/autodiscover/outlook/response
                      schema/2006a

                       

                    

                      [2017-09-08 04:38:10Z] Autodiscover response:
                      request-id: eee84b62-bc41-4363-90b6-4c47e136a08d
                      X-SOAP-Enabled: True
                      X-WSSecurity-Enabled: True
                      X-WSSecurity-For: None
                      X-OAuth-Enabled: True
                      Server: Microsoft-IIS/8.5
                      WWW-Authenticate: Negotiate,NTLM,Basic realm="mail.domain.com"
                      X-Powered-By: ASP.NET
                      X-FEServer: exchangeserver
                      Date: Fri, 08 Sep 2017 04:38:10 GMT
                      Content-Length: 0
                      [2017-09-08 04:38:10Z] Autodiscover response:
                      System.Net.WebException: The remote server returned an error: (401) Unauthorized.
                         at System.Net.HttpWebRequest.GetResponse()
                         at
                      Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
                         at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
MonitoringEventId   : 6001RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : ExchangeWebServices
ScenarioDescription : Exchange Web Services
Result              : Skipped
Latency             : 0
Error               : Skipped testing Exchange Web Services because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5002

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : AvailabilityService
ScenarioDescription : Availability Service
Result              : Skipped
Latency             : 0
Error               : Skipped testing Availability Service because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5003

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : OfflineAddressBook
ScenarioDescription : Offline Address Book
Result              : Skipped
Latency             : 0
Error               : Skipped testing Offline Address Book because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5004


To resolve this issue I compared all attributes from the bad mailbox "journal" against a working mailbox.  To quickly get an attribute dump from a user account in Active Directory you can use the following command:

Get-ADUser username -Properties * | Select *

To compare the attributes against a working account, I simply used the windiff tool available from http://www.grigsoft.com/download-windiff.htm

I noticed the problematic account had the protocolSettings set as shown in the screenshot below:

 
All other accounts had protocolSettings set to "RemotePowerShell§1", so I corrected this as shown in the screenshot below.
 
 
After making this change on the mailbox I tested again - it failed.  This is because Exchange caches Active Directory objects and attributes - usually for up to an hour to reduce load on Domain Controllers.  To get the web app to flush its cache, I simply did an "iisreset".
 
Running the command again and the Autodiscover test passed.
 

Thursday, August 31, 2017

Outlook is unable to connect to the proxy server. (Error Code 0)

I'm currently in the process of performing another Exchange 2010 to Exchange 2016 migration for a customer.  When moving the first mailbox to Exchange 2016, the following error occurred:

There is a problem with the proxy server's security certificate.
The name on the security certificate is invalid or does not match the name of the target site mail.domain.com

Outlook is unable to connect to the proxy server. (Error Code 0)

 
Most information on the internet regarding this error points at either a certificate issue as per https://support.microsoft.com/kb/923575 or the Exchange System Attendant Service not running which no longer exists in later versions past Exchange 2010.
 
The Microsoft Remote Connectivity analyzer passed fine with no certificate areas and I validated that the certificate was correct:
  • The Root certificate is installed correctly on the server with the correct thumbprint.
  • The intermediate certificate is installed correctly on the server with the correct thumbprint.
  • Awildcard certificate is installed on the server with private key and the certificate chain is healthy.
  • The certificate is valid expiry date and has a valid subject name.
  • All names on the virtual directories for Exchange match a valid name trusted by the wildcard certificate.
After hours of troubleshooting I isolated the issue down to Group Policy and then finally down to this specific policy setting applied to the User Account:

"RPC/HTTP Connection Flags"

This is located under User Configuration --> Administrative Templates --> Microsoft Office 2013 --> Account Settings --> Exchange

Provided you have the Exchange 2013 ADMX installed in the Group Policy Central Store.
 

 
After setting this policy back to "Not Configured" and refreshing policy on the users, the error was resolved.

Tuesday, August 15, 2017

Slow Internet Surfing on Cisco 891fw

I was doing a new config for a Cisco 891fw but had extremely slow internet surfing speeds.

When doing a speed test however, speed was fine.

The issue was MTU.

To fix this, on my VLAN that I assigned to my clients I needed to run the following command to stop the packets fragmenting:

ip tcp adjust-mss 1200

Make sure you put this on the VLAN itself, not the Dialer Interface.

Monday, August 14, 2017

Removing Icons in Windows 10 1703 after Removing Bloatware with Remove-AppxPackage

We had an issue creating a Windows 10 image at another customer site after upgrading the Image to 1703.  The 1703 update introduced yet more bloatware.

After removing the additional bloatware from our image introduced in Windows 10 1703 with "Get-AppxPackage -AllUsers | Remove-AppxPackage", we had a number of apps which were still in the start menu for all users but could not be removed and were not functional.
  • Adobe Photoshop Express
  • Eclipse Manager
  • MSN News
  • And a few others...
 
In order to remove these apps from the start menu, we first needed to add all the bloatware back with the following command:
 
Get-AppXPackage -AllUsers | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\AppXManifest.xml"}
 
Once all modern apps were in an active state, we then removed the problematic applications which we could not remove... Adobe Photoshop Express, Eclipse Manager etc.  Once they were removed we then re-ran the "Get-AppxPackage -AllUsers | Remove-AppxPackage" cmdlet.
Now they were gone.  We created a new default profile, cleaned up the image and sysprepped it for deployment.
 
Hope this is helpful to someone else.

Thursday, August 10, 2017

What is the difference between Resume-MailboxDatabaseCopy and Update-MailboxDatabaseCopy?

Resume-MailboxDatabaseCopy and Update-MailboxDatabaseCopy - two separate commands which perform similar actions for similar purposes.  But what is the difference and when would you use one over the other?

When a mailbox database in a Database Availability Group environment enters a FailedAndSuspended state, you will be needing to run one of these commands!

The Resume-MailboxDatabaseCopy will attempt to replicate only uncopied transaction logs to bring the database back into a healthy state.  Minimal bandwidth utilization and effort required by the servers.

The Update-MailboxDatabaseCopy command on the other hand will perform a complete database reseed between the servers to bring the database back into a consistent state.

As a rule of thumb, you always want to try the Resume-MailboxDatabase cmdlet before you attempt a full reseed.

Be very careful when running the Update-MailboxDatabaseCopy cmdlet if your servers are in different datacenters connected by slow WAN links.  If this is the case and you need to run this command, you want to schedule it for a time which will least impact users.  Remember, Exchange DAG's do not support bandwidth throttling even in Exchange 2016 primarily because Microsoft thinks everyone should run DAG Replication on a separate dedicated isolated network.

Thursday, August 3, 2017

In-Place Hold vs Litigation Hold vs Outlook Protect Rules

In this post I am going to address what the difference between In-Place Hold vs Litigation Hold vs Outlook Protect Rules in Microsoft Exchange.

Litigation Hold

Litigation Hold was first introduce in Exchange 2010 and is configured per mailbox.  You set mailboxes on litigation hold on a per mailbox basis.  Exchange 2016 also allows you to specify a hold duration - "how long you want to hold items in a mailbox for".

To put a mailbox on Litigation Hold, simply use the following command:

Set-Mailbox clint@contoso.com -LitigationHoldEnabled $true

In-Place Hold

Unlike Litigation Hold which is on a mailbox level, In-Place Hold allows you to hold items across your organization based on a query such as keywords, senders and recipients, start and end dates, and also specify the message types such as email messages, calendar items, and Skype for Business conversations that you want to place on hold.

In other words its a query based search and puts individual items such as emails on hold instead of an entire mailbox.

In-Place Holds are created using the Compliance management > eDiscovery Center.  For more information on how to do this, refer to the following article.

https://technet.microsoft.com/en-us/library/dd979797(v=exchg.160).aspx

Outlook Protection Rules

Unlike Litigation Hold and In-Place Hold, Outlook protection Rules help your organization protect against the risk of information leakage by automatically applying Information Rights Management (IRM) protection to messages.

Outlook Protect Rules are created on the Exchange Server with the New-OutlookProtectionRule cmdlet.  These rules are then automatically distributed to the correct Outlook clients via Exchange Web Services.

Before you create Outlook Protect Rules you must have an AD RMS server deployed in the same Active Directory forest as your server running Microsoft Exchange Server.

Hope this information was helpful!

Monday, July 17, 2017

Exchange Managed Availability Spamming the Internet

I ran into an issue at a customer site where Exchange Managed Availability was spamming the Internet with large amounts of outbound to the Internet.  These "Undeliverable: Inbound proxy probe" NDR messages were routing out to the Internet via the Send Connector to inboundproxy@contoso.com - an email address which was intended for internal use within Exchange Environments monitoring purposes.


In the logs we can see the health probe emails are being routed to the Health Mailboxes @domain.local.


These emails are not being delivered and bouncing back to the sender “inboundproxy@contoso.com” which of course “@contoso.com” is not an Accepted Domain in your Exchange environment so it is routed out to the internet for delivery.

The Health Mailboxes (part of Microsoft Exchange Managed Availability) all reside under the Monitoring Mailboxes location.


These mailboxes are automatically created by the "Microsoft Exchange Health Manager" service, if any of these mailboxes get deleted, the service will recreate them upon restart.

All these health mailboxes have a have the external SMTP address configured and a domain.local SIP address.

SIP, not SMTP.  Hence delivery fails.


Managed Availability automatically routes to the FQDN of whatever the forest root domain is for the Health Mailboxes and as a result, this SMTP address must exist on the Health Mailboxes.

I created a PowerShell script to return all health mailboxes which do not have a SMTP address matching alias@domain.local and then add the missing email.

Get-Mailbox -Monitoring | %{ if(!($_.EmailAddresses -contains "smtp:$($_.Alias)@domain.local")) { Write-Host "Adding email address  for $($_.Alias)"; Set-Mailbox $_.Alias -EmailAddresses @{add="$($_.Alias)@domain.local"} } }


We can validate the @domain.local is now on all Health Mailboxes.


The reason why these domain.local addresses were not on the Health Mailboxes was someone removed @domain.local from the default Email Address Policy in the customers environment.

I hope this post was helpful. 

Software no longer displaying in Software Center - SCCM 1702

I had an issue at a customer where software was no longer appearing in Software Center on the workstations.  Please note - this screenshot was taken after I fixed the issue.


After much troubleshooting, I was able to fix the issue.

For each application not appearing, I needed to do an "Update Content" in SCCM.


To force the SCCM client to update the application after the content was updated, in the Configuration Manager client, under Machine Policy Retrieval & Evaluation click Run Now.


Monitor the C:\Windows\CCM\Logs\AppDiscovery.log on the client and you should see the application which you ran the Update Content for appear in the log file.

 
When the application appears, close and re-open Software Center, you will see the application present.
 
I'm not sure what actions lead up to the many of the applications disappearing in Software Center, however these actions resolved the issue. 

Forcing Azure AD Connect to Sync

The new Azure AD Connect Tool was released in April 2017 to replace the legacy DirSync utility for maintaining Identity Management with Office 365.

This new tool syncs much more frequently then the legacy DirSync tool - every 30 minutes by default to be exact.

However when creating a new on-premises user, there may be times where you want to instantly synchronize the user to Azure Active Directory.

This can be done with PowerShell on the new tool with the following cmdlet:

Start-ADSyncSyncCycle -PolicyType Delta

 
I hope this post has been helpful.
 

Wednesday, July 12, 2017

.NET Framework 4.7 and Exchange Servers

.NET Framework 4.7 is currently not supported on Microsoft Exchange Servers. This is heavily documented on the Internet with many articles such as the one below:

https://supertekboy.com/2017/06/17/dont-install-net-framework-4-7-on-exchange/

However Windows Update automatically installs .NET Framework 4.7 now on servers and many of my customers have been caught out by accidently installing this latest framework update on Exchange Servers unknowingly by simply installing the latest patches.

To ensure you don't get caught out, it is recommended to put the following registry key on your Exchange Servers to block the installation of .NET Framework 4.7 on Exchange servers.

  1. Tap on the Windows-key, type regedit.exe, and hit the Enter-key on the keyboard. This should start the Windows Registry Editor.
  2. Go to the key: HKEY_LOCAL_MACHINE\Software\Microsoft\NET Framework Setup\NDP
  3. Right-click on NDP and select New > Key.
  4. Name that key WU.
  5. Right-click on WU, and select New > Dword (32-bit) Value.
  6. Name it BlockNetFramework47.
  7. Set its value to 1 (double-click it to set value).

Wednesday, June 28, 2017

Exchange 2016 or Office 365 Shared Mailbox Not Updating

I was addressing an interesting issue for a customer today where a user with a shared mailbox would not automatically update.  Their primary mailbox updated correctly, only the shared mailbox did not update.

All other shared mailboxes for other users worked correctly.

All users are running Outlook 2013 32bit with all latest patches at the time of this writing.

When a user navigates to the shared mailbox and clicks Send/Receive All Folders or Update Folder, it will not update.


The only way the user can update the shared mailbox is by closing and reopening Microsoft Outlook.

All other users in the environment do not have issues with shared mailboxes.

After doing some research, it appears to be an issue with Microsoft Outlook when dealing with shared mailboxes over 2GB in size.  This shared mailbox having the issue was indeed over 2GB in size.

There are numerous forum threads on the Internet with people experiencing this Outlook issue:
This issue only occurs when shared mailboxes are being cached locally.

As a workaround, simply prevent the shared mailboxes from being cached locally by disabling "Download shared folders" on the users Outlook profile.  This is a confirmed workaround to the issue.

Security Vulnerability in Azure AD Connect

If you have recently upgraded your DirSync synchronization tool to Azure AD Connect to get your contacts up to Office 365, you will need to do it again.

An exploit in the new Microsoft cloud synchronization tool has just been discovered which allows elevation of permissions.  This exploit allows an attacker to reset the password to an on-premises Active Directory account and gain privileged access such as Domain Admin over a companies domain.

The exploit is in the "Password write back is a component of Azure AD Connect" which needs to be enabled for this exploit to work.

A write-up of this security vulnerability can be found here:

https://technet.microsoft.com/library/security/4033453.aspx?f=255&MSPPError=-2147217396

Luckily most my customers are still using DirSync and are not affected by this vulnerability.

For a comparison between DirSync and Azure AD Connect please see:

https://docs.microsoft.com/en-us/azure/active-directory/active-directory-hybrid-identity-design-considerations-tools-comparison

Sunday, June 18, 2017

For Each Line in Text File Do - Batch Script

Below is a simple batch script which takes each line of a text file and lets you use it in a script.  I have provided an example of this below.

I have needed FOR EACH, DO batch scripts numerous times over the years and its always hard to find a good one on the Internet.

@ECHO OFF
For /f %%i in (c:\computerlist.txt) do (
Echo ************************
Echo %%i
Echo ************************
psexec \\%%i -h -u domain\username -p password "\\domain\netlogon\mybatchscript.bat"
)
pause

Very handy during day to day sysadmin tasks!

Thursday, May 18, 2017

WSUS Clients not Reporting or downloading updates

I had a large customer where all clients stopped downloading updates in November 2016.  Even after building a new WSUS server, clients would not update.

Clients were reporting the following error:

Code: 80244008 Windows Update encountered an unknown error.


The newly built WSUS server had all clients coming up in the "All Computers" list however no clients were reporting.

To resolve the issue, we had to delete the "C:\Windows\SoftwareDistribution" folder on each workstation.  This can be done with the following batch script:

net stop wuauserv
rd /s /q %windir%\SoftwareDistribution
net start wuauserv
wuauclt /detectnow /reportnow


After running this, the client then reports into WSUS and begins downloading updates.


 
Now this client has approximately 1000 computers on their network.  We do not want to go around to every workstation to run the batch script and delete the SoftwareDistribution.
 
You can use psexec from SystemInternals to do this across all computers in one batch script.  Save the batch script above to \\domain\netlogon as shown below:
 
@ECHO OFF
For /f %%i in (c:\computers.txt) do (
Echo ************************
Echo %%i
Echo ************************
psexec \\%%i -h -u domain\username -p password "\\domain\netlogon\resetsoftwaredistribution.bat"
)
pause
 
You will need to get a list of all computers from WSUS that are not reporting.  As you can not export lists from WSUS Management Console, you will need to install SQL Management Studio and connect to the Windows Internal Database (WID) hosting WSUS - or an external database in the event your not using WID!
 
 
Use the following TSQL query to get the first 1000 rows from the tbComputerTarget table in the SUSDB database.
 
SELECT TOP (1000) [TargetID]
,[ComputerID]
,[SID]
,[LastSyncTime]
,[LastReportedStatusTime]
,[LastReportedRebootTime]
,[IPAddress]
,[FullDomainName]
,[IsRegistered]
,[LastInventoryTime]
,[LastNameChangeTime]
,[EffectiveLastDetectionTime]
,[ParentServerTargetID]
,[LastSyncResult]
FROM [SUSDB].[dbo].[tbComputerTarget]
 
Use the following TSQL query to get the first 1000 rows from the tbComputerTarget table in the SUSDB database.  All the computers who have never reported or synced will have status NULL.
 
Use the FullDomainName column to copy and paste the hostnames of the computers into the c:\computers.txt text file on your PSEXEC computer.
 
 
 Running the script against all the remote workstations will fix your issue!