Office 365 and its import service

Office 365 is the cloud service with a major adoption. One part of this is getting the on-premises Exchange-servers to be removed, and instead leveraging the Outlook Online provided service. The typically increase in allowed mailbox size is a big selling point, but additional benefits are added every day.

Migration of PST

The increased mailbox size does start the discussion of howto eliminate all PST-files spread among all the local client harddrives and the file-servers in an organization. Microsoft has offered the PST Capture tool (scan all devices, locate all PSTs and import them), and as of last year (2015) the Import PST Files to Office 365 was a way to allow sysadmins to perform a more controlled (batch) upload of files.   As always, the end-users can migrate the data via Outlook.

None of these ways are “great”. The PST Capture Tool has a great process of collecting files and dumping them, but its essentially a gather tool that will without any intelligence of what the user wants and then dump anything it can find into a mailbox.

The newly arrived Import PST file-service is a batch-management tool that seems to target the admins that has a bunch of files at a single time (or potentially a few times) to upload. Options are either to upload this into an Office 365 managed Azure Storage Space or simply ship a hard-drive with a collection of files.
There are a few people who have explained that the Office 365 Import service has a powershell interface, unfortunately its not documented and Microsoft support does not acknowledge that it exists.

In addition to the the above options provided by Microsoft there are a few third-party options – such as MessageOps.

End-user driven migration

To make something that is end-user friendly a bit of automation is required of the above tools. Currently a dropbox (a folder where users can dump the PST-files) has been designed, however inorder to get that working there quite a few hurdles that someone has to overcome inorder to produce any type of favourable result. The below notes are for my own memory…

Office 365 Import Service

As the Office 365 Import Service is a Microsoft supported tool it was it was considered the most reliable option of all of the above. Requiring Outlook for end-users to migrate the data seemed to be a high-cost and not very friendly solution. The PST Capture tool were most likely to migrate data which wasn’t relevant to the user, and the risk was of course that something was missed in the process. Third-party options inccured additional licensing cost (ontop of any Office 365 licensing) and was therefore discarded early in the process.

Account requirements

To initiate anything for the Office 365 PST Import service you are required to have certain permissions, aswell as only leveraging simple authentication if there is an intent to automate the process. If any type of multifactor authentication is enabled the ability to connect to the Office 365 via Powershell session is disabled.image

Permissions

A service account has to be setup that has the role Mailbox Import Export assigned to it. This isn’t directly granted any set of permissions so its recommended to create a new group, assign the role to it and make the service-account a member of the group.

It also seems that to be able to access the Office 365 PST Import Service from the portal one has to be a Global Admin aswell. Powershell cmdlets are only available once the Role Mailbox Import Export has been assigned.

Storage (Azure)

Office 365 Import Service offers to setup a Azure Storage Space for the tenant, and will provide the Shared Access Signature required to upload files (if using the network upload) or a storage key if using the option to shipping a harddrive. To leverage any kind of automation the only accessible path for the Office 365 PST Import service is the blob located on an Azure Storage Space. It seems that sometime in March (2016) the previous method of using the storage key to generate a Shared Access Signature (SAS) to allow for read-operations for the Import Service (technically this is performed by the Mailbox Replication Service provided by Office 365) was discontinued. One can find a storage key for the option to send in a hard-drive, however that seems to not leverage the same upload space as the network upload and therefore the storage key can’t be used.

Fortunately enough the option is only requiring an Azure Storage Space which can be provided via a normal Azure-subscription. Setup a Blob in an Azure Storage Space, and immediately you have access to the storage space. Once the Storage Space is setup you can retrieve the storage key by locating the key-icon

image

There will be two keys which you can leverage.

image

To generate a Shared Access Signature (needed for the automation part) you can download the Azure Storage Explorer 6. The tool allows a quick and easy way to view whats in a Blob on the Storage Space, aswell as generating the SAS-key.

Once you start the tool choose to add your storage space with the storage key above. Remember to check HTTPS.

image

Once you have connected to your Storage Space, choose to create a new Blob (with no anonymous access). Once this is created you can press Security to start generating the SAS-key.

image

Generate the signature by selecting a start-date (keep track of what timezone you are in) and the end-date. These dates will set the validity for the period of your SAS-key. Remember to define the actions you want to allow. To upload files you need to allow write, and to use the SAS-key for importing the files you need to allow read. There is the possibility to generate multiple SAS-keys and use them for different parts of the process.

image

A SAS-key are built of multiple parts – here comes a brief explanation;

#sv = storage services version; 2014-02-14
#sr  = storage resource; b (blob), c (container)
#sig = signature
#st = start time; 2016-02-01T13%3A30%3A00Z
#se = expiration time; 2016-02-09T13%3A30%3A00Z
#sp = permissions; rw (read,write)

 

Sample:

?sv=2012-02-12&se=9999-12-31T23%3A59%3A59Z&sr=c&si=IngestionSasForAzCopy201601121920498117&sig=Vt5S4hVzlzMcBkuH8bH711atBffdrOS72TlV1mNdORg%3D

 

Copy files to Azure Storage Space

There are multiple ways to copy files to any Azure Storage Space. You can use the Azure Storage Explorer 6 that was used to generate the SAS-key. Someone has provided a GUI for AZCopy command-line tool, but for automation the command-line usage for AZCopy is the route to go. Microsoft has written an excellent guide for this which doesn’t need any additional information.

Connecting to Office 365

Managing Office 365 for any type of automated manage is performed via a PSSession (PowerShell Session). A PSSession will import all available cmdlets from Office 365. As you can imagine quite a few are similar to Exchange, and it may therefore provide some overlap. To avoid confusion the recommended approach is to append a prefix for all cmdlets from the session which can be defined when the session is imported. This is a sample script that will provide the username and password that is required to connect to, configure the proxy-options for the Powershell session and setup the session with O365.

$password = ConvertTo-SecureString "password" -AsPlainText -Force
$userid = "name-admin@company.onmicrosoft.com" 
$cred = New-Object System.Management.Automation.PSCredential $userid,$password 
$proxyOptions = New-PSSessionOption -ProxyAccessType IEConfig -ProxyAuthentication Negotiate -OperationTimeout 360000
$global:session365 = New-PSSession -configurationname Microsoft.Exchange -connectionuri https://ps.outlook.com/powershell/ -credential $cred -authentication Basic -AllowRedirection -SessionOption $proxyOptions
Import-PSSession $global:session365 -Prefix  O365

 

Once the session is started the modules are imported with the prefix O365, as an example commands go from:

Get-Mailbox

to

Get-O365Mailbox

 

Using the Import-service via Powershell

As noticed the Office 365 Import service is a GUI only approach that is not supported for automation. That beeing said there are options to start this via Powershell. Multiple blog-posts are documenting the New-MailboxImportRequest cmdlet (and with the prefix its now: New-O365MailboxImportRequest), however Microsoft support will barely acknowledge its existance.

As long as you have the previous stated account permissions assigned (Mailbox Import Export Role) the cmdlet will be available and can be used.

For Office 365 the only supported source is an Azure Storage Space. The import-service is creating one for you, however today (2016-05-12) we are unable to create the Shared Access Signature to allow the automation part use that Storage Space. January 2016 this doesn’t seem to be the case and therefore we can assume that potentially this will change in the future.

Below command-line will allow you to start an import. If you receive the error 404 most likely there is an bad path to the file, and a result of 403 most likely is a bad SAS-key.

Remember: The O365 is the prefix we choose to use when running Import-PSSession. The actual cmdlet is New-MailboxImportRequest

New-o365MailboxImportRequest -Mailbox user@mailbox.com -AzureBlobStorageAccountUri https://yourstorage.blob.core.windows.net/folder/User/test.pst -BadItemLimit unlimited -AcceptLargeDataLoss –AzureSharedAccessSignatureToken “?sv=2012-02-12&se=9999-12-31T23%3A59%3A59Z&sr=c&si=IngestionSasForAzCopy201601121920498117&sig=Vt5S4hVzlzMcBkuH8bH711atBffdrOS72TlV1mNdORg%3D" -TargetRootFolder Nameoffolderinmailbox

 

Retrieving statistics

Once the import is started it fires off and actually goes through pretty quickly. As you can imagine the results can be retrieved by using Get-O365MailboxImportRequest and Get-O365MailboxImportRequestStatistics. One oddity was that the pipe of passing on Get-O365MailboxImportRequest to the Get-O365MailboxImportRequestStatistics didn’t work as expected. Apparently the required identifier is named Identity and it actually requests the RequestGuid.

Sample loop;

$mbxreqs = Get-O365MailboxImportRequest
foreach ($mbx in $mbxreqs) {
$mbxstat = Get-O365MailboxImportRequestStatistics -Identity $mbx.RequestGuid
$mbxstat | Select-Object TargetAlias,Name,targetrootfolder, estimatedtransfersize,status, azureblobstorageaccounturi,StartTimeStamp,CompletionTimeStamp,FailureTimeStamp, identity
}

The above data are things which was useful for a brief overview. Sometimes you can manage with the Get-O365MailboxImportRequest.

Cleanup of Azure Storage Space

What does not happen automatically (well, nothing in this process happens automatically) is the removal of the PST-files uploaded to the Azure Storage Space. Having the users PST-files located in a Storage Space will consume resources (and money), aswell as the user might be a bit uncomfortable about it. As always the attempt is to automate this process. To retrieve the cmdlets for managing the Azure Storage Space (remember, multiple ways to handle this. AZCopy is a single-purpose tool) you need to download Azure Powershell. Microsoft again has an excellent guide howto get started. What would be even faster is if all these services could provide a common approach of management. For Office 365 you import a session, but for Azure you download and install cmdlets?

Once the Azure Powershell cmdlets are installed you can easily create a cleanup job that will delete any file older than 15 days. First a time is defined. Secondaly we setup a connection to the Azure Storage Space (New-AzureStorageContext), and then we retrieve all files in our specific blob, filter based on our time-limit and then start removing them.

Good to know: Remove-AzureStorageBlob does accept –Whatif. However, –Whatif will still execute the remove. Test your code carefully… Most likely this is true for many other cmdlets.

[datetime]$limit = (Get-Date).AddDays(-15)
$context = New-AzureStorageContext -StorageAccountName $straccountname -StorageAccountKey $straccountkey -ErrorAction Stop

Get-AzureStorageBlob -Container $strblob -blob *.pst -Context $context | Where-Object { $_.LastModified -lt $limit } | ForEach-Object {Remove-AzureStorageBlob -Blob $_.Name -Container $strblob -Context $context}

 

Summary

A long rant that haven’t given anything to you. To be honest – this is memory notes for myself. The parts that are involved in creating an automated workflow requires a lot of moving bits and pieces that utilizes what a common-man would define as the cloud. The cloud is several messy parts that aren’t polished, not well documented, always in preview (technical preview, beta, early release, not launched..) and constantly changing.

All of the above are things that provided a bit of struggle. Most likely the struggle is due to lack of insight into a few of the technologies, and as more insight was gained the right questions were asked. If you read all of the links above carefully you will most likely see a few comments from me.

Exchange, Autodiscover and Exchange Proxy Settings

Ever wondered howto check the “On fast networks, connect using HTTP first, then connect using TCP/IP” from an Exchange 2010 environment and its Autodiscover functionality? Per default when configuring a profile – it is not checked. This is best practice from Microsoft if your users are connecting internally once in a while. Unfortunately there isn’t any logical name for this anywhere, but if you review the documentation for Powershell commands within Exchange 2010 – it all becomes clear. Set-outlookprovider (which is primarily used to configure the Certificate Principal Name) is the command to use and there is even an example which states the following;

This example causes Outlook 2010 clients to connect exclusively through RPC over HTTP (Outlook Anywhere) before trying RPC over TCP connections when connecting over the Internet

Reading the clarification for the OutlookProviderFlags makes it even more clear;

The OutlookProviderFlags parameter specifies that Outlook 2010 clients should connect using RPC over HTTP (Outlook Anywhere) before trying RPC over TCP connections. This increases the speed at which Outlook 2010 clients will connect when clients are primarily accessing Exchange over the Internet. The value can be set to ServerExclusiveConnect or to None to clear the flags. For Outlook 2010 clients that access Exchange over both organization intranets and the Internet, the recommended value is None, which is also the default setting.

So – to set the “ On fast networks, connect using HTTP first then Connect using TCP/IP “ you simply run the following powershell command;

Set-OutlookProvider EXPR -OutlookProviderFlags:ServerExclusiveConnect 

After you run this command you might require to restart World Wide Web Publishing Service – as per a more extensive blog-article.

Its unchecked in the below screen shot;
image

Address Book and Too Many Connections for Exchange 2010 SP1

Have you ever read the Address Book log file on a Client Access Server for Exchange 2010? Its quite interesting.

Well, not really.. It’s the log-file that is beeing produced by Microsoft Exchange Address Book Service – and details any attempts to use the address book (duh) – such as when connecting a client and looking up if the user is actually a user within the organization.

The below is an extract that occurred when all that the Outlook 2007 client connecting to the Exchange 2010 SP1 environment via RPC over HTTPC;
<date>,65690,0<ldap-user>,<ip>,<casserver>,ncacn_http,GetNewDSA,,6,Self,,Ntlm,
<date>,0,<ldap-user>,<ip>,<casserver>,ncacn_http,Bind,80004005,3
,,TooManyConnections,Ntlm,

Too Many Connections? Apparently there seems to be a limit to the to the amount of connections a specific user can have – as only one user was experiencing the problem. As opposed to the usual suspect of Client Throttling Policy – which we temporarily applied a custom one with no restrictions did not impact the behavior. Why did this occur?

Well – a single mailbox was heavily used by about 10 people. From it, they connected to several other mailboxes, and they also used mobile devices to synchronize their mail. That apparently hits a limit somewhere. Unfortunately it wasn’t to easy to find where the MSAB-service had a limit imposed – but it could be spotted in the article Configure Static RPC Ports on a Exchange 2010 Client Access server. In the article there is an Exchange 2010 Address Book Service section that details how to configure the RPC ports primarily. However, the section also contains a screenshot of a .config-file which is used by Exchange 2010 RTM and especially contains the value MaxSessionsPerUser. Perhaps this could be something? 

As it seems – the environment currently in question was an Exchange 2010 SP1 environment and apparently the previously used .config-file was now moved to the registry. However – nothing in the article (or any other article that was read) explained exactly what values could be entered. Some guess work where performed and MaxSessionsPerUser where simply entered as a DWORD value under the Address Book service configuration. As the .config-file had the default value of 50 – we simply entered a decimal value of 100 to see if the situation would improve.

regkey

Restarting the service – would of course close all connections. So we had to wait one day to confirm this would really resolve the problem…