The majority of data in the enterprise today has a location component, and this includes much of the entity data in Dynamics CRM. We can leverage the location attributes of our CRM data to provide a wide variety of location-based functionality, including geospatial visualization, finding nearest service agents to jobs, optimizing routes for mobile sales people, analyzing our data in heat maps and thematic maps, and much more. One of the fundamentals required to be able to leverage this location data is the “geocoding” of our location data. Geocoding is the process of taking text-based location data such as addresses or place names, and turning them into geographic coordinates. With accurate geographic coordinates (or latitudes and longitudes) for our entities, we can visualize them on a map, and analyze them spatially. In this blog post, we will review options available for geocoding Dynamics CRM data, including:
● Batch geocoding with Spatial Data Services
● Geocoding via Dynamics CRM Plug-ins
We will be working with:
● Dynamics CRM Online
● Excel 2013
● Visual Studio 2010
● The Developer Toolkit for Microsoft Dynamics CRM 2011 and Microsoft Dynamics CRM Online
● The Plug-in Registration Tool from the Microsoft Dynamics CRM SDK download
● A Bing Maps Account that has access to the Bing Maps Spatial Data Services
Batch Geocoding with Spatial Data Services:
We will first walk through the steps involved in geocoding entity data that is already contained within our CRM instance with the use of the Bing Maps Spatial Data Services Geocode Dataflow API. At a high level, we will be exporting entity data using the Dynamics CRM Web Client, geocoding the data using the Geocode Dataflow API, and then re-importing the data with latitudes and longitudes appended to each record.
In this example, we will geocode all Account records which have not already been geocoded. We start by logging into Dynamics CRM Online, selecting Accounts from the left navigation, and then selecting Advanced Find from the Data group on the Accounts tab of the ribbon.
In the resulting window, create a New query, and select Account records in which either the address Latitude or Longitude do not have data:
Now select Edit Columns for the View, and choose those representing the Street, Town, State/Province, ZIP/Postal Code, Country/Region, Latitude, and Longitude of the account address you wish to geocode:
Rearrange the order of the columns as shown below by selecting a column header and using the green arrow buttons:
Now execute your query to view the results by selecting Results. Once the results are displayed, Choose ‘Data… Export Accounts’ from the ribbon:
Choose to export to a ‘Static worksheet’, with data from all pages in the current view, if applicable. Make sure you select the checkbox to ‘Make the data available for re-importing’
When prompted, save the resulting XML Spreadsheet file to your machine, rather than opening it directly. Now open the file in Excel. Note that there are several hidden columns, including column A, which contains a unique ID for each record, along with the columns D through H which contain the address data needed for geocoding. Unhide all columns:
We will now extract the relevant data from this spreadsheet, and create a second spreadsheet which we will use with the Bing Maps Spatial Data Services Geocode Dataflow API.
Since the console application we will use conforms to the Geocode Dataflow API Data Schema v1, we will arrange our data to conform to that. Our mapping of columns will be:
● Column A: Unique Account ID
● Column D: Street
● Column E: State / Province
● Column F: Country / Region
● Column I: Town
● Column J: ZIP / Postal Code
We will now save the spreadsheet as Text (Tab-delimited), ready for use with the sample console application.
Now use Visual Studio to create a Visual C# Console Application, and copy the code from the Geocode Dataflow Sample Code. If necessary, make two small tweaks to the sample code. To make it easy to open our geocoding output data in Excel, we will use the WriteLine() method of the StreamWriter class, instead of the Write() method, to include line breaks in the output data (successfile and failedfile). Build the application, in preparation for geocoding the data.
The Geocode Dataflow API uses a REST API to enable you to:
● HTTP POST up to 200,000 records per job
● Check the status of your job, to determine when it has completed
● Download the results
Our Console Application will handle the interactions with the Geocode Dataflow API for us.
The Console Application takes in four parameters:
● dataFilePath: The path to the file that contains the spatial data to geocode
● dataFormat: The format of the input data. Possible values are xml, csv, tab and pipe. We are using tab format
● key: The Bing Maps Key to use for this job. The same key is used to get job status and download results.
● description: Text that is used to describe the geocode dataflow job.
Open up a Command Prompt, and execute the Console Application as shown below:
Upon completion, our geocoding output should be in a file named Success.txt in the current directory. Import this tab-delimited file into Excel, using UTF-8 encoding. The format of this file will conform to the Geocode Dataflow Schema v1.
The fields which will be of main interest to us will be these ones:
● GeocodeEntity/GeocodeResponse/RooftopLocation/@Latitude (Excel Column U)
● GeocodeEntity/GeocodeResponse/RooftopLocation/@Longitude (Excel Column V)
● GeocodeEntity/GeocodeResponse/InterpolatedLocation/@Latitude (Excel Column W)
● GeocodeEntity/GeocodeResponse/InterpolatedLocation/@Longitude (Excel Column X)
However, if you wish to be more selective about the geocoding results that you import into Dynamics CRM, you could use other fields as well. For example, you could choose to import only those results which returned an Address EntityType, with a High Confidence.
Generally speaking, unless we are using our latitudes and longitudes for routing purposes, we will want the Rooftop coordinates when available, and taking the Interpolated coordinates when Rooftop coordinates are not available.
We can now retrieve the most appropriate coordinates from our Success.txt file, and bring them into the Latitude and Longitude columns of our original spreadsheet. There are a number of options for doing this, but in this case, we use the VLOOKUP function in Excel to retrieve the coordinates from the appropriate record in the Success.txt file. If Rooftop coordinates are available, we use them, and if they are not, we use the Interpolated coordinates. If neither is available, or if the VLOOKUP fails, we leave the cell blank. The actual Excel formula used to populate the Latitude column for row 2 is:
=IF(ISERROR(VLOOKUP(A2,Success.txt!$A$1:$AB$6,21,FALSE)),"",IF(VLOOKUP(A2,Success.txt!$A$1:$AB$6,21,FALSE)<>"",VLOOKUP(A2,Success.txt!$A$1:$AB$6,21,FALSE),IF(VLOOKUP(A2,Success.txt!$A$1:$AB$6,23,FALSE)<>"",VLOOKUP(A2,Success.txt!$A$1:$AB$6,23,FALSE),""))) |
For longitude, we simply add one to the Column Index Number values of each VLOOKUP.
Now convert the formulas to values by highlighting the Latitude and Longitude cells, copying the data, and then Paste Values in the same cells.
After saving, we now have an updated Excel file ready for re-import to Dynamics CRM, with coordinates for our records:
Now we log back into Dynamics CRM Online, and select Accounts from the navigation. From the Accounts tab of the ribbon, choose Import Data from the Data group. In the resulting window, select the location of the XML Spreadsheet file containing our updates.
When prompted, select No for Allow Duplicates, and click Submit:
The data will now be imported, with Latitudes and Longitudes appended to the relevant records in Dynamics CRM.
Geocoding via Plugins
We will now walk through the process of creating a Dynamics CRM Plug-in which will enable CRM entity data to be geocoded whenever a new entity is added or updated. Our MSDN documentation describes a plug-in as custom business logic (code) that you can integrate with Microsoft Dynamics CRM 2011 and Microsoft Dynamics CRM Online to modify or augment the standard behavior of the platform. In our case, our plug-in will leverage the REST Locations API to geocode the address of any new accounts that are added or existing accounts that are updated, adding the resulting Latitude and Longitude to the record.
In Visual Studio, we will create a new Project, and will select a Dynamics CRM 2011 Package using Visual C#:
We are now prompted to Connect to Dynamics CRM Server. Since we are using Dynamics CRM Online, we enter dev.crm.dynamics.com as our CRM Discovery Server Name. We also enter our Microsoft account credentials for authentication:
We now add a new C# Dynamics CRM 2011 Plug-in Library project to our package:
We will now add a Visual C# Class to our GeocodeAccountsPlugin, called GeocodeAccounts.cs. In this class, we will add our code to read the address data from an Account record, geocode the address data using the REST Locations API, and then update the Latitude and Longitude of the Account record if a successful geocoding result is obtained.
When we register our plug-in, we will have execution take place post-operation, and asynchronously, for performance reasons. We will register and make use of a Post Image of our entity, which is effectively a reflection of the Account attributes once the Account Create or Update operation has completed.
The logic of our plug-in can be summarized as:
● The Account address details are retrieved from the Post Image
● The address details are used to prepare a URL for a request to the Bing Maps Locations API, specifying an XML response format
● The request is executed and the response retrieved with the use of the HttpWebRequest class
● The XML response is parsed and the Latitude and Longitude values of the first geocoding result are retrieved using XPath
● The Latitude and Longitude of the Account record are updated using IOrganizationService.Update()
namespace GeocodeAccountsPackage.GeocodeAccountsPlugin { using System; using System.ServiceModel; using Microsoft.Xrm.Sdk; using System.Xml; using System.Net; using System.Xml.XPath;
///<summary> /// Geocode Account Plugin. /// Fires on Create of Account, or Update of Account address properties ///</summary> publicclassGeocodeAccounts : IPlugin { privatereadonlystring postImageAlias = "PostImage"; privatereadonlystring BingMapsKey = "Insert Key Here";
publicvoid Execute(IServiceProvider serviceProvider) { // Obtain the execution context from the service provider. Microsoft.Xrm.Sdk.IPluginExecutionContext context = (Microsoft.Xrm.Sdk.IPluginExecutionContext) serviceProvider.GetService(typeof(Microsoft.Xrm.Sdk.IPluginExecutionContext));
IOrganizationServiceFactory serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory)); IOrganizationService service = serviceFactory.CreateOrganizationService(context.UserId);
// Obtain the post entity image: Entity postImage = (Entity)context.PostEntityImages[postImageAlias];
// The InputParameters collection contains all the data passed in the message request. if (context.InputParameters.Contains("Target") && context.InputParameters["Target"] isEntity) { // Obtain the target entity from the input parmameters. Entity entity = (Entity)context.InputParameters["Target"];
// Verify that the target entity represents an account. if (entity.LogicalName == "account") { try {
// Grab the postImage address attributes for our geocoding request: string addressLine = (postImage.Attributes.ContainsKey("address1_line1")) ? (string)postImage.Attributes["address1_line1"] : ""; string locality = (postImage.Attributes.ContainsKey("address1_city")) ? (string)postImage.Attributes["address1_city"] : ""; string adminDistrict = (postImage.Attributes.ContainsKey("address1_stateorprovince")) ? (string)postImage.Attributes["address1_stateorprovince"] : ""; string postalCode = (postImage.Attributes.ContainsKey("address1_postalcode")) ? (string)postImage.Attributes["address1_postalcode"] : ""; string countryRegion = (postImage.Attributes.ContainsKey("address1_country")) ? (string)postImage.Attributes["address1_country"] : "";
//Get latitude and longitude coordinates for specified location XmlDocument searchResponse = Geocode(addressLine, locality, adminDistrict, postalCode, countryRegion);
//Create namespace manager XmlNamespaceManager nsmgr = newXmlNamespaceManager(searchResponse.NameTable); nsmgr.AddNamespace("rest", "http://schemas.microsoft.com/search/local/ws/rest/v1");
//Get all geocode locations in the response XmlNodeList locationElements = searchResponse.SelectNodes("//rest:Location", nsmgr); if (locationElements.Count == 0) { // No geocoding results, so do nothing } else { //Get the geocode location points that are used for display (UsageType=Display) XmlNodeList displayGeocodePoints = locationElements[0].SelectNodes(".//rest:GeocodePoint/rest:UsageType[.='Display']/parent::node()", nsmgr); string latitude = displayGeocodePoints[0].SelectSingleNode(".//rest:Latitude", nsmgr).InnerText; string longitude = displayGeocodePoints[0].SelectSingleNode(".//rest:Longitude", nsmgr).InnerText;
Entity account = newEntity("account"); account.Attributes.Add("accountid", entity.Attributes["accountid"]); account.Attributes.Add("address1_latitude", double.Parse(latitude)); account.Attributes.Add("address1_longitude", double.Parse(longitude)); service.Update(account);
}
} catch (FaultException<OrganizationServiceFault> ex) { thrownewInvalidPluginExecutionException("An error occurred in the Account Update plug-in.", ex); } } } }
// Submit a REST Services or Spatial Data Services request and return the response protectedXmlDocument GetXmlResponse(string requestUrl) {
HttpWebRequest request = WebRequest.Create(requestUrl) asHttpWebRequest; using (HttpWebResponse response = request.GetResponse() asHttpWebResponse) { if (response.StatusCode != HttpStatusCode.OK) thrownewException(String.Format("Server error (HTTP {0}: {1}).", response.StatusCode, response.StatusDescription)); XmlDocument xmlDoc = newXmlDocument(); xmlDoc.Load(response.GetResponseStream()); return xmlDoc; } }
// Geocode an address and return the XML response: protectedXmlDocument Geocode(string addressLine, string locality, string adminDistrict, string postalCode, string countryRegion) { //Create REST Services geocode request using Locations API string geocodeRequest = String.Format("https://dev.virtualearth.net/REST/v1/Locations?countryRegion={0}&adminDistrict={1}&locality={2}&postalCode={3}&addressLine={4}&o=xml&key={5}", System.Uri.EscapeDataString(countryRegion), System.Uri.EscapeDataString(adminDistrict), System.Uri.EscapeDataString(locality), System.Uri.EscapeDataString(postalCode), System.Uri.EscapeDataString(addressLine), BingMapsKey);
//Make the request and get the response XmlDocument geocodeResponse = GetXmlResponse(geocodeRequest);
return (geocodeResponse); }
} }
|
After successfully building, we will sign the assembly, through the plug-in Properties…Signing (shown below) and then build in preparation for registering our plug-in:
We will now register our plug-in, using the process outlined in the Walkthrough: Register a Plug-in Using the Plug-in Registration Tool on MSDN (you can refer to this article for additional step-by-step information).
We run the Plug-in Registration tool, connect to the Dynamics CRM Online server, and then register our plug-in assembly:
We can follow the step-by-step instructions in the MSDN walkthrough to Register a Plug-in Assembly.
Next, we will register the plug-in for two events: Account Create and Account Update.
In each case, we will select (Assembly) GeocodeAccountsPackage.GeocodeAccountsPlugin in the tree view, and then select Register New Step from the Register menu.
For the Account Create event, we will select the following options:
For the Account Update event, we will select very similar options, adding in Filtering Attributes to ensure the plug-in is only executed on update when the address details for the account change. The Filtering Attributes used will be:
● Street 1 (address1_line1)
● City (address1_city)
● State/Province (address1_stateorprovince)
● ZIP/Postal Code (address1_postalcode)
● Country/Region (address1_country)
We will now Register a New Image for each step, to give us access to the address details of the Account after the Create and Update operations have executed. We specify an Entity Alias (‘PostImage’) which matches the Post Image alias we referenced in our plug-in code. Note how we specify only the address attributes for the Parameters, as these will be used when geocoding the address:
We register images for both the Create and Update steps.
Our Tree View in the Plug-in Registration Tool should now look like this:
We are now ready to add new account records, and update existing account records through the Dynamics CRM Online Web Application, to verify that our plug-in is geocoding the respective addresses. Note that because our plug-in is executing asynchronously after execution of the creation or update of our Accounts, the new Latitude and Longitude values may not necessarily be reflected immediately. After we reopen a newly created or updated record, we should see the new Latitude and Longitude values reflected.
Other Geocoding Options
In addition to geocoding via batch jobs, and via plug-ins, some other options that could be explored include:
● Geocoding individual addresses through HTML Web Resources in forms with the AJAX v7 map control and Search module
● Geocoding individual addresses through Form Scripting using the Bing Maps Locations API
Once our Dynamics CRM Accounts and other entities are geocoded, we can start to take advantage of your location data to find nearby accounts, create optimized itineraries, visualize data in specific territories or areas, and much more. For information on how to create a heat map of our data in Dynamics CRM, see my previous post on this topic.
The code for the GeocodeAccounts.cs class and the Geocode Dataflow console application can be found here
-Geoff Innis, Bing Maps Technical Specialist