Dynamics 365 CE as Microservice using Azure Service Bus

In this post we would like to discuss, whether Dynamics 365 CE can be used and connected to an online shop and Navision as a microservice. We think it can and we would like to present an example, how we were able to leverage Dynamics 365 CE Online as validation service (a lightweight master data management service – MDM) using Azure Service Bus.

Microservices

Concerning the explication of microservices we would like to refer to the description of Martin Fowler which can be found here. For the German speaking audience we would recommend the following talk of Rainer Stropek at the Vienna Tech Conference about Cloud Native and microservices.

The key attributes of microservices we refer to are:

• loosely coupled services/components
• organized around business capabilities
• smart endpoints and dumb pipes
• decentralized data management

Requirements

The customer approached us with the need to upgrade MS CRM 2011, move it to the cloud, and connect it to a custom shop implementation as well as to Microsoft Navision (NAV) (on premise – upgraded as well). One of the most challenging requirements for the new system architecture was to connect those services but to send only validated data to Navision (for billing and order processing).

Solution in a nutshell

The new system architecture, that we recommended, was built upon the usage of Azure Service Bus as messaging bus for decoupling the systems (CRM, Shop, ERP, and future systems to be connected). Dynamics 365 CE for Sales (formerly known as Microsoft Dynamics Customer Relationship Management – MS CRM) was set up for managing master data and to handle the validation of data before being sent to Navision for billing. Data responsibility for master data is split, and lies in Dynamics 365 (if no customer yet) or in Navision (if already customer). During the first order processing the responsibility is handed over from CRM to NAV. CRM always distributes data changes (no matter if caused by the customer in the self service portal in the shop, in Navision or in CRM), hence it serves as a lightweight MDM service. If the customer changes data in the online shop, it’s not directly sent to Navision, but first has to be validated in CRM. Purchases in the online shop are furthermore either sent directly to Navision (if the customer is validated) or kept in CRM and delayed until a successful validation, if a data change is pending and the customer’s data is currently not valid.

Why Azure Service Bus?

Azure Services Bus is easy to set up and Dynamics 365 CE comes with an integration out of the box (Azure-Aware out of the box (OOB) plugin). For the basic idea have a look here. Using the plugin registration tool it’s possible to register a service endpoint.

Having registered the endpoint it is very similar to registering a plugin. Just define the event and entity and use it asynchronous. The reason why it is called out of the box (OOB) plugin is, because you do not have to code anything. The azure aware out of the box plugin is already shipped with Dynamics 365 and it makes it quite easy to pass the execution context onto Azure Service Bus. For every event that now triggers on the defined event, a message will be sent to the queue/topic that you used in the registration of the service endpoint above. Of course you need to create the queue/topic in Azure beforehand and pass the SAS key above. This we will describe in a separate blog post where we focus on the technical part only.

We chose Azure Service Bus in our projects for several reasons:

• loosely coupling of systems
  • asynchronous communication -> less impacts on other systems on deployments, upgrades
  • systems can be more easily replaced with solutions of other providers
  • hiding implementation details -> other systems do not need to know anything about the Dynamics 365 APIs
• fairly easy to connect new systems -> creating new subscriptions and resend messages for existing data
• AMQP standard -> makes it easy to switch to another messaging bus provider if necessary
• low cost for set up -> platform as a service (PAAS)

The main advantage of Azure Service Bus over other messaging bus providers is the out of the box integration with Dynamics 365 CE. Furthermore the Azure Service Bus is no full blown Enterprise Service Bus. Experience in many projects over the last 10 years has shown that usage of an Enterprise Service Bus (with orchestration, transformation, etc.) often results in a very difficult set up for those projects since this part of the projects tends to become the most complex one where lots of integration problems have to be solved by only one team. This is why it’s one of the principles of the microservices architecture (smart endpoints and dumb pipes). Of course any other messaging technology (Apache MQ, IBM MQ, Amazon SQS, etc.) will do its job quite as good (apart from the integration to Dynamics 365).

Decoupling Shop and Navision – Registration

The above graphic shows the registration process of a new customer in the custom online shop. It consists of:

1. Registration in the online shop and sending a message to Dynamics 365 CE. This is done in the topic “masterData_fromWeb”.
2. Dynamics 365 CE receives the message and sends on its own a message on the topic “masterData”. This is done because Dynamics 365 serves as master data service and Navision only excepts messages from Dynamics 365. Same is true for the online shop.

The message in 2) during the registration is basically the same as sent by the shop, except for the customer number which is generated within CRM (another microserivce CRM provides here). But the important thing to note here is, that it does not have to be, since we decoupled it. When a new customer is registered it is in an invalid state and although sent to Navision, it cannot be used for billing there yet. In case a new customer is entered directly within CRM, only 2) would be sent and the online shop would be informed about the new account.

Upon data manipulation by the customer in online shop the messages 1) and 2) are sent. If data is changed by a CRM user then only 2) will take effect.

The validation of the account data can now happen at this stage within CRM, but probably won’t, since the customer did not buy anything yet, so no need to take the effort of validation.

Decoupling Shop and Navision – Purchase in Online Shop

Upon a customer ordering a product in the online shop, the following steps happen:

1. Online Shop sends a message in the topic “webPurchase” to Dynamics 365 CE. Navision does not listen onto this message – so no subscription existing for NAV (again for decoupling purposes).
2. Dynamics 365 receives the web purchase and proceeds depending on the validation status of the account.
   • if valid, the web purchase is directly forwarded to Navision and the process continues below at 3). In the background an opportunity record is created with Dynamics 365 CE (see below) that contains the order details. Communication here happens directly using SOAP service (see below for the reason).
   • if not valid, the web purchase is not directly forwarded to Navision. An opportunity is created in Dynamics 365 CE that holds the detail of the online shop purchase. Dynamics 365 CE queues and activities are used to inform the CRM validation team that a validation is necessary, since a web purchase is pending validation for being executed and being billed. This validation step can here take some minutes or hours. Since the process is decoupled, the customer in the shop will not notice anything about this validation process.
3. Upon receiving the first web purchase of a customer, Navision takes over data responsibility. To do this, a synchronous REST call from Navision to Dynamics 365 CE is executed, which locks the fields in CRM. This is done synchronously, since it is important that data responsibility handover actually worked.
4. Navision sends an order message in the topic “orders” to inform Dynamics 365 of the processed / billed order, which Dynamics 365 can use to close the opportunity and to store further details that then can be used in segmentation for marketing.

Sending the web purchase from Dynamics 365 CE to Navision is not implemented via the Azure Service bus and asynchronous messaging but with a synchronous SOAP call. The reason is, that in the manual process of creating a quotation, the CRM user is waiting in the CRM user interface and this step needs to be fast and needs immediate response from Navision to provide feedback to the user and open the Navision URL to the just created quote in NAV. The same SOAP call is called automatically in the background when an online purchase occurs and the customer account is already validated.

Opportunities in CRM / Quotes in NAV

Using the Dynamics 365 CE business process and JavaScript it was quite easy to trigger the SOAP call from within the business process stage change (details upon that following in a future post). In this SOAP call, executed from a custom workflow activity, Navision returns the URL and ID of the quote in NAV. The URL is then opened in a new tab and the CRM user can continue in Navision to fill in the products to offer, etc. The same process is used for online purchases where the validation was still pending and the web purchases could not be automatically forwarded to Navision.

Data responsibility handover to NAV

As mentioned above Buy Acimox Amoxil , data responsibility is handed over to NAV, once a customer becomes a debtor. The fields in CRM are then locked and data required for billing can be changed in NAV only. Since not all CRM users have access to NAV and only accounting team is allowed to change billing critical master data, writable fields are provided where CRM users can enter their changes (see below the writable fields on the right – where CRM users can enter change “wishes” – and the locked fields to the left that can only be changed in NAV). Upon entering change wishes the account loses validation status and the validation team gets informed using CRM workflows, queues and activities. The same is true if the customer wants to change its data in the online shop. Also in this case the data is actually written into the change fields and validation status is lost.

Changing the data in NAV, the process looks as following:

1. Navision sends message with the changed master data in the topic “masterData_fromNav”
2. Dynamics 365 updates the values in the locked fields and spreads message to shop and back to Navision using the topic “masterData”

Advantages of the chosen solution

• Only validated account data is used for billing
• Navision holds those accounts for billing as debtors that it really needs (not all accounts of Dynamics 365 CE)
• Upon data changes by the customer in the online shop, validation status is lost again, and further web purchases have to wait until the data change request (validation) is done before the order processing proceeds
• Exact same process flow for manual quotes and online web purchases (which anyway just represent already accepted quotes). For the CRM users and the validation team it makes no difference. Only difference is the entry point into the process (online shop or CRM).

Conclusion

In the above mentioned solution we accomplished to decouple the web shop and Navision by using Dynamics 365 CE as validation service (a lightweight MDM). By choosing Azure Service Bus we fulfilled the principles of loosely coupling and the smart endpoints and dumb pipes. Dynamics 365 CE has been chosen for the validation, since it can serve that well with its workflow engine, hence fulfilling organized around business capabilities. Concerning decentralized data management CRM does a good job as well, when we see it in the context of the entire system architecture, where every service (shop, CRM, ERP) works upon its own data store.

Nevertheless, obvious, here we have to mention that CRM will comprise much more services (marketing, service, email sending, etc.) which are working on the same data set within Dynamics 365 CE and where those micro/macroservices share the same environment. So if we would look at Dynamics 365 CE like this, then it would not fulfill the principles of microservices. Having a look onto the future of Dynamics 365 CE Online and especially the use of the Common Data Service (CDS) CRM would never be able to be a microservice where all the services are decoupled. Nevertheless, we would like to emphasize that it still makes sense to thrive towards this goal, even though some services share the same data. No matter if it’s then a real microservice or if it’s more like service oriented architecture (SOA), it does not matter as long as we are able to benefit from some of the advantages of that type of architecture. And the benefits can be clearly seen by the customer and us, since lately we were able to connect a fourth system easily by setting up new subscriptions and using Logic Apps to handle the messages and save them to the new application. This we will then describe in a future post. Moreover in one of the next posts we will show more of the technical details concerning this set up, since this post was meant to describe the general architecture.