Introduction
Eugene Berman is ModusBox’s distinguished engineer who splits his time working with the Product Team and delivering solutions for the North American Consulting Team. Prior to working at ModusBox, Eugene spent over ten years at MuleSoft delivering integration solutions for customers ranging from small startups to members of the Fortune 50. We are pleased to share Eugene’s perspective on Digital Transformation, which he will present as an ongoing series with technical examples. By the end of this series, Eugene will provide best practices sample code and documents to help you build an actual use case.
Overview
Today practically all companies are focused on some form of digital transformation, even as they struggle to understand exactly what that entails. Typically, this is driven by the constant pressure across industries to deliver faster and more intuitive experiences to customers who are used to getting rapid responses to their business problems online and also to compete with other companies who are racing to do the same. Digital Transformation is a multifaceted change often introducing new technologies, requiring new skills, and resulting in an entirely new operating model.
Within this context, everything is an integration and integration is everything. At the most basic level, any task that requires moving data between two or more different endpoints is an integration. Surprisingly, many businesses fail to realize their integration needs, while others do but struggle to manage the resulting backlog of projects. During my ten years of delivering customer integrations, I’ve seen companies succeed or fail depending on their ability to address a few key technical areas. These are not the be-all and end-all of a company’s digital transformation but they are critical to succeed with integration projects and are often overlooked. In this post, I’ll cover these areas and provide detailed examples in the posts that follow.
Develop clear short-term and long-term goals
When starting an integration project it is important to consider both your company’s short-term goals and broader strategic objectives. A typical short-term goal may include replacing an individual legacy system that is outdated or no longer supported. Broader strategic objectives may include reducing the number of systems and technical staff required to support the business, or providing easy access to data so that partners and vendors can be leveraged to build compelling solutions. In the ideal scenario, you have defined your strategic roadmap and prioritized individual initiatives. Your roadmap would also include opportunities for optimizing your systems and promoting reusability of project assets. Where possible, each project leverages existing reusable assets as well as producing new ones that will be usable by future projects.
Identify Your Endpoints and Have Most Ready in a Pre-Production Environment
Where possible, you will want to identify your endpoints in advance of the design or implementation work. To accomplish this, create a diagram that clearly lists all the third-party systems and their interfaces. Make sure you have a pre-production copy of all your systems and also ensure that their structure and data are identical to the production versions. Make sure all endpoints are available, functional and configured correctly before the actual coding of the integration project starts. You can then proceed to build system APIs for your endpoints.
The two common stumbling blocks you want to avoid are:
- Rushing into the project before your organization has identified all the endpoints and agreed on their configuration and data
- Failing to have pre-production copies of the endpoints that mirror production as closely as possible
This is something your organization can control up front before even starting an integration project. It is necessary pre-work that any skilled leader can initiate and drive to successful conclusion. Failure to do this, however, results in increased cost and rework and frequently in project cancellation or failure.
Document Use Cases
Identify your use cases and document them in as much detail as possible.
For example, “User places order” can be documented in terms of more detailed flows:
- Website sends order information
- Payment gateway processes the payment
- Order is written into the database
- User receives confirmation
There are many standard use case templates you can use to ensure that you are documenting your use cases consistently and with the correct level of information. At Modus Box, our use case template typically cover the following:
Title – A clear verb/noun or actor/verb/noun descriptor that communicates the scope of the use case.
Brief Description – A brief paragraph of text describing the scope of the use case.
Actors – A list of the types of users who can engage in the activities described in the use case.
Actor names should not correspond to job titles. With some technical use cases, the actors may be systems or processes as well as people.
Preconditions – Anything the solution can assume to be true when the use case begins.
Basic Flow – The set of steps the actors take to accomplish the goal of the use case. A clear description of what the system does in response to each user action. NOTE: It is always good if the flow can include an interaction diagram.
Alternate Flows – Capture the less common user/system interactions, such as trying to place an order but the user is on a new computer and they need to answer a security question.
Exception Flows – The things that can happen that prevent the user from achieving their goal, such as providing an incorrect username and password when trying to place an order.
Post Conditions – Anything that must be true when the use case is complete.
You will want to validate your use cases typically with the owners of systems and business processes. Later in this series, I will provide a use case example and then build it out in subsequence posts with documentation and code.
Represent Your Use Cases in Detailed Sequence Diagrams
You will want to illustrate your use cases in detailed sequence diagrams. The purpose of doing this is to visually model what you have written so that developers can clearly understand the flow of logic in your system, the entities involved, and the messages exchanged.
Producing and validating these diagrams (typically with the owners of systems and business processes) will greatly enable your developers to understand the flow of business logic and to build process APIs. Sequence diagrams can easily be translated into integration flows.
Develop a Canonical Data Model
You will want to develop a canonical data model which accommodates fields and objects from all different endpoints. The canonical data model will identify the mapping rules and relationships between objects and will be used by all of your APIs.
While this may seem logical, it is surprising how challenging it can be for organizations to gather the required owners and SMEs to identify the mappings, review them, and validate them as many times as possible.
Imagine NetSuite, Salesforce, and Magento and that the customer object exists in each of them. It may be named differently and may exchange data in different formats within each system, but it’s still there.
Create an abstraction of the customer object in a data model. Some fields would be common to a customer for all systems such as first name, last name, etc. Some fields may be specific to individual systems but if they are mapped to other systems, you should include them in the canonical data model as well.
This abstraction can be used and reused by all APIs. If another API requires the customer object then you pass the object in the canonical data model.
It is also a good idea to identify one system that will be a master system of record. For example, a customer record in NetSuite, Salesforce, and Magento would have different IDs for the same customer. One can select a system that will be a master and propagate its changes to other systems.For example, Salesforce will be a master system and other systems will contain salesforceID field for each record. Or a generic customerID field can be created and included in data models of all systems that may contain customer records.
Develop a detailed test plan
It is important to develop test plans as early as possible. These will frequently be derived in part from your use cases and sequence diagrams.
You should think about your test plans when you write your use cases and they should identify success criteria. At that point, you should also ask how you test your solution and what specifically you test.
Unit tests are as important as functional tests. Test your mappings in isolation, and then test the entire workflow.
Mapping tests are unit test. But not all unit tests are mapping tests.
Unit tests for canonical data model are critical. For example, each mapping from the canonical data model to the format required by a system proxied by an API would have a corresponding MUnit test. Make sure the CDM has examples for each object, and use the same examples for the test. For example, if you have a Salesforce System API where you map an Account object from CDM to the format required by the Salesforce connector (sObject), have an example of Account in CDM format and an example of Account object in the sObject format. Then create an MUnit test which loads the CDM example, runs your mapping and compares the result to the example of Account in sObject format.
I usually extract the RAML containing the CDM types to a separate module, along with all the necessary schemas and examples, and then all my APIs have this module as a dependency. This way I can ensure the integrity of my mapping tests because they are all tested against the same set of examples.
Conclusion
If you have read this post and concluded that it is asking you to do very basic things that are already germane to your organization then that’s great news. Likely you have solid practices in place that will facilitate successful delivery of your short-term integration projects and that will help support your long-term integration roadmap. But beware that while many individuals in an organization understand the value of each of these things, many organizations also find that doing all of them correctly is a surprisingly challenging practice.
ModusBox powers the digital transformations of organizations across the globe and across verticals. Want more information on how ModusBox can help you? Contact us at [email protected].
Thank you for the use case template. Makes a lot of sense to define our flows in this format as we prepare our integration platform and it is useful to know that this is what engineers will need in order to execute the plan to success.