Artifact Synchronization with GitHub: WSO2 API Manager
Configure and use Artifact Synchronization with GitHub in WSO2 API Manager
Greetings Everyone!!! 👋
Looping back again with another blog and a custom implementation specific to the Artifact Synchronization feature of the WSO2 API Manager. In this writing, I will be providing a brief understanding of the Artficat Synchronization feature and the custom implementation made to achieve the Sync using GitHub as the storage space.
So, let’s get started … ✏️
WSO2 API Manager contains two types of artifacts; one is Runtime artifacts and the other is Design-time artifacts. The API Synapse artifacts which act as the API definitions for the Gateway nodes fall into the Runtime artifacts category. Without the Synapse artifact, the Gateway node will not able to serve the specific API requests during the runtime.
In a distributed deployment, where there are multiple Gateway nodes deployed, it is required to share the runtime artifacts among the nodes (basically within the Gateway nodes). For a long time, the API Manager documentation suggested and pointed to two mechanisms to achieve the sync between the nodes.
👉 One way is using Rsync. Rsync needs to be configured as a CRON job to check the differences and to sync between the nodes. Further, Rsync gets more complicated when handling two way syncing, and can result in a Single Point of Failure, etc.
👉 The other way is using an NFS (Network File System) mounted between the nodes to share the artifacts across them. In such a scenario, it is required to have a high-availability of NFS mounts in the Gateway node. Failing will result in the downtime of APIs served by the Gateway nodes. Further, at some point, the NFS can be costly enough to think about another solution.
To overcome these complex and failure scenarios and many others, the In-place Artifact Synchronization is introduced. This feature ensures that all (respective) Gateway nodes are updated with the respective API Synapse artifacts in the memory and to serve the APIs without any failure.
Now, you all may get a question on how this actually works? How the Gateway gets to know about the API artifacts how do they store them?…
You can find all the answers to your questions from here with pictorial explanations.
⭐️ Out-of-the-box the WSO2 API Manager supports Artifact Synchronization feature with Database as the storage. And, as usual, the API Manager provides a couple of extension points to build and deploy our own implementations to achieve the artifact sync with other types of storage.
So in this writing, I will be presenting you with a custom implementation to perform sync using GitHub as the storage in API Manager v3.2.0.
Given below is the Git repo containing the complete source and the guide to configure the API Manager and the GitHub to achieve the requirement
Let’s get a brief understanding of how the custom implementation is composed and stores the artifacts in GitHub.
The presented implementation has 3 sub-components where each component is designated to perform a set of operations
- Core component: The core module on controlling the communication with GitHub using Feign and utilities
- Artifact Saver component: The saver module to perform artifact saving and updating operations with GitHub (committing and maintaining)
- Artifact Retriever component: The retriever module to perform artifact retrieval from GitHub and pre-processing ’em
The core component requires access to GitHub to perform commit operations and retrieval on behalf of the user. Hence, we have to generate a Personal Access Token from GitHub with only the required permissions to perform the sync operations.
You can find more detailed instructions on generating a Personal Access Token from GitHub here
Once the repo and Personal Access Token is ready, as the next step, we have to configure our API Manager nodes, especially the Publisher and the Gateway nodes.
Given below is a high-level diagram plotted with the communication links between the nodes
As you all may know, the Publisher node is used to create and publish APIs in the API Manager echo system. Hence, our Saver module needs to be deployed and configured in the Publisher nodes to communicate with the GitHub and to store them with related meta-information. The related configurations can be found here
Then, the Gateway nodes need to be configured with the Retriever module to communicate and pull the artifacts from the GitHub repo. You can find the required configuration steps and instructions here
Note that the Core module needs to be deployed in both Publisher and Gateway nodes because the core implementations of communication links and utilities are packaged in the Core implementation
Further, for the Artifact Synchronization to work in a distributed environment, it is required to have active communication 🔗 with the Traffic Manager node. Because the events and notifications on API updates are passed to the Gateway servers through the Traffic Manager node.
Hope the presented writing, provides you a quick understanding of the Artifact Synchronization feature of the WSO2 API Manager and a brief understanding of configuring GitHub as a storage type to achieve the sync between the Gateway nodes. 👌 😃
In addition to the above, Have a look at another extension implemented by Shanaka Sandanayaka here which uses the Amazon S3 Bucket 🔥 🔥 as the storage to perform the sync 👏 👌
