Our Event Store – A move towards event-driven microservices

·

We have many ambitious projects to help our clients achieve operational excellence. These include making the switch to modern IT architectures and technologies. Last month we already took a look at how we use APIs and webhooks to integrate customers’ IT systems with our solutions. These integrations are the ground work to allow data-exchanges; you can read more about that here: https://www.egssis.com/integrating-egssport-in-your-it-ecosystem/

Once the systems have been integrated and data-flows are possible, we can take the next step towards event-driven patterns. Jan Corluy, our CTO, explains why this is so important:

To begin with: in an event-driven architecture, IT services communicate with each-other through event messages. When a change in one of the systems occurs, an event is published. At the same time, other services can react to these events for example: a forecasting update triggers a notification via sms, an alert, or popup.

What are the main benefits of becoming more event-driven for companies active in gas & power markets?:

  • Improved operational excellence: Every organization needs to respond quickly and effectively to changing markets or circumstances. Event-driven architecture combines data-pattern identification with automatic alerts and notifications to the right people. This enables businesses to make operational decisions in real-time.
  • Versatile & scalable: Event-driven applications are typically nonlinear and asynchronous, which makes them more adaptable. These loosely coupled computing nodes work together to form a cohesive event-processing engine with unlimited scale.

How can you as a customer benefit from events coming from the egssPort system? That is where the EGSSIS Event Store enters the picture.

Our CTO’s Vision on the EGGSIS Event Store

At E-World 2020 I shared my vision on how we can integrate our data with our customers’ data. The goal was to set-up our own EGSSIS EventStore where our customers could ‘subscribe’ to events. This means if an ‘event’ takes place, for example ‘a new forecast from the ETRM system is imported‘, then you as a customer can react to these events.

Whenever there is a change in egssPort, our hub for balancing your gas & power portfolios, the change will be sent to the EventStore where it will be stored & put on the Azure EventHub.

An EGSSIS customer who is connected to the EventHub will be notified whenever a new event arrives so that they can process that event in their IT systems of choice. The Event Store system runs in the Azure cloud-hosted on an Azure Kubernetes Service.

This infrastructure allows us to automatically scale the Event Store system based on load.  The load comes from either egssPort that is pushing in events, or from the customer’s systems which are reading out events from the Event Store Database.

Design of the system

There’s a lot of in-depth information available on our website: https://docs.egssis.com/eventhub/

Below you can see an example of a real-life scenario and how events trigger different data flows or actions:

As you can see in the picture the flow of data goes as follows in this example:

  1. A TSO sends a NominationResponse (Confirmation) to egssPort for customer X
  2. egssPort will process the NominationResponse
  3. egssPort stores these values in our time-series database for that customers on the specific contract
  4. In the same transaction, these changes are sent to the EGSSIS EventStore
  5. The EventStore will capture these changes as Events (=JSON payloads) which contain the data of this change 
  6. When the data is stored in the EventStore DB, an event is published on the Azure EventHub FYI: each customer will have their own eventhub!
  7. The customer’s IT system are listening/subscribe to the Azure EventHub
  8. The listening IT systems process the event payload/data and trigger actions (alerts, communication via sms/phone/email, etc.)

Our first real-life implementation

During Q4 2020 we started an integration project with Ørsted. Together with their teams we implemented the EventHub subscriber and the integration with our EventStore API to fetch event payloads.

This system ran in parallel with existing data exports to make sure that their data warehouse contained the same time series values. The advantages for Ørsted in using EventStore instead of relying on data exports:

  • Each change is pushed instead of relying on data exports, now they have all changes in the data in real-time
  • Scalable architecture: EventStore API and Azure EventHub can be scaled and processed in parallel.

One of our key contacts at Ørsted shares his experience below:

At Ørsted we need to trust that our data is correct and most recent, and we were really pleased when EGSSIS introduced data streaming via EventHub. This enabled us to rely fully on our data even when using an external hosted SaaS IT solution like egssPort. This also enables the possibilities for some applications to subscribe to certain data events for faster decision-making.

Anders Aaholm Pedersen – Senior Business Analyst – Portfolio Management NWE