Event-driven ingestion of Keycloak entities

You’ve probably experienced the annoyance of delayed updates to critical entity information, or witnessed the strain of constant, inefficient polling mechanisms chewing up resources just to keep data somewhat current. But what if your developer catalog could react instantly to changes in your identity provider? The Backstage Events System provides a solution to just that. This article will outline a Proof of Concept (PoC) for an event-driven ingestion mechanism, demonstrating how we can achieve near real-time synchronization of entities from Keycloak into Red Hat Developer Hub (RHDH), moving beyond the limitations of traditional polling.

So, what’s this “Events System” in Backstage all about? Think of it as an efficient messaging service. It can be used in a variety of use cases. For instance, it can be used by different plugins to communicate with each other and also by external services to interact with Backstage. In the context of this PoC, the Events System is specifically leveraged to facilitate the ingestion of entities from a Keycloak instance into the RHDH catalog.

Benefits of event-driven

There are benefits to going event-driven such as:

• Near real-time updates

  Entities in the RHDH catalog are updated almost immediately after changes occur in the external system. This ensures developers always see the most accurate and current information, without the delays associated with interval-based polling.

• Efficient, incremental syncing

  Rather than performing periodic full syncs, this event-driven approach updates only the affected entities that were actually changed, added, or deleted in the external system.

This results in:

• Fewer API calls, helping avoid rate limits.
• Lower CPU usage, since there is no unnecessary processing.
• Reduced database load because only specific records are modified.

Setting up the PoC

This section walks through setting up a PoC where the Backstage Keycloak plugin is modified to send user and group events to RHDH via webhooks. Since Keycloak does not natively support webhooks, we extend its functionality using the ext-event-webhook EventListener provided by p2-inc/keycloak-events. The following steps will guide you through building a custom Keycloak image, configuring realms and clients, enabling the event listener, and finally wiring it up with a locally running RHDH instance.

You can take a look at the PoC code here: https://github.com/04kash/community-plugins/pull/9 

Step 1 : Set up your Keycloak server

Clone the p2-inc/keycloak-events repository:

git clone https://github.com/p2-inc/keycloak-events
cd keycloak-events

Build the project:

mvn clean install -DskipTests

This will produce a JAR file located at target/keycloak-events-0.47-SNAPSHOT.jar, or a similarly named version based on your build.

Create a Containerfile with the following content:

FROM quay.io/keycloak/keycloak:26.2.4 (or use RHBK image: registry.redhat.io/rhbk/keycloak-rhel9:26.2) 
COPY target/keycloak-events-*.jar /opt/keycloak/providers/

Build the container image:

podman build -t my-keycloak-with-events .

Confirm the image is listed using the following command:

podman images

Start the Keycloak server by creating a podman network so that the Keycloak container and the RHDH container can communicate with each other.

podman network create rhdhnet

Run the Keycloak container using the image built in the previous step:

podman run -p 8080:8080 --network rhdhnet \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
  -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  my-keycloak-with-events start-dev

Next, disable SSL for development.

Access the running container:

podman exec -it <container-name> /bin/bash
cd /opt/keycloak/bin

Authenticate as the admin user:

./kcadm.sh config credentials --server http://localhost:8080 --realm master --user admin --password admin

Disable SSL for development:

./kcadm.sh update realms/master -s sslRequired=NONE

Create realm and client for webhooks as follows.

In the Keycloak admin console:

1. Create a new realm named demo-rhdh-events.
2. Inside this realm, create a client named rhdh-events-webhook.
3. Assign the following roles to the client’s service account:
   • view-events
   • manage-events
   • query-groups
   • query-users
   • view-users

Enable ext-event-webhook under Configure > Realm Settings > Events.

Now we will generate an access token.

NOTE: You might have to disable SSL for the demo-rhdh-events realm to proceed.

Run the following:

TOKEN=$(curl -X POST http://localhost:8080/realms/demo-rhdh-events/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "client_id=rhdh-events-webhook" \
  -d "grant_type=client_credentials" \
  -d "client_secret=<secret>")
ACCESS_TOKEN=$(echo $TOKEN | jq -r '.access_token')

Replace <secret> with the actual secret from the Keycloak admin console.

Create the webhook as follows:

curl -X POST http://localhost:8080/realms/demo-rhdh-events/webhooks/ \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "url": "http://<rhdh-container-name>:7007/api/events/http/keycloak",
    "eventTypes": [
      "admin.GROUP-CREATE",
      "admin.GROUP-DELETE",
      "admin.USER-UPDATE",
      "admin.USER-CREATE",
      "admin.USER-DELETE",
      "admin.GROUP_MEMBERSHIP-CREATE",
      "admin.GROUP_MEMBERSHIP-DELETE"
    ]
  }'

Verify the webhook by running the following:

curl -X GET http://localhost:8080/realms/demo-rhdh-events/webhooks \
  -H "Authorization: Bearer $ACCESS_TOKEN"

Step 2: Set up RHDH local

1. Clone the 04kash/rhdh-local repository.

git clone https://github.com/04kash/rhdh-local.git && cd rhdh-local

2. Under configs/dynamic-plugins, create a new file dynamic-plugins.override.yaml and copy over the contents from dynamic-plugins.override.example.yaml into it.

3. Under configs/app-config, create a file named app-config.local.yaml and copy the contents from app-config.local.example.yaml into it. In this new file, set the value of catalog.providers.keycloakOrg.default.clientSecret to your Keycloak client secret. Additionally, configure catalog.providers.keycloakOrg.default.baseUrl with the base URL of your Keycloak server, typically in the format http://<keycloak-container-name>:8080/.

4. Run the following command to start RHDH local:

podman-compose up -d

5. Add the RHDH container to the rhdhnet network:

podman network connect rhdhnet rhdh

6. Rerun the install-dynamic-plugins container and restart the rhdh container

podman-compose run install-dynamic-plugins
podman-compose stop rhdh && podman-compose start rhdh

Open http://localhost:7007 in your browser to access RHDH. You can now try creating or modifying users and groups in your Keycloak server. You should observe the corresponding entities added or updated in your RHDH catalog in real time.

Limitations and future improvements

There are a couple of other scenarios that can be added to this PoC:

• Updates to the Keycloak group, such as renames or property changes. It can potentially be supported by adding a function to handle admin.GROUP-UPDATE events.
• Adding users and groups to a Keycloak instance via partial imports. Adding users and groups using this method emits admin.REALM-CREATE, and this scenario can potentially be supported by adding a function to handle this event.

In a real-world production setup, there may be edge cases where events fail to reach RHDH, due to:

• Message delivery issues
• Temporary network outages
• Application crashes
• Unexpected bugs or configuration errors

To mitigate these risks, we recommend retaining periodic polling. This ensures that missed events don’t lead to catalog drift over time.

Rather than treating event-driven ingestion as a complete replacement for polling, view it as a performance optimization that enables near-instant updates, while falling back to scheduled syncs to guarantee consistency.

The post Event-driven ingestion of Keycloak entities appeared first on Red Hat Developer.