Spring Cloud Config, Spring Cloud Bus & Kafka | How to set up automatic updates of your configuration
In this article, I will show you how to achieve automatic updates with Spring Cloud Bus and Kafka
The Spring Cloud Bus, according to documentation:
Spring Cloud Bus links nodes of a distributed system with a lightweight message broker. This can then broadcast state changes (e.g., configuration changes) or other management instructions. AMQP and Kafka broker implementations are included in the project. Alternatively, any Spring Cloud Stream binder on the classpath will work out of the box as a transport.
I will start from common practice nowadays without a message broker.
First, we need to create a repository with your app’s configuration. There is no rocket science to create a new repository. After that, we can edit with web IDE, clone, add, commit, and push the app’s configuration to the remote repository.
Example of configuration in repository
echo:
message:
text: 'Hello, world!'The second part is to create a cloud config server and set up the correct configuration for this server. We must set up a remote repository and credentials to make the configuration accessible.
Before developing a service that will be consuming configuration, we need to add an annotation that enables the config server and provides configuration.
package io.vrnsky.cloudconfigservice;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.config.server.EnableConfigServer;
@SpringBootApplication
@EnableConfigServer
public class CloudConfigServiceApplication {
public static void main(String[] args) {
SpringApplication.run(CloudConfigServiceApplication.class, args);
}
}server:
port: 8888
spring:
cloud:
config:
enabled: true
server:
git:
uri: https://github.com/vrnsky/medium-config
username: vrnsky
password: your_github_personal_access_token here
try-master-branch: false
clone-on-start: true
search-paths:
- medium-service
logging:
level:
org.springframework.cloud: DEBUGAfter a successful start, we can check if the configuration is available.
GitCredentialsProviderFactory : Constructing UsernamePasswordCredentialsProvider for URI https://github.com/vrnsky/medium-configcurl "http://localhost:8888/medium-service/main"
{"name":"medium-service","profiles":["main"],"label":null,"version":"f3c546784ab421046174a4119f2ca250184e740b","state":null,"propertySources":[{"name":"https://github.com/vrnsky/medium-config/medium-service/application.yml","source":{"echo.message.text":"Hello, world!"}}]}%
Now, let’s configure our service to interact with the cloud-config server. I will configure the application to use the config server application running on my machine.
spring:
application:
name: medium-service
cloud:
config:
enabled: true
uri: http://localhost:8888The next part is to create configuration properties of the medium service.
package io.vrnsky.mediumservice.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties(prefix = "echo.message")
public class MediumServiceConfig {
private String text;
}Now, let’s create a controller that will return the message value from the configuration.
package io.vrnsky.mediumservice.controller;
import io.vrnsky.mediumservice.config.MediumServiceConfig;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequiredArgsConstructor
public class MediumController {
private final MediumServiceConfig config;
@GetMapping("/sayHello")
public String getMessage() {
return config.getText();
}
}Let’s check if the property has been read from the cloud config server.
curl http://localhost:8080/sayHello
Hello, world!After successfully managing to run both the service and cloud config server, we are ready to move with adding Spring Cloud Bus integration.
First, we need to bootstrap with docker-compose Kafka and Zookeeper.
version: "3"
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ports:
- "22181:2181"
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,EXTERNAL://localhost:29092
KAFKA_LISTENERS: PLAINTEXT://:9092,EXTERNAL://:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ports:
- "9092:9092"
- "29092:29092"Start Kafka and Zookeeper with the following command:
docker compose upThe next thing we will add is to configure our Cloud Config Server and our service to push and consume events from the message broker.
In your cloud config server, pom.xml add the following dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-monitor</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>The next step is to add properties in application.yml inside the resources folder of the config server.
spring:
kafka:
bootstrap-servers:
- http://localhost:29092
cloud:
bus:
enabled: trueNow, try to run the cloud config server and check if it succeeded in connecting with the message broker. You will see similar log messages if the connection has been established successfully.
.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137-2, groupId=anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137] Adding newly assigned partitions: springCloudBus-0
2023-12-21T13:31:23.119+08:00 INFO 7578 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137-2, groupId=anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137] Found no committed offset for partition springCloudBus-0
2023-12-21T13:31:23.123+08:00 INFO 7578 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137-2, groupId=anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137] Found no committed offset for partition springCloudBus-0
2023-12-21T13:31:23.133+08:00 INFO 7578 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState : [Consumer clientId=consumer-anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137-2, groupId=anonymous.7c65c8b8-caaa-45e1-b5bb-2f9c8eb3b137] Resetting offset for partition springCloudBus-0 to position FetchPosition{offset=3, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[localhost:29092 (id: 1001 rack: null)], epoch=Now, move on to the service side; in the pom.xml of service, add the following dependency:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-kafka</artifactId>
</dependency>The next step is configuring the service to listen to messages from the message broker.
spring:
config:
import: optional:configserver:http://localhost:8888
application:
name: medium-service
cloud:
config:
enabled: true
uri: http://localhost:8888
stream:
kafka:
binder:
brokers:
- http://localhost:29092
bus:
trace:
enabled: true
refresh:
enabled: true
env:
enabled: trueNow, we can start our config server and server. Please ensure that Kafka and Zookeeper have been created before bootstrapping the cloud config server and service.
The automatic updates use webhooks from version control system platforms such as GitHub and GitLab. In this article, for correct working, the services should be deployed somewhere, like Cloud Platform Providers, but to keep things simple, we will do it on a local machine.
You can create a webhook inside the repository’s settings section and get an idea of what information can be obtained from the webhook payload. I have used https://webhook.site to get webhooks from the medium-config repository.
Let’s change our property in a repository and manually trigger the configuration update.
echo:
message:
text: 'Hello, Medium'You may see https://webhook.site, which has successfully caught the webhook. Copy the payload of the webhook we are going to use for the update configuration.
It’s time to trigger the update by following the command:
curl -X POST 'http://localhost:8888/monitor?path=medium-service" -d '{webhook_payload}'You should see something similar in the logs of the cloud-config server. The logs tell us the message has been sent to the broker.
o.s.c.s.m.DirectWithAttributesChannel : postSend (sent=true) on channel 'bean 'springCloudBusInput'', message: GenericMessage [payload=byte[370], headers={kafka_offset=5, scst_nativeHeadersPresent=true, kafka_consumer=org.springframework.kafka.coreIn logs of service, you should see the following logs:
2023-12-21T13:47:55.412+08:00 INFO 4389 --- [medium-service] [container-0-C-1] o.s.c.c.c.ConfigServerConfigDataLoader : Located environment: name=medium-service, profiles=[default], label=null, version=a4ea5778de48d08b36e6a8ee310cf2e76ebde68f, state=null
2023-12-21T13:47:55.438+08:00 INFO 4389 --- [medium-service] [container-0-C-1] o.s.cloud.bus.event.RefreshListener : Keys refreshed [config.client.version, echo.message.text]To check that the configuration was updated, we can again curl the sayHello method of the medium service.
curl http://localhost:8080/sayHello
Hello, Medium