Skip to main content

Running Multiple Consumers

The Solifi Consumer application can run multiple consumers in parallel, all writing to the same database. Parallel running is helpful when there are topics that need more capacity to process data (like transactional data) or where there are multiple numbers of topics to read from.

Benefits of Multiple Consumers​

  • Increased throughput - Process more messages simultaneously
  • Topic isolation - Allocate different topics to different consumers
  • Reduced processing time - Distribute workload across consumers
  • Better resource utilization - Right-size resources per consumer

Example 1: Multiple Consumers with Different Topics​

Start separate consumer instances, each handling different topics.

Consumer 1 Configuration​

solifi:
  topics:
    - TopicA

Start with:

java -jar solifi-consumer-<version>.jar

Consumer 2 Configuration​

In a separate folder, configure:

solifi:
  topics:
    - TopicB

Start with:

java -jar solifi-consumer-<version>.jar

Result​

  • Tables are created only once
  • Consumer1 connects and reads all data from TopicA
  • Consumer2 connects and reads all data from TopicB
  • Both process data separately without interference
  • Both insert/update data into the tables in parallel

Example 2: Multiple Consumers with Overlapping Topics​

You can configure consumers with overlapping topic assignments. Kafka will automatically distribute partitions among consumers in the same group.

Consumer 1 Configuration​

spring:
  kafka:
    consumer:
      group-id: mycompany-consumer-group
solifi:
  topics:
    - TopicA
    - TopicB

Consumer 2 Configuration​

spring:
  kafka:
    consumer:
      group-id: mycompany-consumer-group  # Same group ID
solifi:
  topics:
    - TopicB
    - TopicC

Consumer 3 Configuration​

spring:
  kafka:
    consumer:
      group-id: mycompany-consumer-group  # Same group ID
solifi:
  topics:
    - TopicD

Result​

  • Tables are created only once
  • Consumer1 reads all data from TopicA and TopicB
  • Consumer2 reads all data from TopicC
    • If TopicB has multiple partitions, Consumer2 may also receive data from TopicB (Kafka will distribute partitions)
    • If TopicB has only one partition, Consumer2 will not receive data from TopicB
  • Stopping Consumer2 will re-allocate TopicB data entirely to Consumer1
  • Consumer3 reads all data from TopicD
  • All consumers process data separately and insert/update in parallel
Kafka Partition Assignment

When multiple consumers subscribe to the same topic with the same group-id, Kafka automatically distributes topic partitions among them. The distribution depends on the partition assignment strategy configured.

Running Consumers in Different Environments​

You need to update the application.yml file according to the environment.

For Kubernetes deployments using ConfigMaps:

  • Update the ConfigMaps for each namespace
  • Apply the updated configuration
# Update ConfigMap for UAT
kubectl create configmap consumer-config \
  --from-file=application-uat.yml=application.yml \
  -n uat --dry-run=client -o yaml | kubectl apply -f -

# Update ConfigMap for Production
kubectl create configmap consumer-config \
  --from-file=application-prod.yml=application.yml \
  -n prod --dry-run=client -o yaml | kubectl apply -f -

Docker Compose with Multiple Consumers​

services:
  consumer-topics-a:
    image: limepoint/solifi-consumer:2.2.4
    volumes:
      - ./application-topics-a.yml:/application.yml
      - ./license.license:/license.license
      - ./sign256.sign:/sign256.sign
    environment:
      - spring.config.additional-location=/application.yml

  consumer-topics-b:
    image: limepoint/solifi-consumer:2.2.4
    volumes:
      - ./application-topics-b.yml:/application.yml
      - ./license.license:/license.license
      - ./sign256.sign:/sign256.sign
    environment:
      - spring.config.additional-location=/application.yml

Best Practices​

  1. Use consistent group IDs for consumers that should share partitions
  2. Use different group IDs for consumers that need independent processing
  3. Monitor consumer lag to identify bottlenecks
  4. Right-size resources based on topic volume
  5. Consider partition count when planning consumer count

Next Steps​